Status
Current state: Accepted
Discussion thread: https://lists.apache.org/thread/1fy4rcxfhlnrglb7zt64jply5vqgj89x
JIRA: CASSANDRA-18592
Released: Unreleased
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Users onboarding to Cloud have different ways of running and accessing Cassandra servers,
- Running Cassandra clusters on on-premise data centers and accessing them from internal network and/or from various external clouds like AWS, GCP, .. etc
- Running Cassandra clusters on Cloud environment and accessing them from cloud and/or internal network
In either of these scenarios, admins may need to restrict a set of users or teams to access Cassandra cluster only from a set of regions. For example, assuming a Cassandra cluster running on on-premise data center,
- One requirement can be, admins/superusers credentials should be allowed only from internal network, to avoid misusing or hacking special privileges from an external cloud which can damage Cluster data
- If the Cluster is shared by different teams in a organization, allowing certain teams to access the cluster only from certain regions
- If an organization has multiple cloud environments or external networks, another requirement can be, allowing users access to the Cluster only from certain Cloud environment(s)
These requirements may not be possible to fulfill using network firewalls or security policies, because
- Some of these requirements are at user/team level, that means the same cluster may need to be accessible to other users/teams from the same region
- There can be multiple Cassandra clusters running on on-premise datacenter with the same internal network, some clusters need to be accessible from cloud environment(s) while other clusters shouldn’t be.
To support these scenarios, Cassandra server need ability to restrict users accesses based on incoming request’s IP range, aka, CIDRs. For reference, few other database servers already having similar features - MySql, MariaDB and PostgreSQL.
Note: From here onwards this document refers a ‘region’ as a ‘CIDR group’.
Audience
- Organizations with on-premise data centers and one or more Cloud environments
- Organizations with multiple users/teams sharing the same Cassandra cluster and different users/teams having restricted access from different CIDR groups
- Organizations with multiple Cassandra clusters running under the same network, exposed to access from internal and external networks, and different clusters having different access restrictions
Goals
- Ability to restrict individual or set of users accessing C* cluster from different CIDR groups
- Ability to restrict Cassandra cluster to be accessible only from certain networks when it’s not easy to achieve the same with network security
- Avoiding usage of credentials/certificates copied from one CIDR group to another, either by mistake or by hacking
Approach
Proposed changes highlighted below
- Retrieve user details from the incoming request.
- Authenticate incoming user, as it’s happening currently. Reject the request If not a valid user
- Perform currently existing authorization checks for the role
- Lookup system_auth.cidr_permissions table to retrieve CIDR permissions of the user. If the user has CIDR permissions ‘ALL’, i.e, allowed to access from any CIDR group, Approve the incoming request
- Retrieve the source IP from the incoming request.
- Lookup our new C* table system_auth.cidr_groups, which maps <CIDR groups to CIDRs>, to resolve the CIDR group of the source IP (Note: this won’t be a simple select query, next section describes the algorithm used to achieve this).
- Using CIDR permissions retrieved in step 4, check is incoming user enabled to access from the CIDR group of the incoming IP. If not enabled, reject the request. Otherwise approve the request
The same flow works for both Password based and mTLS based authentications. Incase of mtls based authentication, we will retrieve identity from the incoming certificate, map that identity to a role, and use that role for CIDR checks as listed above.
Proposed Changes
Mapping of CIDR groups to CIDRs
Created a new table system_auth.cidr_groups, which maintains mapping of CIDR groups to CIDRs. Assuming organizations have a way to map their internal CIDR groups to a list of CIDRs. We expect them to have a script/workflow that writes these mappings to the Cassandra table listed below. Organizations can have automated script/workflow to periodically update this table as and when their network changes.
A CIDR group can be associated with multiple CIDRs. A CIDR group without a deterministic IP range can be added as a wild card entry, for example 0.0.0.0/0 which matches every IP. We use longest matching prefix algorithm to find the narrowest matching CIDR for the incoming IP and associate it with corresponding CIDR group. For example, assuming below table,
- IP 10.30.1.1 matches with CIDR of AWS and incoming request would be tagged as coming from AWS.
- IP 10.30.20.1 matches with CIDRs of both AWS and GCP and would be tagged with narrowest matching CIDR, i.e, GCP
- IP 12.30.1.1 doesn’t match AWS and GCP, matches with wild card 0.0.0.0/0 and would be tagged as coming from Internal network.
CIDR_GROUP | CIDRs
-----------+-----------------
AWS | { 10.30.0.0/16, 11.20.1.0/24 }
GCP | { 10.30.20.0/24 }
Internal | { 0.0.0.0/0 }
Appendix - CIDRS Interval tree section describes the algorithm used for finding longest matching CIDR for an IP.
Associating roles to CIDR groups
Created a new table system_auth.cidr_permissions, to maintain mapping of roles to CIDR permissions, i.e, a set of CIDR groups that role is enabled for. Below table is an example.
role | CIDR_GROUPS
------------+----------------------------------------
roles/user1 | {}
roles/user2 | {'AWS', 'GCP}
roles/user3 | {'Internal'}
roles/user4 | {'AWS'}
Enabling and disabling roles for access from CIDR groups
Create and alter role CQL commands are modified to allow admin/superusers to enable or disable roles for CIDR groups. Here are examples.
CREATE ROLE user WITH ACCESS FROM ALL CIDRS ;
ALTER ROLE user WITH ACCESS FROM CIDRS { 'AWS', 'GCP' };
ALTER ROLE user WITH ACCESS FROM CIDRS { 'Internal' };
Feature flag and options in cassandra.yaml file
A new feature flag provided to enable/disable CIDR authorizer.
- By default AllowAllCIDRAuthorizer is used, which allows any user to access from any CIDR.
- CassandraCIDRAuthorizer provides MONITOR and ENFORCE modes. ENFORCE mode rejects users accesses from unauthorized CIDR groups. Where as, MONITOR mode doesn’t reject incoming connections, instead logs a warning message when an access attempted from unauthorized CIDR group. So MONITOR mode is useful to recognize from which CIDR groups a Cassandra cluster being accessed, for example, as an intermediate step during the migration to CIDR authorizer ENFORCE mode.
By default CIDR authorizer enabled only for non-superusers. We are providing an option cidr_checks_for_superusers to enable CIDR authorizer for super users as well. Note that CIDR checks can’t be performed during JMX calls, because currently there is no easy way to get client IP in the JMX request path (see CassandraLoginModule.java).
# CIDR authorization backend, implementing ICIDRAuthorizer; used to restrict user
# access from certain CIDRs
# Out of the box, Cassandra provides org.apache.cassandra.auth.{AllowAllCIDRAuthorizer,
# CassandraCIDRAuthorizer}.
# - AllowAllCIDRAuthorizer allows access from any CIDR to any user - set it to disable CIDR authorization.
# - CassandraCIDRAuthorizer stores user's CIDR permissions in system_auth.cidr_permissions table. Please
# increase system_auth keyspace replication factor if you use this authorizer, otherwise any changes to
# system_auth tables being used by this feature may be lost when a host goes down.
cidr_authorizer:
class_name: AllowAllCIDRAuthorizer
# Below parameters are used only when CIDR authorizer is enabled
# parameters:
# CIDR authorizer when enabled, i.e, CassandraCIDRAuthorizer, is applicable for non-superusers only by default.
# Set this setting to true, to enable CIDR authorization for superusers as well.
# Note: CIDR checks cannot be performed for JMX calls
# cidr_checks_for_superusers: true
# CIDR authorizer when enabled, supports MONITOR and ENFORCE modes. Default mode is MONITOR
# In MONITOR mode, CIDR checks are NOT enforced. Instead, CIDR groups of users accesses are logged using
# nospamlogger. A warning message would be logged if a user accesses from unauthorized CIDR group (but access won't
# be rejected). An info message would be logged otherwise.
# In ENFORCE mode, CIDR checks are enforced, i.e, users accesses would be rejected if attempted from unauthorized
# CIDR groups.
# cidr_authorizer_mode: MONITOR
# Refresh interval for CIDR groups cache, this value is considered in minutes
# cidr_groups_cache_refresh_interval: 5
# Maximum number of entries an IP to CIDR groups cache can accommodate
# ip_cache_max_size: 100
Compatibility, Deprecation, and Migration Plan
- Upgrade from older C* version to new (with this feature) shouldn’t be impacted because
- Only new tables introduced with this feature, schema of existing tables remains unchanged
- New feature flag introduced, behavior of existing feature flags remains unchanged
- By default CIDR filtering is disabled, so existing clusters upgrades can be smooth without manual intervention
- Default behavior for new and existing roles with CIDR authorizer
- Super users: By default super users are allowed to access from all CIDR groups. So there is no change in default behavior, unless cidr_checks_for_superusers is enabled
- Exiting users: All existing users, will continue to have access from all CIDR groups, without a change in existing behavior, as long as alter role command not ran on them to enable/disable CIDR groups explicitly
- New users: New users created using existing ‘create role’ command , i.e, without using the new clause ‘ACCESS FROM CIDRS’ to explicitly restrict CIDR groups, will have access from all CIDR groups. So there is no change in default behavior when using existing commands
- New clusters creation will have CIDR filtering disabled in cassandra.yaml file by default, so no manual action needed for default behavior
- Inter node authentication code doesn’t have ‘User’ and doesn’t perform roles/permissions checks, hence shouldn’t be impacted by CIDR filtering authorizer
New or Changed Public Interfaces
New error message
Below error message would be seen by a user trying to access the cluster and not having access from that IP (i.e, associated CIDR group)
"You do not have access from this IP <IP address of the request>"
For example, below output would be seen by a CQL user login from an IP not enabled to access from
% python3 cqlsh.py localhost -u user1 -p user1
Connection error: ('Unable to connect to any servers', {'::1:9042': ConnectionRefusedError(61, "Tried connecting to [('::1', 9042, 0, 0)]. Last error: Connection refused"),
'127.0.0.1:9042': Unauthorized('Error from server: code=2100 [Unauthorized] message="You do not have access from this IP 127.0.0.1"')})
Changes to CQL commands
'Create role’ command modified to be able to add CIDR permissions during the role creation. Below options added to the existing command:
| ACCESS FROM CIDRS set_literal
| ACCESS FROM ALL CIDRS
Examples:
CREATE ROLE role1 WITH LOGIN = true and PASSWORD = 'password_a' AND ACCESS FROM ALL CIDRS;
CREATE ROLE role2 WITH LOGIN = true and PASSWORD = 'password_b' AND ACCESS FROM CIDRS { 'aws' } ;
‘Alter role’ command modified to be able to update CIDR permissions of a role. Below options added to the existing command:
| ACCESS FROM CIDRS set_literal
| ACCESS FROM ALL CIDRS
Examples:
ALTER ROLE role2 with ACCESS FROM ALL CIDRS ;
ALTER ROLE role1 with ACCESS FROM CIDRS { 'internal' };
Changes to nodetool commands
New nodetool command provided to add/update a CIDR group name to CIDRs mapping
nodetool updatecidrgroup <CIDR group name> <CIDRs separated by space>
Example:
% nodetool updatecidrgroup aws 127.0.0.0/24 227.0.0.0/24
New nodetool command provided to list available CIDR groups, and also to list CIDRs associated with a CIDR group name
nodetool listcidrgroups
nodetool listcidrgroups <CIDR group name>
Examples:
% nodetool listcidrgroups CIDR Groups: internal aws gcp % nodetool listcidrgroups aws CIDRs: 127.0.0.0/24
New nodetool command provided to drop a CIDR group and associated mapping
nodetool dropcidrgroup <CIDR group name>
Example:
% nodetool dropcidrgroup aws
New nodetool command provided to invalidate CIDR permissions cache, for all roles or for a particular role
nodetool invalidatecidrpermissionscache nodetool invalidatecidrpermissionscache <role name>
New nodetool command provided to reload CIDR groups cache, i.e, populate CIDR groups cache with the latest state of the system_auth.cidr_groups table. Note that as part of this command, IP to CIDR groups cache which is maintained along with CIDR groups cache gets invalidated.
nodetool reloadcidrgroupscache
New nodetool command provided to see metrics related to CIDR filtering
nodetool cidrfilteringstats
New virtual tables
New virtual tables provided to list metrics of CIDR filtering authorizer. Below is the sample output of these tables.
cqlsh> select * from system_views.cidr_filtering_metrics_counts; name | value -----------------------------------------------------------+------- CIDR groups cache reload count | 2 Number of CIDR accesses accepted from CIDR group - aws | 15 Number of CIDR accesses accepted from CIDR group - gcp | 30 Number of CIDR accesses rejected from CIDR group - gcp | 6 cqlsh> select * from system_views.cidr_filtering_metrics_latencies; name | max | p50th | p95th | p999th | p99th ---------------------------------------------+-------+-------+-------+--------+------- CIDR checks latency (ns) | 24601 | 1 | 11864 | 24601 | 24601 CIDR groups cache reload latency (ns) | 42510 | 42510 | 42510 | 42510 | 42510 Lookup IP in CIDR groups cache latency (ns) | 1 | 1 | 1 | 1 | 1
Test Plan
- Unit tests added to test the new and changed code
- Ran Cassandra server with CIDR filtering disabled and tested no change in existing behavior
- Ran Cassandra server with CIDR filtering enabled, populated CIDR groups mappings, created users with different CIDR permissions and tested users accesses. Verified CIDR authorizer working as expected
- Developed JMH benchmark to measure impact of CIDR authorizer on Cassandra server authorization latency (Please see Benchmarks section)
- Plan is to run Cassandra upgrade test, from older version to new version with this feature and ensure no change in the functionality for existing users
Benchmarks
Below are results of benchmark of CIDR authorizer in ENFORCE mode (measured using JMH benchmark test on single C* instance). This score indicates additional nanoseconds to the current latency during C* server authorization path, when CIDR authorizer is enabled.
Benchmark Mode Score Error Units CIDRAuthorizerBench.benchCidrAuthorizer_InvalidLogin avgt 979.823 ± 8.837 ns/op CIDRAuthorizerBench.benchCidrAuthorizer_ValidLogin avgt 1076.707 ± 14.199 ns/op InvalidLogin - benchmarks the case of unsuccessful CIDR access ValidLogin - benchmarks the case of successful CIDR access
Potential Improvements
Currently we are maintaining explicitly enabled CIDR permissions for a user. This can be extended to also maintain explicitly denied CIDR permissions for a user. For example, if a user is allowed to access from all but one CIDR group, instead of explicitly enabling that user for all but one, can use deny list to restrict only from one CIDR group
Rejected Alternatives
Explored an option of using identity and location information of a mtls certificate for doing CIDR filtering. But it’s not reliable as it’s possible to copy a mtls certificate to a different location. Also, this approach restricts CIDR filtering only to mtls based authentication
Appendix - CIDRs Interval tree
(Suggested and implemented by Yifan Cai )
CIDRs interval tree is a variant of interval tree. Each node contains a CIDR and a value. In this specific case, the value is CIDR group name(s). A node has left children array and the right children array.
- The left children's CIDRs are either less than the starting IP of parent or overlaps with the parent node.
- The right children's CIDRs are either greater than the ending IP of the parent or overlaps with the parent node.
Note that the nodes that overlap with the parent node are included in both left and right children arrays.
The tree organizes nodes by placing non-overlapping CIDRs at the same level. In general, CIDRs with the same net mask do not overlap, hence are placed in the same level. CIDRs with different net mask may overlap, hence placed at different levels in the tree. In addition to this, there is an optimization to promote a CIDR to an upper level, if it is not overlapping with any CIDR in the parent level, that means, in such cases a CIDR with different net mask can co-locate in the same level with other CIDRs.
Levels closer to the root contains CIDRs with higher net mask value. Net mask value decreases as levels further down from the root. i.e, Nearer the level to the root, the narrower the CIDR, meaning matching the longer IP prefix.
Example:
Assume below CIDRs
"128.10.120.2/10", (i.e, IP range 128.0.0.0 - 128.63.255.255, netmask 10)
"128.20.120.2/20", (i.e, IP range 128.20.112.0 - 128.20.127.255, netmask 20)
"0.0.0.0/0", (i.e, IP range 0.0.0.0 - 255.255.255.255, netmask 0)
"10.1.1.2/10" (i.e, IP range 10.0.0.0 - 10.63.255.255, netmask 10)
The resulting interval tree looks like below. CIDR interval tree root points to level0 array.
level0 [ (10.0.0.0 - 10.63.255.255, 10) (128.20.112.0 - 128.20.127.255, 20) ]
/ \ / \
level1 / (128.0.0.0 - 128.63.255.255, 10)
/ / \
level2 (0.0.0.0 - 255.255.255.255, 0)
Note: In the above example, ideally CIDRs with netmask 20 should be at level 0, CIDRS with netmask 10 should be at level 1, CIDRs with net mask 0 should be at level 2. But due to the optimization, as (10.0.0.0 - 10.63.255.255, 10) does not have any overlapping CIDR, it is moved up a level
Search for Longest matching CIDR for an IP starts at level 0, if not found a match, search continues to the next level, until it finds a match or reaches leaf nodes without a match. That means search terminates on the first match closest to the root, i.e, locates narrowest matching CIDR.
CIDR interval tree on average provides O(logN) time complexity where N is number of number of CIDRs added to the tree. That’s because in most of practical cases input CIDRs won't be overlapping, so most of them locate in the same level, i.e, in the first level, so algorithm results into binary search of CIDRs in one level.
Worst case time complexity can be O(N) when all CIDRs inserted are overlapping with each other, so the tree is completely skewed to one side, and when the IP to be searched matches only with the broadest CIDR at the leaf level of the tree, i.e, search resulting into comparing with every CIDR at every level from the root to leaf, which is not a practically common scenario.
Here are benchmarks of finding longest matching CIDR in a CIDR interval tree (measured using JMH benchmark test)
Benchmark Mode Score Error Units LongestMatchingCIDRBench.benchLongestMatchingCidr_CidrExistsCase avgt 28.392 ± 0.164 ns/op LongestMatchingCIDRBench.benchLongestMatchingCidr_CidrNotExistsCase avgt 23.743 ± 0.105 ns/op CidrExistsCase - benchmarks the case matching CIDR exists in the tree for a given IP CidrNotExistsCase - benchmarks the case matching CIDR not exists in the tree for a given IP