Skip to content

Multi-cloud Kubernetes security controls

This outlines the recommended security controls for Kubernetes. It applies to on-premise Kubernetes deployments like OpenShift, as well as public-cloud managed Kubernetes services such as GKE, EKS, AKS. This provides the common configuration steps and as much as possible is agnostic of configuration tools such as Terraform, CloudFormation, etc.

Control plane components

These are the security recommendations for the direct configuration of the Kubernetes control plane processes.

Master node configuration files

Set API server pod specification file permissions to 644 or more restrictive

The API server pod specification file controls various parameters that govern behavior of the API server. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the following command (based on the file location on your system) on the master node.

stat -c %a /etc/kubernetes/manifests/kube-apiserver.yaml

Confirm the permissions are 644 or more restrictive.

Run the following command (based on the file location on your system) on the master node.

chmod 644 /etc/kubernetes/manifests/kube-apiserver.yaml

Set the API server pod specification file ownership to root:root

The API server pod specification file controls various parameters that govern behavior of the API server. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the following command (based on the file location on your system) on the master node.

stat -c %U:%G /etc/kubernetes/manifests/kube-apiserver.yaml

Confirm the ownership is set to root:root.

Run the following command (based on the file location on your system) on the master node.

chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml

Set the controller manager pod specification file permissions to 644 or more restrictive

The controller manager pod specification file controls various parameters that govern behavior of the Controller Manager on the master node. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the following command (based on the file location on your system) on the master node.

stat -c %a /etc/kubernetes/manifests/kube-controller-manager.yaml

Confirm the permissions are 644 or more restrictive.

Run the following command (based on the file location on your system) on the master node.

chmod 644 /etc/kubernetes/manifests/kube-controller-manager.yaml
Set the controller manager pod specification file ownership to root:root

The controller manager pod specification file controls various parameters that govern behavior of various components of the master node. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the following command (based on the file location on your system) on the master node.

stat -c %U:%G /etc/kubernetes/manifests/kube-controller-manager.yaml

Confirm the ownership is set to root:root.

Run the following command (based on the file location on your system) on the master node.

chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml
Set the scheduler pod specification file permissions to 644 or more restrictive

The scheduler pod specification file controls various parameters that govern behavior of the Scheduler service in the master node. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the following command (based on the file location on your system) on the master node.

stat -c %a /etc/kubernetes/manifests/kube-scheduler.yaml

Confirm the permissions are 644 or more restrictive.

Run the following command (based on the file location on your system) on the master node.

chmod 644 /etc/kubernetes/manifests/kube-scheduler.yaml
Set the scheduler pod specification file ownership to root:root

The scheduler pod specification file controls various parameters that govern behavior of the kube-scheduler service in the master node. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the following command (based on the file location on your system) on the master node.

stat -c %U:%G /etc/kubernetes/manifests/kube-scheduler.yaml

Confirm the ownership is set to root:root.

Run the following command (based on the file location on your system) on the master node.

chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml
Set the etcd pod specification file permissions to 644 or more restrictive

The etcd pod specification file /etc/kubernetes/manifests/etcd.yaml controls various parameters that govern behavior of the etcd service in the master node. etcd is a highly-available key-value store which Kubernetes uses for persistent storage of all of its REST API object. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the following command (based on the file location on your system) on the master node.

stat -c %a /etc/kubernetes/manifests/etcd.yaml

Confirm the permissions are 644 or more restrictive.

Run the following command (based on the file location on your system) on the master node.

chmod 644 /etc/kubernetes/manifests/etcd.yaml
Set the etcd pod specification file ownership to root:root

The etcd pod specification file /etc/kubernetes/manifests/etcd.yaml controls various parameters that govern behavior of the etcd service in the master node. etcd is a highly-available key-value store which Kubernetes uses for persistent storage of all of its REST API object. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the following command (based on the file location on your system) on the master node.

stat -c %U:%G /etc/kubernetes/manifests/etcd.yaml

Confirm the ownership is set to root:root.

Run the following command (based on the file location on your system) on the master node.

chown root:root /etc/kubernetes/manifests/etcd.yaml
Set the Container Network Interface file permissions to 644 or more restrictive

Container Network Interface provides various networking options for overlay networking. You should consult their documentation and restrict their respective file permissions to maintain the integrity of those files. Those files should be writable by only the administrators on the system.

Run the following command (based on the file location on your system) on the master node.

stat -c %a <path/to/cni/files>

Confirm the permissions are 644 or more restrictive.

Run the following command (based on the file location on your system) on the master node.

chmod 644 <path/to/cni/files>
Set the Container Network Interface file ownership to root:root

Container Network Interface provides various networking options for overlay networking. You should consult their documentation and restrict their respective file permissions to maintain the integrity of those files. Those files should be owned by root:root.

Run the following command (based on the file location on your system) on the master node.

stat -c %U:%G <path/to/cni/files>

Confirm the ownership is set to root:root.

Run the following command (based on the file location on your system) on the master node.

chown root:root <path/to/cni/files>
Set the etcd data directory permissions to 700 or more restrictive

etcd is a highly-available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. This data directory should be protected from any unauthorized reads or writes. It should not be readable or writable by any group members or the world.

On the etcd server node, get the etcd data directory, passed as an argument --data-dir, from the following command:

ps -ef | grep etcd

Run the following command (based on the etcd data directory found above).

stat -c %a /var/lib/etcd

Confirm the permissions are 700 or more restrictive.

On the etcd server node, get the etcd data directory, passed as an argument --data-dir, from the following command:

ps -ef | grep etcd

Run the following command (based on the etcd data directory found above).

chmod 700 /var/lib/etcd
Set the etcd data directory ownership to etcd:etcd

etcd is a highly-available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. This data directory should be protected from any unauthorized reads or writes. It should be owned by etcd:etcd.

On the etcd server node, get the etcd data directory, passed as an argument --data-dir, from the following command:

ps -ef | grep etcd

Run the following command (based on the etcd data directory found above).

stat -c %U:%G /var/lib/etcd

Confirm the ownership is set to etcd:etcd.

On the etcd server node, get the etcd data directory, passed as an argument --data-dir, from the following command:

ps -ef | grep etcd

Run the following command (based on the etcd data directory found above).

chown etcd:etcd /var/lib/etcd
Set the admin.conf file permissions to 644 or more restrictive

The admin.conf is the administrator kubeconfig file defining various settings for the administration of the cluster. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the following command (based on the file location on your system) on the master node.

stat -c %a /etc/kubernetes/admin.conf

Confirm the permissions are 644 or more restrictive.

Run the following command (based on the file location on your system) on the master node.

chmod 644 /etc/kubernetes/admin.conf
Set the admin.conf file ownership to root:root

The admin.conf file contains the admin credentials for the cluster. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the following command (based on the file location on your system) on the master node.

stat -c %U:%G /etc/kubernetes/admin.conf

Confirm the ownership is set to root:root.

Run the following command (based on the file location on your system) on the master node.

chown root:root /etc/kubernetes/admin.conf
Set the scheduler.conf file permissions to 644 or more restrictive

The scheduler.conf file is the kubeconfig file for the Scheduler. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the following command (based on the file location on your system) on the master node.

stat -c %a /etc/kubernetes/scheduler.conf

Confirm the permissions are 644 or more restrictive.

Run the following command (based on the file location on your system) on the master node.

chmod 644 /etc/kubernetes/scheduler.conf
Set the scheduler.conf file ownership to root:root

The scheduler.conf file is the kubeconfig file for the Scheduler. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the following command (based on the file location on your system) on the master node.

stat -c %U:%G /etc/kubernetes/scheduler.conf

Confirm the ownership is set to root:root.

Run the following command (based on the file location on your system) on the master node.

chown root:root /etc/kubernetes/scheduler.conf
Set the controller-manager.conf file permissions to 644 or more restrictive

The controller-manager.conf file is the kubeconfig file for the Controller Manager. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the following command (based on the file location on your system) on the master node.

stat -c %a /etc/kubernetes/controller-manager.conf

Confirm the permissions are 644 or more restrictive.

Run the following command (based on the file location on your system) on the master node.

chmod 644 /etc/kubernetes/controller-manager.conf
Set the controller-manager.conf file ownership to root:root

The controller-manager.conf file is the kubeconfig file for the Controller Manager. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the following command (based on the file location on your system) on the master node.

stat -c %U:%G /etc/kubernetes/controller-manager.conf

Confirm the ownership is set to root:root.

Run the following command (based on the file location on your system) on the master node.

chown root:root /etc/kubernetes/controller-manager.conf
Set the Kubernetes PKI directory and file ownership to root:root

Kubernetes makes use of a number of certificates as part of its operation. You should set the ownership of the directory containing the PKI information and all files in that directory to maintain their integrity. The directory and files should be owned by root:root.

Run the following command (based on the file location on your system) on the master node.

ls -laR /etc/kubernetes/pki/

Confirm the ownership of all files and directories in this hierarchy is set to root:root.

Run the following command (based on the file location on your system) on the master node.

chown -R `root:root` /etc/kubernetes/pki/
Set the Kubernetes PKI certificate file permissions to 644 or more restrictive

Kubernetes makes use of a number of certificate files as part of the operation of its components. The permissions on these files should be set to 644 or more restrictive to protect their integrity.

Run the following command (based on the file location on your system) on the master node.

ls -laR /etc/kubernetes/pki/*.crt

Confirm the permissions are 644 or more restrictive.

Run the following command (based on the file location on your system) on the master node.

chmod -R 644 /etc/kubernetes/pki/*.crt
Set the Kubernetes PKI key file permissions to 600

Kubernetes makes use of a number of key files as part of the operation of its components. The permissions on these files should be set to 600 to protect their integrity and confidentiality.

Run the following command (based on the file location on your system) on the master node.

ls -laR /etc/kubernetes/pki/*.key

Confirm the permissions are 600.

Run the following command (based on the file location on your system) on the master node.

chmod -R 600 /etc/kubernetes/pki/*.key

API server

Set the --anonymous-auth argument to false

When enabled, requests that are not rejected by other configured authentication methods are treated as anonymous requests. These requests are then served by the API server. You should rely on authentication to authorize access and disallow anonymous requests.

If you are using RBAC authorization, it is generally considered reasonable to allow anonymous access to the API Server for health checks and discovery purposes, and hence this recommendation is not scored. However, you should consider whether anonymous discovery is an acceptable risk for your purposes.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --anonymous-auth argument is set to false.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the below parameter.

--anonymous-auth=false
Ensure the --basic-auth-file argument is not set

Basic authentication uses plaintext credentials for authentication. Currently, the basic authentication credentials last indefinitely, and the password cannot be changed without restarting the API server. The basic authentication is currently supported for convenience. Hence, basic authentication should not be used.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --basic-auth-file argument does not exist.

Follow the documentation and configure alternate mechanisms for authentication. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and remove the --basic-auth-file=<filename> parameter.

Ensure the --token-auth-file parameter is not set

The token-based authentication utilizes static tokens to authenticate requests to the API server. The tokens are stored in clear-text in a file on the API server, and cannot be revoked or rotated without restarting the API server. Hence, do not use static token-based authentication.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --token-auth-file argument does not exist.

Follow the documentation and configure alternate mechanisms for authentication. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and remove the --token-auth-file=<filename> parameter.

Set the --kubelet-https argument to true

Connections from apiserver to kubelets could potentially carry sensitive data such as secrets and keys. It is thus important to use in-transit encryption for any communication between the API server and kubelets.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --kubelet-https argument either does not exist or is set to true.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and remove the --kubelet-https parameter.

Ensure the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate

The API server, by default, does not authenticate itself to the kubelet’s HTTPS endpoints. The requests from the API server are treated anonymously. You should set up certificate-based kubelet authentication to ensure the API server authenticates itself to kubelets when submitting requests.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --kubelet-client-certificate and --kubelet-client-key arguments exist and are set as appropriate.

Follow the Kubernetes documentation and set up the TLS connection between the API server and kubelets. Then, edit API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the kubelet client certificate and key parameters as below.

--kubelet-client-certificate=<path/to/client-certificate-file>
--kubelet-client-key=<path/to/client-key-file>
Ensure the --kubelet-certificate-authority argument is set as appropriate

The connections from the API server to the kubelet are used for fetching logs for pods, attaching (through kubectl) to running pods, and using the kubelet’s port-forwarding functionality. These connections terminate at the kubelet’s HTTPS endpoint. By default, the API server does not verify the kubelet’s serving certificate, which makes the connection subject to man-in-the-middle attacks, and unsafe to run over untrusted and/or public networks.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --kubelet-certificate-authority argument exists and is set as appropriate.

Follow the Kubernetes documentation and setup the TLS connection between the API server and kubelets. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the -- kubelet-certificate-authority parameter to the path to the cert file for the certificate authority.

--kubelet-certificate-authority=<ca-string>
Ensure the --authorization-mode argument is not set to AlwaysAllow

The API Server, can be configured to allow all requests. This mode should not be used on any production cluster.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --authorization-mode argument exists and is not set to AlwaysAllow.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --authorization-mode parameter to values other than AlwaysAllow. One such example could be as below.

--authorization-mode=RBAC
Ensure the --authorization-mode argument includes Node

The Node authorization mode only allows kubelets to read Secret, ConfigMap, PersistentVolume, and PersistentVolumeClaim objects associated with their nodes.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --authorization-mode argument exists and is set to a value to include Node.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --authorization-mode parameter to a value that includes Node.

--authorization-mode=Node,RBAC
Ensure the --authorization-mode argument includes RBAC

Role Based Access Control (RBAC) allows fine-grained control over the operations that different entities can perform on different objects in the cluster. It is recommended to use the RBAC authorization mode.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --authorization-mode argument exists and is set to a value to include RBAC.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --authorization-mode parameter to a value that includes RBAC, for example:

--authorization-mode=Node,RBAC
Ensure the admission control plugin EventRateLimit is set

Using EventRateLimit admission control enforces a limit on the number of events that the API Server will accept in a given time slice. A misbehaving workload could overwhelm and DoS the API Server, making it unavailable. This particularly applies to a multi-tenant cluster, where there might be a small percentage of misbehaving tenants which could have a significant impact on the performance of the cluster overall. Hence, it is recommended to limit the rate of events that the API server will accept.

Note: This is an Alpha feature in the Kubernetes 1.15 release.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --enable-admission-plugins argument is set to a value that includes EventRateLimit.

Follow the Kubernetes documentation and set the desired limits in a configuration file. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml and set the below parameters.

--enable-admission-plugins=...,EventRateLimit,...
--admission-control-config-file=<path/to/configuration/file>
Ensure the admission control plugin AlwaysAdmit is not set

Setting admission control plugin AlwaysAdmit allows all requests and do not filter any requests.

The AlwaysAdmit admission controller was deprecated in Kubernetes v1.13. Its behavior was equivalent to turning off all admission controllers.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm if the --enable-admission-plugins argument is set, its value does not include AlwaysAdmit.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and either remove the --enable-admission-plugins parameter, or set it to a value that does not include AlwaysAdmit.

Ensure the admission control plugin AlwaysPullImages is set

Setting admission control policy to AlwaysPullImages forces every new pod to pull the required images every time. In a multi-tenant cluster users can be assured that their private images can only be used by those who have the credentials to pull them. Without this admission control policy, once an image has been pulled to a node, any pod from any user can use it simply by knowing the image’s name, without any authorization check against the image ownership. When this plug-in is enabled, images are always pulled prior to starting containers, which means valid credentials are required.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --enable-admission-plugins argument is set to a value that includes AlwaysPullImages.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --enable-admission-plugins parameter to include AlwaysPullImages.

--enable-admission-plugins=...,AlwaysPullImages,...
Ensure the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used

The SecurityContextDeny admission controller can be used to deny pods which make use of some SecurityContext fields which could allow for privilege escalation in the cluster. This should be used where PodSecurityPolicy is not in place within the cluster.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --enable-admission-plugins argument is set to a value that includes SecurityContextDeny, if PodSecurityPolicy is not included.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --enable-admission-plugins parameter to include SecurityContextDeny, unless PodSecurityPolicy is already in place.

--enable-admission-plugins=...,SecurityContextDeny,...
Ensure the admission control plugin ServiceAccount is set

When you create a pod, if you do not specify a service account, it is automatically assigned the default service account in the same namespace. You should create your own service account and let the API server manage its security tokens.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --disable-admission-plugins argument is set to a value that does not includes ServiceAccount.

Follow the documentation and create ServiceAccount objects as per your environment. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and ensure that the --disable-admission-plugins parameter is set to a value that does not include ServiceAccount.

Ensure the admission control plugin NamespaceLifecycle is set

Setting admission control policy to NamespaceLifecycle ensures that objects cannot be created in non-existent namespaces, and that namespaces undergoing termination are not used for creating the new objects. This is recommended to enforce the integrity of the namespace termination process and also for the availability of the newer objects.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --disable-admission-plugins argument is set to a value that does not include NamespaceLifecycle.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --disable-admission-plugins parameter to ensure it does not include NamespaceLifecycle.

Ensure the admission control plugin PodSecurityPolicy is set

A Pod Security Policy is a cluster-level resource that controls the actions that a pod can perform and what it has the ability to access. The PodSecurityPolicy objects define a set of conditions that a pod must run with in order to be accepted into the system. Pod Security Policies are comprised of settings and strategies that control the security features a pod has access to and hence this must be used to control pod access permissions.

Note: When the PodSecurityPolicy admission plugin is in use, there needs to be at least one PodSecurityPolicy in place for ANY pods to be admitted.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --enable-admission-plugins argument is set to a value that includes PodSecurityPolicy.

Follow the documentation and create Pod Security Policy objects as per your environment. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --enable-admission-plugins parameter to a value that includes PodSecurityPolicy:

--enable-admission-plugins=...,PodSecurityPolicy,...

Then restart the API Server.

The policy objects must be created and granted before pod creation would be allowed.

Ensure the admission control plugin NodeRestriction is set

Using the NodeRestriction plug-in ensures that the kubelet is restricted to the Node and Pod objects that it could modify as defined. Such kubelets will only be allowed to modify their own Node API object, and only modify Pod API objects that are bound to their node.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --enable-admission-plugins argument is set to a value that includes NodeRestriction.

Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --enable-admission-plugins parameter to a value that includes NodeRestriction.

--enable-admission-plugins=...,NodeRestriction,...
Ensure the --insecure-bind-address argument is not set

If you bind the API server to an insecure address, basically anyone who could connect to it over the insecure port, would have unauthenticated and unencrypted access to your master node. The API server doesn't do any authentication checking for insecure binds and traffic to the Insecure API port is not encrpyted, allowing attackers to potentially read sensitive data in transit.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --insecure-bind-address argument does not exist.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and remove the --insecure-bind-address parameter.

Set the --insecure-port argument to 0

Setting up the API server to serve on an insecure port would allow unauthenticated and unencrypted access to your master node. This would allow attackers who could access this port, to easily take control of the cluster.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --insecure-port argument is set to 0.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the below parameter.

--insecure-port=0
Ensure the --secure-port argument is not set to 0

The secure port is used to serve https with authentication and authorization. If you disable it, no https traffic is served and all traffic is served unencrypted.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --secure-port argument is either not set or is set to an integer value between 1 and 65535.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and either remove the --secure-port parameter or set it to a different (non-zero) desired port.

Set the --profiling argument to false

Profiling allows for the identification of specific performance bottlenecks. It generates a significant amount of program data that could potentially be exploited to uncover system and program details. If you are not experiencing any bottlenecks and do not need the profiler for troubleshooting purposes, it is recommended to turn it off to reduce the potential attack surface.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --profiling argument is set to false.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the below parameter.

--profiling=false
Ensure the --audit-log-path argument is set

Auditing the Kubernetes API Server provides a security-relevant chronological set of records documenting the sequence of activities that have affected system by individual users, administrators or other components of the system. Even though currently, Kubernetes provides only basic audit capabilities, it should be enabled. You can enable it by setting an appropriate audit log path.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --audit-log-path argument is set as appropriate.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --audit-log-path parameter to a suitable path and file where you would like audit logs to be written, for example:

--audit-log-path=/var/log/apiserver/audit.log
Set the --audit-log-maxage argument to 30 or as appropriate

Retaining logs for at least 30 days ensures that you can go back in time and investigate or correlate any events. Set your audit log retention period to 30 days or as per your business requirements.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --audit-log-maxage argument is set to 30 or as appropriate.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --audit-log-maxage parameter to 30 or as an appropriate number of days:

--audit-log-maxage=30
Set the --audit-log-maxbackup argument to 10 or as appropriate

Kubernetes automatically rotates the log files. Retaining old log files ensures that you would have sufficient log data available for carrying out any investigation or correlation. For example, if you have set file size of 100 MB and the number of old log files to keep as 10, you would have approximately 1 GB of log data that you could potentially use for your analysis.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --audit-log-maxbackup argument is set to 10 or as appropriate.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --audit-log-maxbackup parameter to 10 or to an appropriate value.

--audit-log-maxbackup=10
Set the --audit-log-maxsize argument to 100 or as appropriate

Kubernetes automatically rotates the log files. Retaining old log files ensures that you would have sufficient log data available for carrying out any investigation or correlation. If you have set file size of 100 MB and the number of old log files to keep as 10, you would approximate have 1 GB of log data that you could potentially use for your analysis.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --audit-log-maxsize argument is set to 100 or as appropriate.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --audit-log-maxsize parameter to an appropriate size in MB. For example, to set it as 100 MB:

--audit-log-maxsize=100
Ensure the --request-timeout argument is set as appropriate

Setting global request timeout allows extending the API server request timeout limit to a duration appropriate to the user's connection speed. By default, it is set to 60 seconds which might be problematic on slower connections making cluster resources inaccessible once the data volume for requests exceeds what can be transmitted in 60 seconds. But, setting this timeout limit to be too large can exhaust the API server resources making it prone to Denial-of-Service attack. Hence, it is recommended to set this limit as appropriate and change the default limit of 60 seconds only if needed.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --request-timeout argument is either not set or set to an appropriate value.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml and set the below parameter as appropriate and if needed. For example,

--request-timeout=300s
Set the --service-account-lookup argument to true

If --service-account-lookup is not enabled, the API server only verifies that the authentication token is valid, and does not validate that the service account token mentioned in the request is actually present in etcd. This allows using a service account token even after the corresponding service account is deleted. This is an example of time of check to time of use security issue.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm if the --service-account-lookup argument exists it is set to true.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the below parameter.

--service-account-lookup=true

Alternatively, you can delete the --service-account-lookup parameter from this file so that the default takes effect.

Ensure the --service-account-key-file argument is set as appropriate

By default, if no --service-account-key-file is specified to the API server, it uses the private key from the TLS serving certificate to verify service account tokens. To ensure that the keys for service account tokens could be rotated as needed, a separate public/private key pair should be used for signing service account tokens. Hence, the public key should be specified to the API server with --service-account-key-file.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --service-account-key-file argument exists and is set as appropriate.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --service-account-key-file parameter to the public key file for service accounts:

--service-account-key-file=<filename>
Ensure the --etcd-certfile and --etcd-keyfile arguments are set as appropriate

etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be protected by client authentication. This requires the API server to identify itself to the etcd server using a client certificate and key.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --etcd-certfile and --etcd-keyfile arguments exist and they are set as appropriate.

Follow the Kubernetes documentation and set up the TLS connection between the API server and etcd. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the etcd certificate and key file parameters.

--etcd-certfile=<path/to/client-certificate-file>
--etcd-keyfile=<path/to/client-key-file>
Ensure the --tls-cert-file and --tls-private-key-file arguments are set as appropriate

API server communication contains sensitive parameters that should remain encrypted in transit. Configure the API server to serve only HTTPS traffic.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --tls-cert-file and --tls-private-key-file arguments exist and they are set as appropriate.

Follow the Kubernetes documentation and set up the TLS connection on the API server. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the TLS certificate and private key file parameters.

--tls-cert-file=<path/to/tls-certificate-file>
--tls-private-key-file=<path/to/tls-key-file>
Ensure the --client-ca-file argument is set as appropriate

API server communication contains sensitive parameters that should remain encrypted in transit. Configure the API server to serve only HTTPS traffic. If --client-ca-file argument is set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --client-ca-file argument exists and it is set as appropriate.

Follow the Kubernetes documentation and set up the TLS connection on the API server. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the client certificate authority file.

--client-ca-file=<path/to/client-ca-file>
Ensure the --etcd-cafile argument is set as appropriate

etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be protected by client authentication. This requires the API server to identify itself to the etcd server using a SSL Certificate Authority file.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --etcd-cafile argument exists and it is set as appropriate.

Follow the Kubernetes documentation and set up the TLS connection between the API server and etcd. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the etcd certificate authority file parameter.

--etcd-cafile=<path/to/ca-file>
Ensure the --encryption-provider-config argument is set as appropriate

etcd is a highly available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be encrypted at rest to avoid any disclosures.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --encryption-provider-config argument is set to a EncryptionConfig file. Additionally, ensure that the EncryptionConfig file has all the desired resources covered especially any secrets.

Follow the Kubernetes documentation and configure a EncryptionConfig file. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --encryption-provider-config parameter to the path of that file:

--encryption-provider-config=</path/to/EncryptionConfig/File>
Ensure encryption providers are appropriately configured

Where etcd encryption is used, it is important to ensure that the appropriate set of encryption providers is used. Currently, the aescbc, kms and secretbox are likely to be appropriate options.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Get the EncryptionConfig file set for --encryption-provider-config argument. Confirm aescbc, kms or secretbox is set as the encryption provider for all the desired resources.

Follow the Kubernetes documentation and configure a EncryptionConfig file. In this file, choose aescbc, kms or secretbox as the encryption provider.

Ensure the API Server only makes use of Strong Cryptographic Ciphers

TLS ciphers have had a number of known vulnerabilities and weaknesses, which can reduce the protection provided by them. By default Kubernetes supports a number of TLS ciphersuites including some that have security concerns, weakening the protection provided.

Run the following command on the master node:

ps -ef | grep kube-apiserver

Confirm the --tls-cipher-suites argument is set as outlined in the remediation procedure below.

Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the below parameter.

--tls-cipher-
suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM
_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM
_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM
_SHA384

Controller manager

Ensure the --terminated-pod-gc-threshold argument is set as appropriate

Garbage collection is important to ensure sufficient resource availability and avoiding degraded performance and availability. In the worst case, the system might crash or just be unusable for a long period of time. The current setting for garbage collection is 12,500 terminated pods which might be too high for your system to sustain. Based on your system resources and tests, choose an appropriate threshold value to activate garbage collection.

Run the following command on the master node:

ps -ef | grep kube-controller-manager

Confirm the --terminated-pod-gc-threshold argument is set as appropriate.

Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube- controller-manager.yaml on the master node and set the --terminated-pod-gc- threshold to an appropriate threshold, for example:

--terminated-pod-gc-threshold=10
Set the --profiling argument to false

Profiling allows for the identification of specific performance bottlenecks. It generates a significant amount of program data that could potentially be exploited to uncover system and program details. If you are not experiencing any bottlenecks and do not need the profiler for troubleshooting purposes, it is recommended to turn it off to reduce the potential attack surface.

Run the following command on the master node:

ps -ef | grep kube-controller-manager

Confirm the --profiling argument is set to false.

Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube- controller-manager.yaml on the master node and set the below parameter.

--profiling=false
Set the --use-service-account-credentials argument to true

The controller manager creates a service account per controller in the kube-system namespace, generates a credential for it, and builds a dedicated API client with that service account credential for each controller loop to use. Setting the --use-service-account- credentials to true runs each control loop within the controller manager using a separate service account credential. When used in combination with RBAC, this ensures that the control loops run with the minimum permissions required to perform their intended tasks.

Run the following command on the master node:

ps -ef | grep kube-controller-manager

Confirm the --use-service-account-credentials argument is set to true.

Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube- controller-manager.yaml on the master node to set the below parameter.

--use-service-account-credentials=true
Ensure the --service-account-private-key-file argument is set as appropriate

To ensure that keys for service account tokens can be rotated as needed, a separate public/private key pair should be used for signing service account tokens. The private key should be specified to the controller manager with --service-account-private-key-file as appropriate.

Run the following command on the master node:

ps -ef | grep kube-controller-manager

Confirm the --service-account-private-key-file argument is set as appropriate.

Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube- controller-manager.yaml on the master node and set the --service-account-private- key-file parameter to the private key file for service accounts.

--service-account-private-key-file=<filename>
Ensure the --root-ca-file argument is set as appropriate

Processes running within pods that need to contact the API server must verify the API server's serving certificate. Failing to do so could be a subject to man-in-the-middle attacks.

Providing the root certificate for the API server's serving certificate to the controller manager with the `--root-ca-file` argument allows the controller manager to inject the trusted bundle into pods so that they can verify TLS connections to the API server.

Run the following command on the master node:

ps -ef | grep kube-controller-manager

Confirm the --root-ca-file argument exists and is set to a certificate bundle file containing the root certificate for the API server's serving certificate.

Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube- controller-manager.yaml on the master node and set the --root-ca-file parameter to the certificate bundle file`.

--root-ca-file=<path/to/file>
Set the RotateKubeletServerCertificate argument to true

RotateKubeletServerCertificate causes the kubelet to both request a serving certificate after bootstrapping its client credentials and rotate the certificate as its existing credentials expire. This automated periodic rotation ensures that the there are no downtimes due to expired certificates and thus addressing availability in the CIA security triad.

Note: This recommendation only applies if you let kubelets get their certificates from the API server. In case your kubelet certificates come from an outside authority/tool (e.g. Vault) then you need to take care of rotation yourself.

Run the following command on the master node:

ps -ef | grep kube-controller-manager

Confirm RotateKubeletServerCertificate argument exists and is set to true.

Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube- controller-manager.yaml on the master node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true.

--feature-gates=RotateKubeletServerCertificate=true
Set the --bind-address argument to 127.0.0.1

The Controller Manager API service which runs on port 10252/TCP by default is used for health and metrics information and is available without authentication or encryption. As such it should only be bound to a localhost interface, to minimize the cluster's attack surface.

Run the following command on the master node:

ps -ef | grep kube-controller-manager

Confirm the --bind-address argument is set to 127.0.0.1

Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube- controller-manager.yaml on the master node and ensure the correct value for the -- bind-address parameter.

Scheduler

Set the --profiling argument to false

Profiling allows for the identification of specific performance bottlenecks. It generates a significant amount of program data that could potentially be exploited to uncover system and program details. If you are not experiencing any bottlenecks and do not need the profiler for troubleshooting purposes, it is recommended to turn it off to reduce the potential attack surface.

Run the following command on the master node:

ps -ef | grep kube-scheduler

Confirm the --profiling argument is set to false.

Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file on the master node and set the below parameter.

--profiling=false
Set the --bind-address argument to 127.0.0.1

The Scheduler API service which runs on port 10251/TCP by default is used for health and metrics information and is available without authentication or encryption. As such it should only be bound to a localhost interface, to minimize the cluster's attack surface.

Run the following command on the master node:

ps -ef | grep kube-scheduler

Confirm the --bind-address argument is set to 127.0.0.1

Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml on the master node and ensure the correct value for the --bind-address parameter.

etcd

Ensure the --cert-file and --key-file arguments are set as appropriate

etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be encrypted in transit.

Run the following command on the etcd server node

ps -ef | grep etcd

Confirm the --cert-file and the --key-file arguments are set as appropriate.

Follow the etcd service documentation and configure TLS encryption. Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master node and set the below parameters.

--cert-file=</path/to/ca-file>
--key-file=</path/to/key-file>
Set the --client-cert-auth argument to true

etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should not be available to unauthenticated clients. You should enable the client authentication via valid certificates to secure the access to the etcd service.

Run the following command on the etcd server node:

ps -ef | grep etcd

Confirm the --client-cert-auth argument is set to true.

Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master node and set the below parameter.

--client-cert-auth="true"
Ensure the --auto-tls argument is not set to true

etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should not be available to unauthenticated clients. You should enable the client authentication via valid certificates to secure the access to the etcd service.

Run the following command on the etcd server node:

ps -ef | grep etcd

Confirm if the --auto-tls argument exists, it is not set to true.

Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master node and either remove the --auto-tls parameter or set it to false.

--auto-tls=false
Ensure the --peer-cert-file and --peer-key-file arguments are set as appropriate

etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be encrypted in transit and also amongst peers in the etcd clusters.

Run the following command on the etcd server node:

ps -ef | grep etcd

Confirm the --peer-cert-file and --peer-key-file arguments are set as appropriate.

Note: This recommendation is applicable only for etcd clusters. If you are using only one etcd server in your environment then this recommendation is not applicable.

Follow the etcd service documentation and configure peer TLS encryption as appropriate for your etcd cluster.

Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master node and set the below parameters.

--peer-client-file=</path/to/peer-cert-file>
--peer-key-file=</path/to/peer-key-file>
Set the --peer-client-cert-auth argument to true

etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be accessible only by authenticated etcd peers in the etcd cluster.

Run the following command on the etcd server node:

ps -ef | grep etcd

Confirm the --peer-client-cert-auth argument is set to true.

Note: This recommendation is applicable only for etcd clusters. If you are using only one etcd server in your environment then this recommendation is not applicable.

Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master node and set the below parameter.

--peer-client-cert-auth=true
Ensure the --peer-auto-tls argument is not set to true

etcd is a highly-available key value store used by Kubernetes deployments for persistent storage of all of its REST API objects. These objects are sensitive in nature and should be accessible only by authenticated etcd peers in the etcd cluster. Hence, do not use self- signed certificates for authentication.

Run the following command on the etcd server node:

ps -ef | grep etcd

Confirm if the --peer-auto-tls argument exists, it is not set to true.

Note: This recommendation is applicable only for etcd clusters. If you are using only one etcd server in your environment then this recommendation is not applicable.

Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master node and either remove the --peer-auto-tls parameter or set it to false.

--peer-auto-tls=false
Ensure a unique Certificate Authority is used for etcd

etcd is a highly available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. Its access should be restricted to specifically designated clients and peers only.

Authentication to etcd is based on whether the certificate presented was issued by a trusted certificate authority. There is no checking of certificate attributes such as common name or subject alternative name. As such, if any attackers were able to gain access to any certificate issued by the trusted certificate authority, they would be able to gain full access to the etcd database.

Review the CA used by the etcd environment and ensure that it does not match the CA certificate file used for the management of the overall Kubernetes cluster. Run the following command on the master node:

ps -ef | grep etcd

Note the file referenced by the --trusted-ca-file argument. Run the following command on the master node:

ps -ef | grep apiserver

Confirm the file referenced by the --client-ca-file for apiserver is different from the - -trusted-ca-file used by etcd.

Follow the etcd documentation and create a dedicated certificate authority setup for the etcd service.

Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master node and set the below parameter.

--trusted-ca-file=</path/to/ca-file>

Control plane configuration

Authentication and authorization

Client certificate authentication should not be used for users

Kubernetes provides the option to use client certificates for user authentication. However as there is no way to revoke these certificates when a user leaves an organization or loses their credential, they are not suitable for this purpose.

It is not possible to fully disable client certificate use within a cluster as it is used for component to component authentication.

With any authentication mechanism the ability to revoke credentials if they are compromised or no longer required, is a key control. Kubernetes client certificate authentication does not allow for this due to a lack of support for certificate revocation.

Review user access to the cluster and ensure that users are not making use of Kubernetes client certificate authentication.

Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented in place of client certificates.

Logging

Ensure a minimal audit policy is created

Kubernetes can audit the details of requests made to the API server. the --audit-policy- file flag must be set for this logging to be enabled.

Run the following command on one of the cluster master nodes:

ps -ef | grep kube-apiserver

Confirm the --audit-policy-file is set. Review the contents of the file specified and ensure that it contains a valid audit policy.

Create an audit policy file for your cluster.

Ensure the audit policy covers key security concerns

Security audit logs should cover access and modification of key resources in the cluster, to enable them to form an effective part of a security environment.

Review the audit policy provided for the cluster and ensure that it covers at least the following areas :

  • Access to Secrets managed by the cluster. Care should be taken to only log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in order to avoid the risk of logging sensitive data.

  • Modification of pod and deployment objects.

  • Use of pods/exec, pods/portforward, pods/proxy and services/proxy.

For most requests, minimally logging at the Metadata level is recommended (the most basic level of logging).

Consider modification of the audit policy in use on the cluster to include these items, at a minimum.

Work nodes

Worker node configuration files

Set the kubelet service file permissions to 644 or more restrictive

The kubelet service file controls various parameters that govern behavior of the kubelet service in the worker node. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the following command (based on the file location on your system) on the each worker node.

stat -c %a /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Confirm the permissions are 644 or more restrictive.

Run the following command (based on the file location on your system) on the each worker node.

chmod 755 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Set the kubelet service file ownership to root:root

The kubelet service file controls various parameters that govern behavior of the kubelet service in the worker node. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the following command (based on the file location on your system) on the each worker node.

stat -c %a /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Confirm the ownership is set to root:root.

Run the following command (based on the file location on your system) on the each worker node.

chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive

The kube-proxy kubeconfig file controls various parameters of the kube-proxy service in the worker node. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

It is possible to run kube-proxy with the kubeconfig parameters configured as a Kubernetes ConfigMap instead of a file. In this case, there is no proxy kubeconfig file.

Find the kubeconfig file being used by kube-proxy by running the following command:

ps -ef | grep kube-proxy

If kube-proxy is running, get the kubeconfig file location from the --kubeconfig parameter.

Run the following command (based on the file location on your system) on the each worker node.

stat -c %a

Confirm a file is specified and it exists with permissions are 644 or more restrictive.

Run the following command (based on the file location on your system) on the each worker node.

chmod 644 <proxy kubeconfig file>
If proxy kubeconfig file exists ensure ownership is set to root:root

The kubeconfig file for kube-proxy controls various parameters for the kube-proxy service in the worker node. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Find the kubeconfig file being used by kube-proxy by running the following command:

ps -ef | grep kube-proxy

If kube-proxy is running, get the kubeconfig file location from the --kubeconfig parameter.

Run the following command (based on the file location on your system) on the each worker node.

stat -c %a

Confirm the ownership is set to root:root.

Run the following command (based on the file location on your system) on the each worker node.

chown root:root <proxy kubeconfig file>
Set the --kubeconfig kubelet.conf file permissions to 644 or more restrictive

The kubelet.conf file is the kubeconfig file for the node, and controls various parameters that govern behavior and identity of the worker node. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the following command (based on the file location on your system) on the each worker node.

stat -c %a /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Confirm the ownership is set to root:root.

Confirm the permissions are 644 or more restrictive.

Run the following command (based on the file location on your system) on the each worker node.

chmod 644 /etc/kubernetes/kubelet.conf
Set the --kubeconfig kubelet.conf file ownership to root:root

The kubelet.conf file is the kubeconfig file for the node, and controls various parameters that govern behavior and identity of the worker node. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the following command (based on the file location on your system) on the each worker node.

stat -c %a /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Confirm the ownership is set to root:root.

Run the following command (based on the file location on your system) on the each worker node.

chown root:root /etc/kubernetes/kubelet.conf
Set the certificate authorities file permissions to 644 or more restrictive

The certificate authorities file controls the authorities used to validate API requests. You should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the following command:

ps -ef | grep kubelet

Find the file specified by the --client-ca-file argument.

Run the following command:

stat -c %a <filename>

Confirm the permissions are 644 or more restrictive.

Run the following command to modify the file permissions of the --client-ca-file

chmod 644 <filename>
Set the client certificate authorities file ownership to root:root

The certificate authorities file controls the authorities used to validate API requests. You should set its file ownership to maintain the integrity of the file. The file should be owned by root:root.

Run the following command:

ps -ef | grep kubelet

Find the file specified by the --client-ca-file argument.

Run the following command:

stat -c %U:%G <filename>

Confirm the ownership is set to root:root.

Run the following command to modify the ownership of the --client-ca-file.

chown root:root <filename>
Ensure the kubelet --config configuration file has permissions set to 644 or more restrictive

The kubelet reads various parameters, including security settings, from a config file specified by the --config argument. If this file is specified you should restrict its file permissions to maintain the integrity of the file. The file should be writable by only the administrators on the system.

Run the following command (based on the file location on your system) on the each worker node.

stat -c %a /var/lib/kubelet/config.yaml

Confirm the permissions are 644 or more restrictive.

Run the following command (using the config file location identied in the Audit step)

chmod 644 /var/lib/kubelet/config.yaml
Set the kubelet --config configuration file ownership to root:root

The kubelet reads various parameters, including security settings, from a config file specified by the --config argument. If this file is specified you should restrict its file permissions to maintain the integrity of the file. The file should be owned by root:root.

Run the following command (based on the file location on your system) on the each worker node.

stat -c %a /var/lib/kubelet/config.yaml

Confirm the ownership is set to root:root.

Run the following command:

chown root:root /etc/kubernetes/kubelet.conf

Kublet

Set the --anonymous-auth argument to false

When enabled, requests that are not rejected by other configured authentication methods are treated as anonymous requests. These requests are then served by the Kubelet server. You should rely on authentication to authorize access and disallow anonymous requests.

If using a Kubelet configuration file, check that there is an entry for authentication: anonymous: enabled set to false.

Run the following command on each node:

ps -ef | grep kubelet

Confirm the --anonymous-auth argument is set to false.

This executable argument may be omitted, provided there is a corresponding entry set to false in the Kubelet config file.

If using a Kubelet config file, edit the file to set authentication: anonymous: enabled to false.

If using executable arguments, edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable.

--anonymous-auth=false

Based on your system, restart the kubelet service. For example:

systemctl daemon-reload
systemctl restart kubelet.service
Ensure the --authorization-mode argument is not set to AlwaysAllow

Kubelets, by default, allow all authenticated requests (even anonymous ones) without needing explicit authorization checks from the API server. You should restrict this behavior and only allow explicitly authorized requests.

Run the following command on each node:

ps -ef | grep kubelet

If the --authorization-mode argument is present check that it is not set to AlwaysAllow. If it is not present check that there is a Kubelet config file specified by --config, and that file sets authorization: mode to something other than AlwaysAllow.

It is also possible to review the running configuration of a Kubelet via the /configz endpoint on the Kubelet API port (typically 10250/TCP). Accessing these with appropriate credentials will provide details of the Kubelet's configuration.

If using a Kubelet config file, edit the file to set authorization: mode to Webhook. If using executable arguments, edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and set the below parameter in KUBELET_AUTHZ_ARGS variable.

--authorization-mode=Webhook

Based on your system, restart the kubelet service. For example:

systemctl daemon-reload
systemctl restart kubelet.service
Ensure the --client-ca-file argument is set as appropriate

The connections from the API server to the kubelet are used for fetching logs for pods, attaching (through kubectl) to running pods, and using the kubelet’s port-forwarding functionality. These connections terminate at the kubelet’s HTTPS endpoint. By default, the API server does not verify the kubelet’s serving certificate, which makes the connection subject to man-in-the-middle attacks, and unsafe to run over untrusted and/or public networks. Enabling Kubelet certificate authentication ensures that the API server could authenticate the Kubelet before submitting any requests.

Run the following command on each node:

ps -ef | grep kubelet

Confirm the --client-ca-file argument exists and is set to the location of the client certificate authority file.

If the --client-ca-file argument is not present, check that there is a Kubelet config file specified by --config, and that the file sets authentication: x509: clientCAFile to the location of the client certificate authority file.

If using a Kubelet config file, edit the file to set authentication: x509: clientCAFile to the location of the client CA file.

If using command line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and set the below parameter in KUBELET_AUTHZ_ARGS variable.

--client-ca-file=<path/to/client-ca-file>

Based on your system, restart the kubelet service. For example:

systemctl daemon-reload
systemctl restart kubelet.service
Confirm the --read-only-port argument is set to 0

The Kubelet process provides a read-only API in addition to the main Kubelet API. Unauthenticated access is provided to this read-only API which could possibly retrieve potentially sensitive information about the cluster.

Run the following command on each node:

ps -ef | grep kubelet

Confirm the --read-only-port argument exists and is set to 0.

If the --read-only-port argument is not present, check that there is a Kubelet config file specified by --config.

Check that if there is a readOnlyPort entry in the file, it is set to 0.

If using a Kubelet config file, edit the file to set readOnlyPort to 0.

If using command line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable.

--read-only-port=0

Based on your system, restart the kubelet service. For example:

systemctl daemon-reload
systemctl restart kubelet.service
Ensure the --streaming-connection-idle-timeout argument is not set to 0

Setting idle timeouts ensures that you are protected against Denial-of-Service attacks, inactive connections and running out of ephemeral ports.

Note: By default, --streaming-connection-idle-timeout is set to 4 hours which might be too high for your environment. Setting this as appropriate would additionally ensure that such streaming connections are timed out after serving legitimate use cases.

Run the following command on each node:

ps -ef | grep kubelet

Confirm the --streaming-connection-idle-timeout argument is not set to 0.

If the argument is not present, and there is a Kubelet config file specified by --config, check that it does not set streamingConnectionIdleTimeout to 0.

If using a Kubelet config file, edit the file to set streamingConnectionIdleTimeout to a value other than 0.

If using command line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable.

--streaming-connection-idle-timeout=5m

Based on your system, restart the kubelet service. For example:

systemctl daemon-reload
systemctl restart kubelet.service
Set the --protect-kernel-defaults argument to true

Kernel parameters are usually tuned and hardened by the system administrators before putting the systems into production. These parameters protect the kernel and the system. Your kubelet kernel defaults that rely on such parameters should be appropriately set to match the desired secured system state. Ignoring this could potentially lead to running pods with undesired kernel behavior.

Run the following command on each node:

ps -ef | grep kubelet

Confirm the --protect-kernel-defaults argument is set to true.

If the --protect-kernel-defaults argument is not present, check that there is a Kubelet config file specified by --config, and that the file sets protectKernelDefaults to true.

If using a Kubelet config file, edit the file to set protectKernelDefaults: true.

If using command line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable.

--protect-kernel-defaults=true

Based on your system, restart the kubelet service. For example:

systemctl daemon-reload
systemctl restart kubelet.service
Set the --make-iptables-util-chains argument to true

Kubelets can automatically manage the required changes to iptables based on how you choose your networking options for the pods. It is recommended to let kubelets manage the changes to iptables. This ensures that the iptables configuration remains in sync with pods networking configuration. Manually configuring iptables with dynamic pod network configuration changes might hamper the communication between pods/containers and to the outside world. You might have iptables rules too restrictive or too open.

Run the following command on each node:

ps -ef | grep kubelet

Confirm if the --make-iptables-util-chains argument exists then it is set to true.

If the --make-iptables-util-chains argument does not exist, and there is a Kubelet config file specified by --config, confirm the file does not set makeIPTablesUtilChains to false.

If using a Kubelet config file, edit the file to set makeIPTablesUtilChains true. If using command-line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and remove the --make-iptables-util-chains argument from the KUBELET_SYSTEM_PODS_ARGS variable.

Based on your system, restart the kubelet service. For example:

systemctl daemon-reload
systemctl restart kubelet.service
Ensure the --hostname-override argument is not set

Overriding hostnames could potentially break TLS setup between the kubelet and the API server. Additionally, with overridden hostnames, it becomes increasingly difficult to associate logs with a particular node and process them for security analytics. Hence, you should setup your kubelet nodes with resolvable FQDNs and avoid overriding the hostnames with IPs.

Run the following command on each node:

ps -ef | grep kubelet

Confirm --hostname-override argument does not exist.

Note This setting is not configurable via the Kubelet config file.

Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and remove the --hostname-override argument from the KUBELET_SYSTEM_PODS_ARGS variable.

Based on your system, restart the kubelet service. For example:

systemctl daemon-reload
systemctl restart kubelet.service
Set the --event-qps argument to 0 or a level which ensures appropriate event capture

Security relevant information should be captured. the --event-qps flag on the Kubelet can be used to limit the rate at which events are gathered. Setting this too low could result in relevant events not being logged, however the unlimited setting of 0 could result in a denial of service on the kubelet.

It is important to capture all events and not restrict event creation. Events are an important source of security information and analytics that ensure that your environment is consistently monitored using the event data.

Run the following command on each node:

ps -ef | grep kubelet

Review the value set for the --event-qps argument and determine whether this has been set to an appropriate level for the cluster. The value of 0 can be used to ensure that all events are captured.

If the --event-qps argument does not exist, check that there is a Kubelet config file specified by --config and review the value in this location.

If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level. If using command-line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable.

Based on your system, restart the kubelet service. For example:

systemctl daemon-reload
systemctl restart kubelet.service
Ensure the --tls-cert-file and --tls-private-key-file arguments are set as appropriate

The connections from the API server to the kubelet are used for fetching logs for pods, attaching (through kubectl) to running pods, and using the kubelet’s port-forwarding functionality. These connections terminate at the kubelet’s HTTPS endpoint. By default, the API server does not verify the kubelet’s serving certificate, which makes the connection subject to man-in-the-middle attacks, and unsafe to run over untrusted and/or public networks.

Run the following command on each node:

ps -ef | grep kubelet

Confirm the --tls-cert-file and --tls-private-key-file arguments exist and they are set as appropriate.

If these arguments are not present, check that there is a Kubelet config specified by --config and that it contains appropriate settings for tlsCertFile and tlsPrivateKeyFile.

If using a Kubelet config file, edit the file to set tlsCertFile to the location of the certificate file to use to identify this Kubelet, and tlsPrivateKeyFile to the location of the corresponding private key file.

If using command line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and set the below parameters in KUBELET_CERTIFICATE_ARGS variable.

--tls-cert-file=<path/to/tls-certificate-file> --tls-private-key-file=<path/to/tls-key-file>

Based on your system, restart the kubelet service. For example:

systemctl daemon-reload
systemctl restart kubelet.service
Ensure the --rotate-certificates argument is not set to false

The --rotate-certificates setting causes the kubelet to rotate its client certificates by creating new CSRs as its existing credentials expire. This automated periodic rotation ensures that the there is no downtime due to expired certificates and thus addressing availability in the CIA security triad.

This recommendation only applies if you let kubelets get their certificates from the API server. In case your kubelet certificates come from an outside authority/tool (e.g. Vault) then you need to take care of rotation yourself.

This feature also require the RotateKubeletClientCertificate feature gate to be enabled (which is the default since Kubernetes v1.7)

Run the following command on each node:

ps -ef | grep kubelet

Confirm the --rotate-certificates argument is not present, or is set to true.

If the --rotate-certificates argument is not present, confirm if there is a Kubelet config file specified by --config, that file does not contain rotateCertificates: false.

If using a Kubelet config file, edit the file to add the line rotateCertificates: true or remove it altogether to use the default value.

If using command-line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS variable.

Based on your system, restart the kubelet service. For example:

systemctl daemon-reload
systemctl restart kubelet.service
Confirm the RotateKubeletServerCertificate argument is set to true

RotateKubeletServerCertificate causes the kubelet to both request a serving certificate after bootstrapping its client credentials and rotate the certificate as its existing credentials expire. This automated periodic rotation ensures that the there are no downtimes due to expired certificates and thus addressing availability in the CIA security triad.

Note: This recommendation only applies if you let kubelets get their certificates from the API server. In case your kubelet certificates come from an outside authority/tool (e.g. Vault) then you need to take care of rotation yourself.

Ignore this check if serverTLSBootstrap is true in the kubelet config file or if the --rotate-server-certificates parameter is set on kubelet.

Run the following command on each node:

ps -ef | grep kubelet

Confirm RotateKubeletServerCertificate argument exists and is set to true.

Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable.

--feature-gates=RotateKubeletServerCertificate=true

Based on your system, restart the kubelet service. For example:

systemctl daemon-reload
systemctl restart kubelet.service
Ensure the Kubelet only makes use of Strong Cryptographic Ciphers

TLS ciphers have had a number of known vulnerabilities and weaknesses, which can reduce the protection provided by them. By default Kubernetes supports a number of TLS cipher suites including some that have security concerns, weakening the protection provided.

The set of cryptographic ciphers currently considered secure is the following:

TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_RSA_WITH_AES_256_GCM_SHA384
TLS_RSA_WITH_AES_128_GCM_SHA256

Run the following command on each node:

ps -ef | grep kubelet

If the --tls-cipher-suites argument is present, ensure it only contains values included in this set.

If it is not present check that there is a Kubelet config file specified by --config, and that file sets TLSCipherSuites to only include values from this set.

If using a Kubelet config file, edit the file to set TLSCipherSuites to

TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_RSA_WITH_AES_256_GCM_SHA384
TLS_RSA_WITH_AES_128_GCM_SHA256

or to a subset of these values.

If using executable arguments, edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and set the --tls-cipher-suites parameter as follows, or to a subset of these values.

--tls-cipher-
suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256

Based on your system, restart the kubelet service. For example:

systemctl daemon-reload
systemctl restart kubelet.service

Policies

RBAC and service accounts

Ensure the cluster-admin role is only used where required

The RBAC role cluster-admin provides wide-ranging powers over the environment and should be used only where and when needed.

Kubernetes provides a set of default roles where RBAC is used. Some of these roles such as cluster-admin provide wide-ranging privileges which should only be applied where absolutely necessary. Roles such as cluster-admin allow super-user access to perform any action on any resource. When used in a ClusterRoleBinding, it gives full control over every resource in the cluster and in all namespaces. When used in a RoleBinding, it gives full control over every resource in the rolebinding's namespace, including the namespace itself.

Obtain a list of the principals who have access to the cluster-admin role by reviewing the ClusterRoleBinding output for each role binding that has access to the cluster-admin role.

kubectl get clusterrolebindings -o=custom-
columns=NAME:.metadata.name,ROLE:.roleRef.name,SUBJECT:.subjects[*].name

Review each principal listed and ensure that cluster-admin privilege is required for it.

Identify all clusterrolebindings to the cluster-admin role. Check if they are used and if they need this role or if they could use a role with fewer privileges.

Where possible, first bind users to a lower privileged role and then remove the clusterrolebinding to the cluster-admin role:

kubectl delete clusterrolebinding [name]
Minimize access to secrets

The Kubernetes API stores secrets, which may be service account tokens for the Kubernetes API or credentials used by workloads in the cluster. Access to these secrets should be restricted to the smallest possible group of users to reduce the risk of privilege escalation.

Inappropriate access to secrets stored within the Kubernetes cluster can allow for an attacker to gain additional access to the Kubernetes cluster or external resources whose credentials are stored as secrets.

Review the users who have get, list or watch access to secrets objects in the Kubernetes API.

Where possible, remove get, list and watch access to secret objects in the cluster.

Minimize wildcard use in Roles and ClusterRoles

Kubernetes Roles and ClusterRoles provide access to resources based on sets of objects and actions that can be taken on those objects. It is possible to set either of these to be the wildcard "*" which matches all items.

Use of wildcards is not optimal from a security perspective as it may allow for inadvertent access to be granted when new resources are added to the Kubernetes API either as CRDs or in later versions of the product.

The principle of least privilege recommends that users are provided only the access required for their role and nothing more. The use of wildcard rights grants is likely to provide excessive rights to the Kubernetes API.

Retrieve the roles defined across each namespaces in the cluster and review for wildcards

kubectl get roles --all-namespaces -o yaml

Retrieve the cluster roles defined in the cluster and review for wildcards

kubectl get clusterroles -o yaml

Where possible replace any use of wildcards in clusterroles and roles with specific objects or actions.

Minimize access to create pods

The ability to create pods in a namespace can provide a number of opportunities for privilege escalation, such as assigning privileged service accounts to these pods or mounting hostPaths with access to sensitive data (unless Pod Security Policies are implemented to restrict this access)

As such, access to create new pods should be restricted to the smallest possible group of users.

The ability to create pods in a cluster opens up possibilities for privilege escalation and should be restricted, where possible.

Review the users who have create access to pod objects in the Kubernetes API.

Where possible, remove create access to pod objects in the cluster.

Ensure default service accounts are not actively used

The default service account should not be used to ensure that rights granted to applications can be more easily audited and reviewed.

Kubernetes provides a default service account which is used by cluster workloads where no specific service account is assigned to the pod.

Where access to the Kubernetes API from a pod is required, a specific service account should be created for that pod, and rights granted to that service account.

The default service account should be configured such that it does not provide a service account token and does not have any explicit rights assignments.

For each namespace in the cluster, review the rights assigned to the default service account and ensure that it has no roles or cluster roles bound to it apart from the defaults. Additionally ensure that the automountServiceAccountToken: false setting is in place for each default service account.

Create explicit service accounts wherever a Kubernetes workload requires specific access to the Kubernetes API server.

Modify the configuration of each default service account to include this value.

automountServiceAccountToken: false
Ensure Service Account Tokens are only mounted where necessary

Service accounts tokens should not be mounted in pods except where the workload running in the pod explicitly needs to communicate with the API server.

Mounting service account tokens inside pods can provide an avenue for privilege escalation attacks where an attacker is able to compromise a single pod in the cluster.

Avoiding mounting these tokens removes this attack avenue.

Review pod and service account objects in the cluster and ensure that the option below is set, unless the resource explicitly requires this access.

automountServiceAccountToken: false

Modify the definition of pods and service accounts which do not need to mount service account tokens to disable it.

Pod security policies

A Pod Security Policy (PSP) is a cluster-level resource that controls security settings for pods. Your cluster may have multiple PSPs. You can query PSPs with the following command:

kubectl get psp

Pod Security Policies are used in conjunction with the Pod Security Policy admission controller plugin.

Minimize the admission of privileged containers

Do not generally permit containers to be run with the securityContext.privileged flag set to true.

Privileged containers have access to all Linux Kernel capabilities and devices. A container running with full privileges can do almost everything that the host can do. This flag exists to allow special use-cases, like manipulating the network stack and accessing devices.

There should be at least one Pod Security Policy (PSP) defined which does not permit privileged containers.

If you need to run privileged containers, this should be defined in a separate PSP and you should carefully check RBAC controls to ensure that only limited service accounts and users are given permission to access that PSP.

Get the set of PSPs with the following command:

kubectl get psp

For each PSP, check whether privileged is enabled:

kubectl get psp <name> -o=jsonpath='{.spec.privileged}'

Confirm there is at least one PSP which does not return true.

Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.privileged field is omitted or set to false.

Minimize the admission of containers wishing to share the host process ID namespace

Do not generally permit containers to be run with the hostPID flag set to true.

A container running in the host’s PID namespace can inspect processes running outside the container. If the container also has access to ptrace capabilities this can be used to escalate privileges outside of the container.

There should be at least one Pod Security Policy (PSP) defined which does not permit containers to share the host PID namespace.

If you need to run containers which require hostPID, this should be defined in a separate PSP and you should carefully check RBAC controls to ensure that only limited service accounts and users are given permission to access that PSP.

Get the set of PSPs with the following command:

kubectl get psp

For each PSP, check whether privileged is enabled:

kubectl get psp <name> -o=jsonpath='{.spec.hostPID}'

Confirm there is at least one PSP which does not return true.

Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.hostPID field is omitted or set to false.

Minimize the admission of containers wishing to share the host IPC namespace

Do not generally permit containers to be run with the hostIPC flag set to true.

A container running in the host’s IPC namespace can use IPC to interact with processes outside the container.

There should be at least one Pod Security Policy (PSP) defined which does not permit containers to share the host IPC namespace.

If you have a requirement to containers which require hostIPC, this should be defined in a separate PSP and you should carefully check RBAC controls to ensure that only limited service accounts and users are given permission to access that PSP.

Get the set of PSPs with the following command:

kubectl get psp

For each PSP, check whether privileged is enabled:

kubectl get psp <name> -o=jsonpath='{.spec.hostIPC}'

Confirm there is at least one PSP which does not return true.

Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.hostIPC field is omitted or set to false.

Minimize the admission of containers wishing to share the host network namespace

Do not generally permit containers to be run with the hostNetwork flag set to true.

A container running in the host's network namespace could access the local loopback device, and could access network traffic to and from other pods.

There should be at least one Pod Security Policy (PSP) defined which does not permit containers to share the host network namespace.

If you have need to run containers which require hostNetwork, this should be defined in a separate PSP and you should carefully check RBAC controls to ensure that only limited service accounts and users are given permission to access that PSP.

Get the set of PSPs with the following command:

kubectl get psp

For each PSP, check whether privileged is enabled:

kubectl get psp <name> -o=jsonpath='{.spec.hostNetwork}'

Confirm there is at least one PSP which does not return true.

Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.hostNetwork field is omitted or set to false.

Minimize the admission of containers with allowPrivilegeEscalation

Do not generally permit containers to be run with the allowPrivilegeEscalation flag set to true.

A container running with the allowPrivilegeEscalation flag set to true may have processes that can gain more privileges than their parent.

There should be at least one PodSecurityPolicy (PSP) defined which does not permit containers to allow privilege escalation. The option exists (and is defaulted to true) to permit setuid binaries to run.

If you have need to run containers which use setuid binaries or require privilege escalation, this should be defined in a separate PSP and you should carefully check RBAC controls to ensure that only limited service accounts and users are given permission to access that PSP.

Get the set of PSPs with the following command:

kubectl get psp

For each PSP, check whether privileged is enabled:

kubectl get psp <name> -o=jsonpath='{.spec.allowPrivilegeEscalation}'

Confirm there is at least one PSP which does not return true.

Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.allowPrivilegeEscalation field is omitted or set to false.

Minimize the admission of root containers

Do not generally permit containers to be run as the root user.

Containers may run as any Linux user. Containers which run as the root user, whilst constrained by Container Runtime security features still have a escalated likelihood of container breakout.

Ideally, all containers should run as a defined non-UID 0 user. There should be at least one PodSecurityPolicy (PSP) defined which does not permit root users in a container.

If you need to run root containers, this should be defined in a separate PSP and you should carefully check RBAC controls to ensure that only limited service accounts and users are given permission to access that PSP.

Get the set of PSPs with the following command:

kubectl get psp

For each PSP, check whether running containers as root is enabled:

kubectl get psp <name> -o=jsonpath='{.spec.runAsUser.rule}'

Confirm there is at least one PSP which returns MustRunAsNonRoot or MustRunAs with the range of UIDs not including 0.

Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.runAsUser.rule is set to either MustRunAsNonRoot or MustRunAs with the range of UIDs not including 0.

Minimize the admission of containers with the NET_RAW capability

Do not generally permit containers with the potentially dangerous NET_RAW capability.

Containers run with a default set of capabilities as assigned by the Container Runtime. By default this can include potentially dangerous capabilities. With Docker as the container runtime the NET_RAW capability is enabled which may be misused by malicious containers.

Ideally, all containers should drop this capability.

There should be at least one PodSecurityPolicy (PSP) defined which prevents containers with the NET_RAW capability from launching.

If you need to run containers with this capability, this should be defined in a separate PSP and you should carefully check RBAC controls to ensure that only limited service accounts and users are given permission to access that PSP.

Get the set of PSPs with the following command:

kubectl get psp

For each PSP, check whether NET_RAW is disabled:

kubectl get psp <name> -o=jsonpath='{.spec.requiredDropCapabilities}'

Confirm there is at least one PSP which returns NET_RAW or ALL.

Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.requiredDropCapabilities is set to include either NET_RAW or ALL.

Minimize the admission of containers with added capabilities

Do not generally permit containers with capabilities assigned beyond the default set.

Containers run with a default set of capabilities as assigned by the Container Runtime. Capabilities outside this set can be added to containers which could expose them to risks of container breakout attacks.

There should be at least one PodSecurityPolicy (PSP) defined which prevents containers with capabilities beyond the default set from launching.

If you need to run containers with additional capabilities, this should be defined in a separate PSP and you should carefully check RBAC controls to ensure that only limited service accounts and users are given permission to access that PSP.

Get the set of PSPs with the following command:

kubectl get psp

Confirm there are no PSPs present which have allowedCapabilities set to anything other than an empty array.

Ensure that allowedCapabilities is not present in PSPs for the cluster unless it is set to an empty array.

Minimize the admission of containers with capabilities assigned

Do not generally permit containers with capabilities

Containers run with a default set of capabilities as assigned by the Container Runtime. Capabilities are parts of the rights generally granted on a Linux system to the root user.

In many cases applications running in containers do not require any capabilities to operate, so from the perspective of the principal of least privilege use of capabilities should be minimized.

Get the set of PSPs with the following command:

kubectl get psp

For each PSP, check whether capabilities have been forbidden:

kubectl get psp <name> -o=jsonpath='{.spec.requiredDropCapabilities}'

Review the use of capabilites in applications runnning on your cluster. Where a namespace contains applicaions which do not require any Linux capabities to operate consider adding a PSP which forbids the admission of containers which do not drop all capabilities.

Network policies and CNI

Ensure the CNI in use supports Network Policies

There are a variety of CNI plugins available for Kubernetes. If the CNI in use does not support Network Policies it may not be possible to effectively restrict traffic in the cluster.

Kubernetes network policies are enforced by the CNI plugin in use. As such it is important to ensure that the CNI plugin supports both Ingress and Egress network policies.

Review the documentation of CNI plugin in use by the cluster, and confirm that it supports Ingress and Egress network policies.

If the CNI plugin in use does not support network policies, consideration should be given to making use of a different plugin, or finding an alternate mechanism for restricting traffic in the Kubernetes cluster.

Ensure all Namespaces have Network Policies defined

Use network policies to isolate traffic in your cluster network.

Running different applications on the same Kubernetes cluster creates a risk of one compromised application attacking a neighboring application. Network segmentation is important to ensure that containers can communicate only with those they are supposed to. A network policy is a specification of how selections of pods are allowed to communicate with each other and other network endpoints.

Network Policies are namespace scoped. When a network policy is introduced to a given namespace, all traffic not allowed by the policy is denied. However, if there are no network policies in a namespace all traffic will be allowed into and out of the pods in that namespace.

Run the following command and review the NetworkPolicy objects created in the cluster.

kubectl --all-namespaces get networkpolicy

Ensure that each namespace defined in the cluster has at least one Network Policy.

Follow the documentation and create NetworkPolicy objects as you need them.

Secrets management

Prefer using secrets as files over secrets as environment variables

Kubernetes supports mounting secrets as data volumes or as environment variables. Minimize the use of environment variable secrets.

It is reasonably common for application code to log out its environment (particularly in the event of an error). This will include any secret values passed in as environment variables, so secrets can easily be exposed to any user or entity who has access to the logs.

Run the following command to find references to objects which use environment variables defined from secrets.

 kubectl get all -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}' -A

If possible, rewrite application code to read secrets from mounted secret files, rather than from environment variables.

Consider external secret storage

Consider the use of an external secrets storage and management system, instead of using Kubernetes Secrets directly, if you have more complex secret management needs. Ensure the solution requires authentication to access secrets, has auditing of access to and use of secrets, and encrypts secrets. Some solutions also make it easier to rotate secrets.

Kubernetes supports secrets as first-class objects, but care needs to be taken to ensure that access to secrets is carefully limited. Using an external secrets provider can ease the management of access to secrets, especially where secrests are used across both Kubernetes and non-Kubernetes environments.

Review your secrets management implementation.

Refer to the secrets management options offered by your cloud provider or a third-party secrets management solution.

Extensible admission control

Configure Image Provenance using ImagePolicyWebhook admission controller

Configure Image Provenance for your deployment.

Kubernetes supports plugging in provenance rules to accept or reject the images in your deployments. You could configure such rules to ensure that only approved images are deployed in the cluster.

Review the pod definitions in your cluster and Confirm image provenance is configured as appropriate.

Follow the Kubernetes documentation and setup image provenance.

General policies

Create administrative boundaries between resources using namespaces

Use namespaces to isolate your Kubernetes objects.

Limiting the scope of user permissions can reduce the impact of mistakes or malicious activities. A Kubernetes namespace allows you to partition created resources into logically named groups. Resources created in one namespace can be hidden from other namespaces. By default, each resource created by a user in Kubernetes cluster runs in a default namespace, called default. You can create additional namespaces and attach resources and users to them. You can use Kubernetes Authorization plugins to create policies that segregate access to namespace resources between different users.

Run the following command and review the namespaces created in the cluster.

kubectl get namespaces

Ensure that these namespaces are the ones you need and are adequately administered as per your requirements.

Follow the documentation and create namespaces for objects in your deployment as you need them.

Set the Seccomp profile to docker/default in your pod definitions

Enable docker/default Seccomp profile in your pod definitions.

Seccomp (secure computing mode) is used to restrict the set of system calls applications can make, allowing cluster administrators greater control over the security of workloads running in the cluster. Kubernetes disables seccomp profiles by default for historical reasons. You should enable it to ensure that the workloads have restricted actions available within the container.

Review the pod definitions in your cluster. It should create a line as below:

annotations:
  seccomp.security.alpha.kubernetes.io/pod: docker/default

Seccomp is an alpha feature currently. By default, all alpha features are disabled. So, you would need to enable alpha features in the API server by passing --feature- gates=AllAlpha=true argument.

Edit the /etc/kubernetes/apiserver file on the master node and set the KUBE_API_ARGS parameter to "--feature-gates=AllAlpha=true"

KUBE_API_ARGS="--feature-gates=AllAlpha=true"

Based on your system, restart the kube-apiserver service. For example:

systemctl restart kube-apiserver.service

Use annotations to enable the docker/default Seccomp profile in your pod definitions. An example is as below:

apiVersion: v1
kind: Pod
metadata:
  name: trustworthy-pod
  annotations:
    seccomp.security.alpha.kubernetes.io/pod: docker/default
spec:
  containers:
    - name: trustworthy-container
    image: sotrustworthy:latest
Apply security context to your pods and containers

A security context defines the operating system security settings (uid, gid, capabilities, SELinux role, etc..) applied to a container. When designing your containers and pods, make sure that you configure the security context for your pods, containers, and volumes. A security context is a property defined in the deployment yaml. It controls the security parameters that will be assigned to the pod/container/volume. There are two levels of security context: pod level security context, and container level security context.

Review the pod definitions in your cluster and confirm you have security contexts defined as appropriate.

Follow the Kubernetes documentation and apply security contexts to your pods. For a suggested list of security contexts, you may refer to the CIS Security Benchmark for Docker Containers.

The default namespace should not be used

Kubernetes provides a default namespace, where objects are placed if no namespace is specified for them. Placing objects in this namespace makes application of RBAC and other controls more difficult.

Resources in a Kubernetes cluster should be segregated by namespace, to allow for security controls to be applied at that level and to make it easier to manage resources.

Run this command to list objects in default namespace

kubectl get all

The only entries there should be system managed resources such as the kubernetes service

Ensure that namespaces are created to allow for appropriate segregation of Kubernetes resources and that all new resources are created in a specific namespace.

Kubernetes-managed services – Google Kubernetes Engine (GKE)

Image registry and image scanning

Ensure image vulnerability scanning using GCR Container Analysis or a third-party provider

Scan images stored in Google Container Registry (GCR) for vulnerabilities.

Vulnerabilities in software packages can be exploited by hackers or malicious users to obtain unauthorized access to local cloud resources. GCR Container Analysis and other third-party products allow images stored in GCR to be scanned for known vulnerabilities.

gcloud services list --enabled --filter containerregistry

Ensure that the Container Scanning API is listed within the output.

gcloud services enable containerscanning.googleapis.com
Minimize user access to GCR

Restrict user access to GCR, limiting interaction with build images to only authorized personnel and service accounts.

Weak access control to GCR may allow malicious users to replace built images with vulnerable or backdoored containers.

To check GCR bucket specific permissions:

gsutil iam get gs://artifacts.[PROJECT_ID].appspot.com

The output of the command will return roles associated with the GCR bucket and which members have those roles.

Additionally, run the following to identify users and service accounts that hold privileged roles at the project level, and thus inherit these privileges within the GCR bucket:

gcloud projects get-iam-policy [PROJECT_ID] \
--flatten="bindings[].members" \
--format='table(bindings.members,bindings.role)' \
--filter="bindings.role:roles/storage.admin OR
bindings.role:roles/storage.objectAdmin OR \
bindings.role:roles/storage.objectCreator OR
bindings.role:roles/storage.legacyBucketOwner OR \
bindings.role:roles/storage.legacyBucketWriter OR
bindings.role:roles/storage.legacyObjectOwner"

The output from the command lists the service accounts that have create/modify permissions.

Users may have permissions to use Service Accounts and thus Users could inherit privileges on the GCR Bucket. To check the accounts that could do this:

gcloud projects get-iam-policy [PROJECT_ID] \
--flatten="bindings[].members" \
--format='table(bindings.members)' \
--filter="bindings.role:roles/iam.serviceAccountUser"

Note that other privileged project level roles will have the ability to write and modify objects and the GCR bucket. Consult the GCP CIS benchmark and IAM documentation for further reference.

To change roles at the GCR bucket level:

Firstly, run the following if read permissions are required:

gsutil iam ch [TYPE]:[EMAIL-ADDRESS]:objectViewer
gs://artifacts.[PROJECT_ID].appspot.com

Then remove the excessively privileged role (Storage Admin / Storage Object Admin / Storage Object Creator) using:

gsutil iam ch -d [TYPE]:[EMAIL-ADDRESS]:[ROLE]
gs://artifacts.[PROJECT_ID].appspot.com

where:

  • [TYPE] can be one of the following:
    • user, if the [EMAIL-ADDRESS] is a Google account
    • serviceAccount, if [EMAIL-ADDRESS] specifies a Service account
  • [EMAIL-ADDRESS] can be one of the following:
    • a Google account (for example, someone@example.com)
    • a Cloud IAM service account

To modify roles defined at the project level and subsequently inherited within the GCR bucket, or the Service Account User role, extract the IAM policy file, modify it accordingly and apply it using:

gcloud projects set-iam-policy [PROJECT_ID] [POLICY_FILE]
Minimize cluster access to read-only for GCR

Configure the Cluster Service Account with Storage Object Viewer Role to only allow read- only access to GCR.

The Cluster Service Account does not require administrative access to GCR, only requiring pull access to containers to deploy onto GKE. Restricting permissions follows the principles of least privilege and prevents credentials from being abused beyond the required role.

gsutil iam get gs://artifacts.[PROJECT_ID].appspot.com

The output of the command will return roles associated with the GCR bucket. If listed, Set the GKE Service account to "role": "roles/storage.objectViewer".

If the GKE Service Account has project level permissions that are inherited within the bucket, ensure that these are not privileged:

gcloud projects get-iam-policy [PROJECT_ID] \
--flatten="bindings[].members" \
--format='table(bindings.members,bindings.role)' \
--filter="bindings.role:roles/storage.admin OR
bindings.role:roles/storage.objectAdmin OR \
bindings.role:roles/storage.objectCreator OR
bindings.role:roles/storage.legacyBucketOwner OR \
bindings.role:roles/storage.legacyBucketWriter OR
bindings.role:roles/storage.legacyObjectOwner"

Your GKE Service Account should not be output when this command is run.

For an account explicitly granted to the bucket. First, add read access to the Kubernetes Service Account:

gsutil iam ch [TYPE]:[EMAIL-ADDRESS]:objectViewer
gs://artifacts.[PROJECT_ID].appspot.com

where:

  • [TYPE] can be one of the following:
  • user, if the [EMAIL-ADDRESS] is a Google account
  • serviceAccount, if [EMAIL-ADDRESS] specifies a Service account
  • [EMAIL-ADDRESS] can be one of the following:
  • a Google account (for example, someone@example.com)
  • a Cloud IAM service account

Then remove the excessively privileged role (Storage Admin / Storage Object Admin / Storage Object Creator) using:

gsutil iam ch -d [TYPE]:[EMAIL-ADDRESS]:[ROLE]
gs://artifacts.[PROJECT_ID].appspot.com

For an account that inherits access to the GCR Bucket through Project level permissions, modify the Projects IAM policy file accordingly, then upload it using:

gcloud projects set-iam-policy [PROJECT_ID] [POLICY_FILE]
Minimize Container Registries to only those approved

Use Binary Authorization to allowlist (whitelist) only approved container registries.

Allowing unrestricted access to external container registries provides the opportunity for malicious or unapproved containers to be deployed into the cluster. Allowlisting only approved container registries reduces this risk.

First, check that Binary Authorization is enabled for the GKE cluster:

gcloud container clusters describe [CLUSTER_NAME] --zone [COMPUTE_ZONE] --
format json | jq .binaryAuthorization

This will return the following if Binary Authorization is enabled:

{
  "enabled": true
}

Then assess the contents of the policy:

gcloud container binauthz policy export > current-policy.yaml

Ensure that the current policy is not configured to allow all images (evaluationMode: ALWAYS_ALLOW).

Review the list of admissionWhitelistPatterns for unauthorized container registries.

cat current-policy.yaml
admissionWhitelistPatterns:
...
defaultAdmissionRule:
  evaluationMode: ALWAYS_ALLOW

First, update the cluster to enable Binary Authorization:

gcloud container cluster update [CLUSTER_NAME] --enable-binauthz

Create a Binary Authorization Policy using the Binary Authorization Policy Reference for guidance.

Import the policy file into Binary Authorization:

gcloud container binauthz policy import [YAML_POLICY]

Identity and access management (IAM)

Ensure GKE clusters are not running using the Compute Engine default service account

Create and use minimally privileged Service accounts to run GKE cluster nodes instead of using the Compute Engine default Service account. Unnecessary permissions could be abused in the case of a node compromise.

A GCP service account (as distinct from a Kubernetes ServiceAccount) is an identity that an instance or an application can use to run GCP API requests on your behalf. This identity is used to identify virtual machine instances to other Google Cloud Platform services. By default, Kubernetes Engine nodes use the Compute Engine default service account. This account has broad access by default, as defined by access scopes, making it useful to a wide variety of applications on the VM, but it has more permissions than are required to run your Kubernetes Engine cluster.

You should create and use a minimally privileged service account to run your Kubernetes Engine cluster instead of using the Compute Engine default service account, and create separate service accounts for each Kubernetes Workload.

Kubernetes Engine requires, at a minimum, the node service account to have the monitoring.viewer, monitoring.metricWriter, and logging.logWriter roles. Additional roles may need to be added for the nodes to pull images from GCR.

To check which Service account is set for an existing cluster, run the following command:

gcloud container node-pools describe [NODE_POOL] \
  --cluster [CLUSTER_NAME] --zone [COMPUTE_ZONE] \
  --format json | jq '.config.serviceAccount'

The output of the above command will return default if default Service account is used for Project access.

To check that the permissions allocated to the service account are the minimum required for cluster operation:

gcloud projects get-iam-policy [PROJECT_ID] \
   --flatten="bindings[].members" \
   --format='table(bindings.role)' \
   --filter="bindings.members:[SERVICE_ACCOUNT]"

Review the output to ensure that the service account only has the roles required to run the cluster:

roles/logging.logWriter
roles/monitoring.metricWriter
roles/monitoring.viewer

Firstly, create a minimally privileged service account:

gcloud iam service-accounts create [SA_NAME] \
  --display-name "GKE Node Service Account"
export NODE_SA_EMAIL=gcloud iam service-accounts list \
  --format='value(email)' \
  --filter='displayName:GKE Node Service Account'

Grant the following roles to the service account:

export PROJECT_ID=gcloud config get-value project
gcloud projects add-iam-policy-binding $PROJECT_ID \
   --member serviceAccount:$NODE_SA_EMAIL \
   --role roles/monitoring.metricWriter
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member serviceAccount:$NODE_SA_EMAIL \
  --role roles/monitoring.viewer
gcloud projects add-iam-policy-binding $PROJECT_ID \
   --member serviceAccount:$NODE_SA_EMAIL \
   --role roles/logging.logWriter

To create a new Node pool using the Service account, run the following command:

gcloud container node-pools create [NODE_POOL] \
  --service-account=[SA_NAME]@[PROJECT_ID].iam.gserviceaccount.com \
  --cluster=[CLUSTER_NAME] --zone [COMPUTE_ZONE]

You will need to migrate your workloads to the new Node pool, and delete Node pools that use the default service account to complete the remediation.

Prefer using dedicated GCP Service Accounts and Workload Identity

Kubernetes workloads should not use cluster node service accounts to authenticate to Google Cloud APIs. Each Kubernetes Workload that needs to authenticate to other Google services using Cloud IAM should be provisioned a dedicated Service account. Enabling Workload Identity manages the distribution and rotation of Service account keys for the workloads to use.

Manual approaches for authenticating Kubernetes workloads running on GKE against Google Cloud APIs are: storing service account keys as a Kubernetes secret (which introduces manual key rotation and potential for key compromise); or use of the underlying nodes’ IAM Service account, which violates the principle of least privilege on a multitenant node, when one pod needs to have access to a service, but every other pod on the node that uses the Service account does not.

Once a relationship between a Kubernetes Service account and a GCP Service account has been configured, any workload running as the Kubernetes Service account automatically authenticates as the mapped GCP Service account when accessing Google Cloud APIs on a cluster with Workload Identity enabled.

gcloud beta container clusters describe [CLUSTER_NAME] --zone [CLUSTER_ZONE]

If Workload Identity is enabled, the following fields should be present, and the [PROJECT_ID] should be set to the namespace of the GCP project containing the cluster:

workloadIdentityConfig:
  identityNamespace:[PROJECT_ID].svc.id.goog

For each Node pool, ensure the following is set.

workloadMetadataConfig:
    nodeMetadata: GKE_METADATA_SERVER

You will also need to manually audit each Kubernetes workload requiring Google Cloud API access to ensure that Workload Identity is being used and not some other method.

gcloud beta container clusters update [CLUSTER_NAME] --zone [CLUSTER_ZONE] \
  --identity-namespace=[PROJECT_ID].svc.id.goog

Note that existing Node pools are unaffected. New Node pools default to --workload- metadata-from-node=GKE_METADATA_SERVER.

Then, modify existing Node pools to enable GKE_METADATA_SERVER:

gcloud beta container node-pools update [NODEPOOL_NAME] \
  --cluster=[CLUSTER_NAME] --zone [CLUSTER_ZONE] \
  --workload-metadata-from-node=GKE_METADATA_SERVER

You may also need to modify workloads in order for them to use Workload Identity as described within https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity. Also consider the effects on the availability of your hosted workloads as Node pools are updated, it may be more appropriate to create new Node Pools.

Cloud key-management service (Cloud KMS)

Ensure Kubernetes Secrets are encrypted using keys managed in Cloud KMS

Encrypt Kubernetes secrets, stored in etcd, at the application-layer using a customer- managed key in Cloud KMS.

By default, GKE encrypts customer content stored at rest, including Secrets. GKE handles and manages this default encryption for you without any additional action on your part.

Application-layer Secrets Encryption provides an additional layer of security for sensitive data, such as user defined Secrets and Secrets required for the operation of the cluster, such as service account keys, which are all stored in etcd.

Using this functionality, you can use a key, that you manage in Cloud KMS, to encrypt data at the application layer. This protects against attackers in the event that they manage to gain access to etcd.

gcloud container clusters describe [CLUSTER_NAME] --format json | jq '.databaseEncryption'

If configured correctly, the output from the command returns a response containing the following detail:

keyName=projects/[PROJECT_ID]/locations/[LOCATION]/keyRings/[RING_NAME]/crypt
oKeys/[KEY_NAME]
state=ENCRYPTED

To enable Application-layer Secrets Encryption, several configuration items are required. These include:

  • A key ring
  • A key
  • A GKE service account with Cloud KMS CryptoKey Encrypter/Decrypter role

Once these are created, Application-layer Secrets Encryption can be enabled on an existing or new cluster.

Create a key ring:

gcloud kms keyrings create [RING_NAME] \
    --location [LOCATION] \
    --project [KEY_PROJECT_ID]

Create a key:

gcloud kms keys create [KEY_NAME] \
    --location [LOCATION] \
    --keyring [RING_NAME] \
    --purpose encryption \
    --project [KEY_PROJECT_ID]

Grant the Kubernetes Engine Service Agent service account the Cloud KMS CryptoKey Encrypter/Decrypter role:

gcloud kms keys add-iam-policy-binding [KEY_NAME] \
  --location [LOCATION] \
  --keyring [RING_NAME] \
  --member serviceAccount:[SERVICE_ACCOUNT_NAME] \
  --role roles/cloudkms.cryptoKeyEncrypterDecrypter \
  --project [KEY_PROJECT_ID]

To create a new cluster with Application-layer Secrets Encryption:

gcloud container clusters create [CLUSTER_NAME] \
  --cluster-version=latest \
  --zone [ZONE] \
  --database-encryption-key
projects/[KEY_PROJECT_ID]/locations/[LOCATION]/keyRings/[RING_NAME]/cryptoKeys/[KEY_NAME] \
  --project [CLUSTER_PROJECT_ID]

To enable on an existing cluster:

gcloud container clusters update [CLUSTER_NAME] \
  --zone [ZONE] \
  --database-encryption-key
projects/[KEY_PROJECT_ID]/locations/[LOCATION]/keyRings/[RING_NAME]/cryptoKey s/[KEY_NAME] \
  --project [CLUSTER_PROJECT_ID]

Node metadata

Ensure legacy Compute Engine instance metadata APIs are Disabled

Disable the legacy GCE instance metadata APIs for GKE nodes. Under some circumstances, these can be used from within a pod to extract the node's credentials.

The legacy GCE metadata endpoint allows simple HTTP requests to be made returning sensitive information. To prevent the enumeration of metadata endpoints and data exfiltration, the legacy metadata endpoint must be disabled.

Without requiring a custom HTTP header when accessing the legacy GCE metadata endpoint, a flaw in an application that allows an attacker to trick the code into retrieving the contents of an attacker-specified web URL could provide a simple method for enumeration and potential credential exfiltration. By requiring a custom HTTP header, the attacker needs to exploit an application flaw that allows them to control the URL and also add custom headers in order to carry out this attack successfully.

To check if the legacy metadata API is disabled for an individual Node pool, run the following command:

gcloud container node-pools describe [NODE_POOL] \
  --cluster [CLUSTER_NAME] --zone [COMPUTE_ZONE] \
  --format json | jq '.config.metadata'

Alternatively, to audit all of the clusters Node pools simultaneously, run the following command:

gcloud container clusters describe [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --format json | jq .nodePools[].config.metadata

For each of the Node pools with the correct setting the output of the above command returns:

"disable-legacy-endpoints"": ""true"

The legacy GCE metadata endpoint must be disabled upon the cluster or node-pool creation. For GKE versions 1.12 and newer, the legacy GCE metadata endpoint is disabled by default.

To update an existing cluster, create a new Node pool with the legacy GCE metadata endpoint disabled:

gcloud container node-pools create [POOL_NAME] \
 --metadata disable-legacy-endpoints=true \
  --cluster [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE]

You will need to migrate workloads from any existing non-conforming Node pools, to the new Node pool, then delete non-conforming Node pools to complete the remediation.

Ensure the GKE Metadata Server is enabled

Running the GKE Metadata Server prevents workloads from accessing sensitive instance metadata and facilitates Workload Identity

Every node stores its metadata on a metadata server. Some of this metadata, such as kubelet credentials and the VM instance identity token, is sensitive and should not be exposed to a Kubernetes workload. Enabling the GKE Metadata server prevents pods (that are not running on the host network) from accessing this metadata and facilitates Workload Identity.

When unspecified, the default setting allows running pods to have full access to the node's underlying metadata server.

To check whether the GKE Metadata Server is enabled for each Node pool within a cluster, run the following command:

gcloud beta container clusters describe [CLUSTER_NAME] \
  --zone [CLUSTER_ZONE] \
  --format json | jq .nodePools[].config.workloadMetadataConfig

This should return the following for each Node pool:

{
   "nodeMetadata": GKE_METADATA_SERVER
}

null ({ }) is returned if the GKE Metadata Server is not enabled.

The GKE Metadata Server requires Workload Identity to be enabled on a cluster. Modify the cluster to enable Workload Identity and enable the GKE Metadata Server.

gcloud beta container clusters update [CLUSTER_NAME] \
  --identity-namespace=[PROJECT_ID].svc.id.goog

Note that existing Node pools are unaffected. New Node pools default to --workload- metadata-from-node=GKE_METADATA_SERVER.

To modify an existing Node pool to enable GKE Metadata Server:

gcloud beta container node-pools update [NODEPOOL_NAME] \
  --cluster=[CLUSTER_NAME] \
  --workload-metadata-from-node=GKE_METADATA_SERVER

You may also need to modify workloads in order for them to use Workload Identity as described within https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity.

Node configuration and maintenance

Ensure Container-Optimized OS (COS) is used for GKE node images

Use Container-Optimized OS (COS) as a managed, optimized and hardened base OS that limits the host's attack surface.

COS is an operating system image for Compute Engine VMs optimized for running containers. With COS, you can bring up your containers on Google Cloud Platform quickly, efficiently, and securely.

Using COS as the node image provides the following benefits:

Run containers out of the box

COS instances come pre-installed with the container runtime and cloud-init. With a COS instance, you can bring up your container at the same time you create your VM, with no on-host setup required.

Smaller attack surface

COS has a smaller footprint, reducing your instance's potential attack surface.

Locked-down by default

COS instances include a locked-down firewall and other security settings by default.

To check Node image type for an existing cluster's Node pool:

gcloud container node-pools describe [NODE_POOL] \
  --cluster [CLUSTER_NAME] --zone [COMPUTE_ZONE] \
  --format json | jq '.config.imageType'

The output of the above command returns COS, if COS is used for Node images.

To set the node image to cos for an existing cluster's Node pool:

gcloud container clusters upgrade [CLUSTER_NAME]\
  --image-type cos \
  --zone [COMPUTE_ZONE] --node-pool [POOL_NAME]
Ensure Node Auto-Repair is enabled for GKE nodes

Nodes in a degraded state are an unknown quantity and so may pose a security risk.

Kubernetes Engine’s node auto-repair feature helps you keep the nodes in your cluster in a healthy, running state. When enabled, Kubernetes Engine makes periodic checks on the health state of each node in your cluster. If a node fails consecutive health checks over an extended time period, Kubernetes Engine initiates a repair process for that node.

To check the existence of node auto-repair for an existing cluster’s Node pool, run:

gcloud container node-pools describe [NODE_POOL_NAME] \
  --cluster [CLUSTER_NAME] --zone [COMPUTE_ZONE] \
  --format json | jq '.management'

Ensure the output of the above command has JSON key attribute autoRepair set to true:

{
  "autoRepair": true
}

To enable node auto-repair for an existing cluster with Node pool, run the following command:

gcloud container node-pools update [POOL_NAME] \
  --cluster [CLUSTER_NAME] --zone [COMPUTE_ZONE] \
  --enable-autorepair
Ensure Node Auto-Upgrade is enabled for GKE nodes

Node auto-upgrade keeps nodes at the current Kubernetes and OS security patch level to mitigate known vulnerabilities.

Node auto-upgrade helps you keep the nodes in your cluster or Node pool up to date with the latest stable patch version of Kubernetes as well as the underlying node operating system. Node auto-upgrade uses the same update mechanism as manual node upgrades.

Node pools with node auto-upgrade enabled are automatically scheduled for upgrades when a new stable Kubernetes version becomes available. When the upgrade is performed, the Node pool is upgraded to match the current cluster master version. From a security perspective, this has the benefit of applying security updates automatically to the Kubernetes Engine when security fixes are released.

To check the existence of node auto-upgrade for an existing cluster's Node pool, run:

gcloud container node-pools describe [NODE_POOL_NAME] \
--cluster [CLUSTER_NAME] --zone [COMPUTE_ZONE] \
--format json | jq '.management'

Ensure the output of the above command has JSON key attribute autoUpgrade set to true:

{
  "autoUpgrade": true
}

If node auto-upgrade is disabled, the output of the above command output will not contain the autoUpgrade entry.

To enable node auto-upgrade for an existing cluster's Node pool, run the following command:

gcloud container node-pools update [NODE_POOL] \
  --cluster [CLUSTER_NAME] --zone [COMPUTE_ZONE] \
  --enable-autoupgrade
Automate GKE version management using Release Channels

Subscribe to the Regular or Stable Release Channel to automate version upgrades to the GKE cluster and to reduce version management complexity to the number of features and level of stability required.

Release Channels signal a graduating level of stability and production-readiness. These are based on observed performance of GKE clusters running that version and represent experience and confidence in the cluster version.

The Regular release channel upgrades every few weeks and is for production users who need features not yet offered in the Stable channel. These versions have passed internal validation, but don’t have enough historical data to guarantee their stability. Known issues generally have known workarounds.

The Stable release channel upgrades every few months and is for production users who need stability above all else, and for whom frequent upgrades are too risky. These versions have passed internal validation and have been shown to be stable and reliable in production, based on the observed performance of those clusters.

Critical security patches are delivered to all release channels.

Run the following command:

gcloud beta container clusters describe [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --format json | jq .releaseChannel.channel

The output of the above command will return regular or stable if these release channels are being used to manage automatic upgrades for your cluster.

Currently, cluster Release Channels are only configurable at cluster provisioning time.

Create a new cluster by running the following command:

gcloud beta container clusters create [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --release-channel [RELEASE_CHANNEL]

where [RELEASE_CHANNEL] is stable or regular according to your needs.

Ensure shielded GKE Nodes are enabled

Shielded GKE Nodes provides verifiable integrity via secure boot, virtual trusted platform module (vTPM)-enabled measured boot, and integrity monitoring.

Shielded GKE nodes protects clusters against boot- or kernel-level malware or rootkits which persist beyond infected OS.

Shielded GKE nodes run firmware which is signed and verified using Google’s Certificate Authority, ensuring that the nodes’ firmware is unmodified and establishing the root of trust for Secure Boot. GKE node identity is strongly protected via virtual Trusted Platform Module (vTPM) and verified remotely by the master node before the node joins the cluster. Lastly, GKE node integrity (i.e., boot sequence and kernel) is measured and can be monitored and verified remotely.

Run the following command:

gcloud beta container clusters describe [CLUSTER_NAME] \
  --format json | jq '.shieldedNodes'

This will return the following if Shielded GKE Nodes are enabled:

{
  "enabled": true
}

To migrate an existing cluster, you will need to specify the --enable-shielded-nodes flag on a cluster update command:

gcloud beta container clusters update [CLUSTER_NAME] \
  --zone [CLUSTER_ZONE] \
  --enable-shielded-nodes
Ensure Integrity Monitoring for Shielded GKE Nodes is enabled

Enable Integrity Monitoring for Shielded GKE Nodes to be notified of inconsistencies during the node boot sequence.

Integrity Monitoring provides active alerting for Shielded GKE nodes which allows administrators to respond to integrity failures and prevent compromised nodes from being deployed into the cluster.

To check if Integrity Monitoring is enabled for the Node pools in your cluster, run the following command for each Node pool:

gcloud beta container node-pools describe [NODEPOOL_NAME] \
  --cluster [CLUSTER_NAME] --zone [COMPUTE_ZONE] \
  --format json | jq .config.shieldedInstanceConfig

This will return:

{
"enableIntegrityMonitoring": true
}

if Integrity Monitoring is enabled.

Once a Node pool is provisioned, it cannot be updated to enable Integrity Monitoring. You must create new Node pools within the cluster with Integrity Monitoring enabled.

To create a Node pool within the cluster with Integrity Monitoring enabled, run the following command:

gcloud beta container node-pools create [NODEPOOL_NAME] \
  --cluster [CLUSTER_NAME] --zone [COMPUTE_ZONE] \
  --shielded-integrity-monitoring

You will also need to migrate workloads from existing non-conforming Node pools to the newly created Node pool, then delete the non-conforming pools.

Ensure Secure Boot for Shielded GKE Nodes is enabled

Enable Secure Boot for Shielded GKE Nodes to verify the digital signature of node boot components.

An attacker may seek to alter boot components to persist malware or root kits during system initialization. Secure Boot helps ensure that the system only runs authentic software by verifying the digital signature of all boot components, and halting the boot process if signature verification fails.

To check if Secure Boot is enabled for the Node pools in your cluster, run the following command for each Node pool:

gcloud beta container node-pools describe [NODEPOOL_NAME] \
  --cluster [CLUSTER_NAME] --zone [COMPUTE_ZONE] \
  --format json | jq .config.shieldedInstanceConfig

This will return:

{
  "enableSecureBoot": true
}

if Secure Boot is enabled.

Once a Node pool is provisioned, it cannot be updated to enable Secure Boot. You must create new Node pools within the cluster with Secure Boot enabled.

To create a Node pool within the cluster with Secure Boot enabled, run the following command:

gcloud beta container node-pools create [NODEPOOL_NAME] \
  --cluster [CLUSTER_NAME] --zone [COMPUTE_ZONE] \
  --shielded-secure-boot

You will also need to migrate workloads from existing non-conforming Node pools to the newly created Node pool, then delete the non-conforming pools.

Cluster networking

Enable VPC Flow Logs and Intranode Visibility

Enable VPC Flow Logs and Intranode Visibility to see pod-level traffic, even for traffic within a worker node.

Enabling Intranode Visibility makes your intranode pod to pod traffic visible to the networking fabric. With this feature, you can use VPC Flow Logs or other VPC features for intranode traffic.

Run this command:

gcloud beta container clusters describe [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --format json | jq '.networkConfig.enableIntraNodeVisibility'

The result should return true if Intranode Visibility is Enabled.

To enable intranode visibility on an existing cluster, run the following command:

gcloud beta container clusters update [CLUSTER_NAME] \
    --enable-intra-node-visibility
Ensure use of VPC-native clusters

Create Alias IPs for the node network CIDR range in order to subsequently configure IP- based policies and firewalling for pods. A cluster that uses Alias IPs is called a 'VPC-native' cluster.

Using Alias IPs has several benefits:

  • Pod IPs are reserved within the network ahead of time, which prevents conflict with other compute resources.

  • The networking layer can perform anti-spoofing checks to ensure that egress traffic is not sent with arbitrary source IPs.

  • Firewall controls for Pods can be applied separately from their nodes.

  • Alias IPs allow Pods to directly access hosted services without using a NAT gateway.

To check Alias IP is enabled for an existing cluster, run the following command:

gcloud container clusters describe [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --format json | jq '.ipAllocationPolicy.useIpAliases'

The output of the above command should return true, if VPC-native (using alias IP) is enabled. If VPC-native (using alias IP) is disabled, the above command will return null ({ }).

Use of Alias IPs cannot be enabled on an existing cluster. To create a new cluster using Alias IPs, follow the instructions below.

To enable Alias IP on a new cluster, run the following command:

gcloud container clusters create [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --enable-ip-alias
Ensure Master Authorized Networks is enabled

Enable Master Authorized Networks to restrict access to the cluster's control plane (master endpoint) to only an allowlist (whitelist) of authorized IPs.

Authorized networks are a way of specifying a restricted range of IP addresses that are permitted to access your cluster's control plane. Kubernetes Engine uses both Transport Layer Security (TLS) and authentication to provide secure access to your cluster's control plane from the public internet. This provides you the flexibility to administer your cluster from anywhere; however, you might want to further restrict access to a set of IP addresses that you control. You can set this restriction by specifying an authorized network.

Master Authorized Networks blocks untrusted IP addresses. Google Cloud Platform IPs (such as traffic from Compute Engine VMs) can reach your master through HTTPS provided that they have the necessary Kubernetes credentials.

Restricting access to an authorized network can provide additional security benefits for your container cluster, including:

Better protection from outsider attacks

Authorized networks provide an additional layer of security by limiting external, non-GCP access to a specific set of addresses you designate, such as those that originate from your premises. This helps protect access to your cluster in the case of a vulnerability in the cluster's authentication or authorization mechanism.

Better protection from insider attacks

Authorized networks help protect your cluster from accidental leaks of master certificates from your company's premises. Leaked certificates used from outside GCP and outside the authorized IP ranges (for example, from addresses outside your company) are still denied access.

To check Master Authorized Networks status for an existing cluster, run the following command:

gcloud container clusters describe [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --format json | jq '.masterAuthorizedNetworksConfig'

The output should return:

{
  "enabled": true
}

if Master Authorized Networks is enabled. If Master Authorized Networks is disabled, the above command will return null ({ }).

To enable Master Authorized Networks for an existing cluster, run the following command:

gcloud container clusters update [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --enable-master-authorized-networks

Along with this, you can list authorized networks using the --master-authorized- networks flag which contains a list of up to 20 external networks that are allowed to connect to your cluster’s control plane through HTTPS. You provide these networks as a comma-separated list of addresses in CIDR notation (such as 90.90.100.0/24).

Ensure clusters are created with Private Endpoint Enabled and Public Access Disabled

Disable access to the Kubernetes API from outside the node network if it is not required.

In a private cluster, the master node has two endpoints, a private and public endpoint. The private endpoint is the internal IP address of the master, behind an internal load balancer in the master’s VPC network. Nodes communicate with the master using the private endpoint. The public endpoint enables the Kubernetes API to be accessed from outside the master’s VPC network.

Although Kubernetes API requires an authorized token to perform sensitive actions, a vulnerability could potentially expose the Kubernetes publicly with unrestricted access. Additionally, an attacker may be able to identify the current cluster and Kubernetes API version and determine whether it is vulnerable to an attack. Unless required, disabling public endpoint will help prevent such threats, and require the attacker to be on the master’s VPC network to perform any attack on the Kubernetes API.

Run this command:

gcloud container clusters describe [CLUSTER_NAME] \
  --format json | jq '.privateClusterConfig.enablePrivateEndpoint'

The output of the above command returns true if a Private Endpoint is enabled with Public Access disabled.

For an additional check, the endpoint parameter can be queried with the following command:

gcloud container clusters describe [CLUSTER_NAME] \
  --format json | jq '.endpoint'

The output of the above command returns a private IP address if Private Endpoint is enabled with Public Access disabled.

Once a cluster is created without enabling Private Endpoint only, it cannot be remediated. Rather, the cluster must be recreated.

Create a cluster with a Private Endpoint enabled and Public Access disabled by including the --enable-private-endpoint flag within the cluster create command:

gcloud container clusters create [CLUSTER_NAME] \
  --enable-private-endpoint

Setting this flag also requires the setting of --enable-private-nodes, --enable-ip-alias and --master-ipv4-cidr=[MASTER_CIDR_RANGE].

Ensure clusters are created with Private Nodes

Disable public IP addresses for cluster nodes, so that they only have private IP addresses. Private Nodes are nodes with no public IP addresses.

Disabling public IP addresses on cluster nodes restricts access to only internal networks, forcing attackers to obtain local network access before attempting to compromise the underlying Kubernetes hosts.

Run this command:

gcloud container clusters describe [CLUSTER_NAME] \
  --format json | jq '.privateClusterConfig.enablePrivateNodes'

The output of the above command returns true if Private Nodes is enabled.

Once a cluster is created without enabling Private Nodes, it cannot be remediated. Rather the cluster must be recreated.

To create a cluster with Private Nodes enabled, include the --enable-private-nodes flag within the cluster create command:

gcloud container clusters create [CLUSTER_NAME] \
  --enable-private-nodes

Setting this flag also requires the setting of --enable-ip-alias and --master-ipv4-cidr=[MASTER_CIDR_RANGE].

Consider firewalling GKE worker nodes

Reduce the network attack surface of GKE nodes by using Firewalls to restrict ingress and egress traffic.

Utilizing stringent ingress and egress firewall rules minimizes the ports and services exposed to a network-based attacker, whilst also restricting egress routes within or out of the cluster in the event that a compromised component attempts to form an outbound connection.

For the instance being evaluated, firstly obtain its Service account and tags:

gcloud compute instances describe [INSTANCE_NAME] \
  --zone [COMPUTE_ZONE] \
  --format json | jq '{tags: .tags.items[],
serviceaccount:.serviceAccounts[].email, network:
.networkInterfaces[].network}'

this will return:

{
  "tags": "[TAG]",
  "serviceaccount": "[SERVICE_ACCOUNT]"
  "network":
"https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/networks/[NETWORK]"
}

Then, observe the firewall rules applied to the instance by using the following command, replacing [TAG] and [SERVICE_ACCOUNT] as appropriate:

gcloud compute firewall-rules list \
  --format="table(
        name,
        network,
        direction,
        priority,
        sourceRanges.list():label=SRC_RANGES,
        destinationRanges.list():label=DEST_RANGES,
        allowed[].map().firewall_rule().list():label=ALLOW,
        denied[].map().firewall_rule().list():label=DENY,
        sourceTags.list():label=SRC_TAGS,
        sourceServiceAccounts.list():label=SRC_SVC_ACCT,
        targetTags.list():label=TARGET_TAGS,
        targetServiceAccounts.list():label=TARGET_SVC_ACCT,
        disabled
      )" \
  --filter="targetTags.list():[TAG] OR
targetServiceAccounts.list():[SERVICE_ACCOUNT]"

Firewall rules may also be applied to a network without specifically targeting Tags or Service Accounts. These can be observed using the following, replacing [NETWORK] as appropriate:

gcloud compute firewall-rules list \
  --format="table(
        name,
        network,
        direction,
        priority,
        sourceRanges.list():label=SRC_RANGES,
        destinationRanges.list():label=DEST_RANGES,
        allowed[].map().firewall_rule().list():label=ALLOW,
        denied[].map().firewall_rule().list():label=DENY,
        sourceTags.list():label=SRC_TAGS,
        sourceServiceAccounts.list():label=SRC_SVC_ACCT,
        targetTags.list():label=TARGET_TAGS,
        targetServiceAccounts.list():label=TARGET_SVC_ACCT,
        disabled
    )" \
  --filter="network.list():[NETWORK] AND -targetTags.list():* AND -
targetServiceAccounts.list():*"

Use the following command to generate firewall rules, setting the variables as appropriate. You may want to use the target [TAG] and [SERVICE_ACCOUNT] previously identified.

gcloud compute firewall-rules create FIREWALL_RULE_NAME \
    --network [NETWORK] \
    --priority [PRIORITY] \
    --direction [DIRECTION] \
    --action [ACTION] \
    --target-tags [TAG] \
    --target-service-accounts [SERVICE_ACCOUNT] \
    --source-ranges [SOURCE_CIDR-RANGE] \
    --source-tags [SOURCE_TAGS] \
    --source-service-accounts=[SOURCE_SERVICE_ACCOUNT] \
    --destination-ranges [DESTINATION_CIDR_RANGE] \
    --rules [RULES]
Ensure Network Policy is Enabled and set as appropriate

Use Network Policy to restrict pod to pod traffic within a cluster and segregate workloads.

By default, all pod to pod traffic within a cluster is allowed. Network Policy creates a pod- level firewall that can be used to restrict traffic between sources. Pod traffic is restricted by having a Network Policy that selects it (through the use of labels). Once there is any Network Policy in a namespace selecting a particular pod, that pod will reject any connections that are not allowed by any Network Policy. Other pods in the namespace that are not selected by any Network Policy will continue to accept all traffic.

Network Policies are managed via the Kubernetes Network Policy API and enforced by a network plugin, simply creating the resource without a compatible network plugin to implement it will have no effect. GKE supports Network Policy enforcement through the use of Calico.

To check Network Policy is enabled for an existing cluster, run the following command,

gcloud container clusters describe [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --format json | jq '.networkPolicy'

The output of the above command should be:

{
  "enabled": true
}

if Network Policy is enabled. If Network policy is disabled, the above command output will return null ({ }).

To enable Network Policy for an existing cluster, firstly enable the Network Policy add-on:

gcloud container clusters update [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --update-addons NetworkPolicy=ENABLED

Then, enable Network Policy:

gcloud container clusters update [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --enable-network-policy
Ensure use of Google-managed SSL Certificates

Encrypt traffic to HTTPS load balancers using Google-managed SSL certificates.

Encrypting traffic between users and your Kubernetes workload is fundamental to protecting data sent over the web.

Google-managed SSL Certificates are provisioned, renewed, and managed for your domain names. This is only available for HTTPS load balancers created using Ingress Resources, and not TCP/UDP load balancers created using Service of type:LoadBalancer.

Firstly, identify any workloads exposed publicly using Services of type LoadBalancer:

kubectl get svc -A -o json | jq '.items[] |
select(.spec.type=="LoadBalancer")'

You will need to consider using ingresses instead of these services in order to use Google-managed SSL certificates.

For the ingresses within your cluster, run the following command:

kubectl get ingress -A -o json | jq .items[] | jq \
  '{name: .metadata.name, annotations: .metadata.annotations, namespace:
  .metadata.namespace, status: .status}'

The above command should return the name of the ingress, namespace, annotations and status. Check that the following annotation is present to ensure managed certificates are referenced.

"annotations": {
    ...
    "networking.gke.io/managed-certificates": "[EXAMPLE_CERTIFICATE]"
  },

For completeness, run the following command to ensure that the managed certificate resource exists:

kubectl get managedcertificates -A

The above command returns a list of managed certificates for which [EXAMPLE_CERTIFICATE] should exist within the same namespace as the ingress.

If services of type LoadBalancer are discovered, consider replacing the Service with an ingress.

To configure the ingress and use Google-managed SSL certificates, follow the instructions.

Logging

Ensure Stackdriver Kubernetes Logging and Monitoring is enabled

Send logs and metrics to a remote aggregator to mitigate the risk of local tampering in the event of a breach.

Exporting logs and metrics to a dedicated, persistent datastore such as Stackdriver ensures availability of audit data following a cluster security event, and provides a central location for analysis of log and metric data collated from multiple sources.

Currently, there are two mutually exclusive variants of Stackdriver available for use with GKE clusters: Legacy Stackdriver Support and Stackdriver Kubernetes Engine Monitoring Support.

Although Stackdriver Kubernetes Engine Monitoring is the preferred option, starting with GKE versions 1.12.7 and 1.13, Legacy Stackdriver is the default option up through GKE version 1.13. The use of either of these services is sufficient to pass the benchmark recommendation.

However, note that as Legacy Stackdriver Support is not getting any improvements and lacks features present in Stackdriver Kubernetes Engine Monitoring, Legacy Stackdriver Support may be deprecated in favor of Stackdriver Kubernetes Engine Monitoring Support in future versions of this benchmark.

Run the following commands:

gcloud container clusters describe [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --format json | jq '.loggingService'

gcloud container clusters describe [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --format json | jq '.monitoringService'

The output of the above commands should return logging.googleapis.com/kubernetes and monitoring.googleapis.com/kubernetes respectively if Stackdriver Kubernetes Engine Monitoring is Enabled.

To enable Stackdriver Kubernetes Engine Monitoring for an existing cluster, run the following command:

gcloud container clusters update [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --enable-stackdriver-kubernetes
Enable Linux Auditd logging

Run the Auditd logging daemon to obtain verbose operating system logs from GKE nodes running Container-Optimized OS (COS).

Auditd logs provide valuable information about the state of the cluster and workloads, such as error messages, login attempts, and binary executions. This information can be used to debug issues or to investigate security incidents.

If using the unmodified example Auditd logging daemonset, run

kubectl get daemonsets -n cos-audit

and observe that the cos-auditd-logging daemonset is running as expected.

If the name or namespace of the daemonset has been modified and is unknown, you can search for the container being used by the daemonset:

kubectl get daemonsets -A -o json | jq '.items[] | select
(.spec.template.spec.containers[].image | contains ("gcr.io/stackdriver-
agents/stackdriver-logging-agent"))'| jq '{name: .metadata.name, annotations:
.metadata.annotations."kubernetes.io/description", namespace:
.metadata.namespace, status: .status}'

The above command returns the name, namespace and status of the daemonsets that use the Stackdriver logging agent. The example Auditd logging daemonset has a description within the annotation as output by the command above:

{
  "name": "cos-auditd-logging",
  "annotations": "DaemonSet that enables Linux auditd logging on COS nodes.",
  "namespace": "cos-auditd",
  "status": {...
  }
}

Confirm the status fields show that the daemonset is running as expected.

Download the example manifests:

curl https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-node-
tools/master/os-audit/cos-auditd-logging.yaml > cos-auditd-logging.yaml

Edit the example manifests if needed. Then, deploy them:

kubectl apply -f cos-auditd-logging.yaml

Confirm the logging Pods have started. If you defined a different Namespace in your manifests, replace cos-auditd with the name of the namespace you're using:

kubectl get pods --namespace=cos-auditd

Authentication and authorization

Ensure Basic Authentication using static passwords is disabled

Disable Basic Authentication (basic auth) for API server authentication as it uses static passwords which need to be rotated.

Basic Authentication allows a user to authenticate to a Kubernetes cluster with a username and static password which is stored in plaintext (without any encryption). Disabling Basic Authentication will prevent attacks like brute force and credential stuffing. It is recommended to disable Basic Authentication and instead use another authentication method such as OpenID Connect.

GKE manages authentication via Gcloud using the OpenID Connect token method, setting up the Kubernetes configuration, getting an access token, and keeping it up to date. This means Basic Authentication using static passwords and Client Certificate authentication, which both require additional management overhead of key management and rotation, are not necessary and should be disabled.

When Basic Authentication is disabled, you will still be able to authenticate to the cluster with other authentication methods, such as OpenID Connect tokens. See also Recommendation 6.8.2 to disable authentication using Client Certificates.

To check Basic Authentication status for an existing cluster's nodes, run the following command:

gcloud container clusters describe [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --format json | jq '.masterAuth.password and .masterAuth.username'

The output of the above command should return false, if Basic Authentication is disabled. If Basic Authentication is enabled, the above command will return true.

To update an existing cluster and disable Basic Authentication by removing the static password:

gcloud container clusters update [CLUSTER_NAME] \
    --no-enable-basic-auth
Ensure authentication using Client Certificates is disabled

Disable Client Certificates, which require certificate rotation, for authentication. Instead, use another authentication method like OpenID Connect.

With Client Certificate authentication, a client presents a certificate that the API server verifies with the specified Certificate Authority. In GKE, Client Certificates are signed by the cluster root Certificate Authority. When retrieved, the Client Certificate is only base64 encoded and not encrypted.

GKE manages authentication via Gcloud for you using the OpenID Connect token method, setting up the Kubernetes configuration, getting an access token, and keeping it up to date. This means Basic Authentication using static passwords and Client Certificate authentication, which both require additional management overhead of key management and rotation, are not necessary and should be disabled.

When Client Certificate authentication is disabled, you will still be able to authenticate to the cluster with other authentication methods, such as OpenID Connect tokens. See also Recommendation 6.8.1 to disable authentication using static passwords, known as Basic Authentication.

To check that the client certificate has not been issued, run the following command:

gcloud container clusters describe [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --format json | jq '.masterAuth.clientKey'

The output of the above command returns null ({ }) if the client certificate has not been issued for the cluster (Client Certificate authentication is disabled).

Create a new cluster without a Client Certificate:

gcloud container clusters create [CLUSTER_NAME] \
  --no-issue-client-certificate
Manage Kubernetes RBAC users with Google Groups for GKE

Cluster Administrators should leverage G Suite Groups and Cloud IAM to assign Kubernetes user roles to a collection of users, instead of to individual emails using only Cloud IAM.

On- and off-boarding users is often difficult to automate and prone to error. Using a single source of truth for user permissions via G Suite Groups reduces the number of locations that an individual must be off-boarded from, and prevents users gaining unique permissions sets that increase the cost of audit.

Using G Suite Admin Console and Google Cloud Console:

  1. Navigate to manage G Suite Groups in the Google Admin console at https://admin.google.com

  2. Ensure there is a group named gke-security-groups@[yourdomain.com]. The group must be named exactly gke-security-groups

  3. Ensure only further groups (not individual users) are included in the gke-security-groups group as members

  4. Go to the Kubernetes Engine

  5. From the list of clusters, click on the desired cluster. In the Details pane, make sure Google Groups for RBAC is set to Enabled.

Follow the G Suite Groups instructions.

Then, create a cluster with:

gcloud beta container clusters create my-cluster \
  --security-group="gke-security-groups@[yourdomain.com]"

Finally create Roles, ClusterRoles, RoleBindings, and ClusterRoleBindings that reference your G Suite Groups.

Ensure Legacy Authorization (ABAC) is Disabled

Legacy Authorization, also known as Attribute-Based Access Control (ABAC) has been superseded by Role-Based Access Control (RBAC) and is not under active development. RBAC is the recommended way to manage permissions in Kubernetes.

In Kubernetes, RBAC is used to grant permissions to resources at the cluster and namespace level. RBAC allows you to define roles with rules containing a set of permissions, whilst the legacy authorizer (ABAC) in Kubernetes Engine grants broad, statically defined permissions. As RBAC provides significant security advantages over ABAC, it is recommended option for access control. Where possible, legacy authorization must be disabled for GKE clusters.

To check Legacy Authorization status for an existing cluster, run the following command:

gcloud container clusters describe [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --format json | jq '.legacyAbac'

The output should return null ({}) if Legacy Authorization is disabled. If Legacy Authorization is enabled, the above command will return true value.

To disable Legacy Authorization for an existing cluster, run the following command:

gcloud container clusters update [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --no-enable-legacy-authorization

Storage

Enable Customer-Managed Encryption Keys (CMEK) for GKE Persistent Disks (PD)

Use Customer-Managed Encryption Keys (CMEK) to encrypt node boot and dynamically-provisioned attached Google Compute Engine Persistent Disks (PDs) using keys managed within Cloud Key Management Service (Cloud KMS).

GCE persistent disks are encrypted at rest by default using envelope encryption with keys managed by Google. For additional protection, users can manage the Key Encryption Keys using Cloud KMS.

For Node Boot Disks:

Run this command:

gcloud beta container node-pools describe [NODE_POOL_NAME] \
  --cluster [CLUSTER_NAME]

Confirm the output of the above command includes a diskType of either pd-standard or pd-ssd, and the bootDiskKmsKey is specified as the desired key.

For Attached Disks:

Identify the Persistent Volumes Used by the cluster:

kubectl get pv -o json | jq '.items[].metadata.name'

For each volume used, check that it is encrypted using a customer managed key by running the following command:

gcloud compute disks describe [PV_NAME] \
  --zone [COMPUTE-ZONE] \
  --format json | jq '.diskEncryptionKey.kmsKeyName'

This returns null ({ }) if a customer-managed encryption key is not used to encrypt the disk.

For Node Boot Disks:

Create a new node pool using customer-managed encryption keys for the node boot disk, of [DISK_TYPE] either pd-standard or pd-ssd:

gcloud beta container node-pools create [CLUSTER_NAME] \
--disk-type [DISK_TYPE] \
--boot-disk-kms-key
projects/[KEY_PROJECT_ID]/locations/[LOCATION]/keyRings/[RING_NAME]/cryptoKey
s/[KEY_NAME]

Create a cluster using customer-managed encryption keys for the node boot disk, of [DISK_TYPE] either pd-standard or pd-ssd:

gcloud beta container clusters create [CLUSTER_NAME] \
--disk-type [DISK_TYPE] \
--boot-disk-kms-key
projects/[KEY_PROJECT_ID]/locations/[LOCATION]/keyRings/[RING_NAME]/cryptoKey
s/[KEY_NAME]

For Attached Disks:

Follow the instructions detailed at https://cloud.google.com/kubernetes-engine/docs/how-to/using-cmek.

Other cluster configurations

Ensure Kubernetes Web UI is disabled

The Kubernetes Web UI (Dashboard) has been a historical source of vulnerability and should only be deployed when necessary.

You should disable the Kubernetes Web UI (Dashboard) when running on Kubernetes Engine. The Kubernetes Web UI is backed by a highly privileged Kubernetes Service Account.

The Google Cloud Console provides all the required functionality of the Kubernetes Web UI and leverages Cloud IAM to restrict user access to sensitive cluster controls and settings.

Run the following command:

gcloud container clusters describe [CLUSTER_NAME] \
  --zone [ZONE] \
  --format json | jq '.addonsConfig.kubernetesDashboard'

Ensure the output of the above command has JSON key attribute disabled set to true:

{
  "disabled": true
}

To disable the Kubernetes Dashboard on an existing cluster, run the following command:

gcloud container clusters update [CLUSTER_NAME] \
  --zone [ZONE] \
  --update-addons=KubernetesDashboard=DISABLED
Ensure Alpha clusters are not used for production workloads

Alpha clusters are not covered by an SLA and are not production-ready.

Alpha clusters are designed for early adopters to experiment with workloads that take advantage of new features before those features are production-ready. They have all Kubernetes API features enabled, but are not covered by the GKE SLA, do not receive security updates, have node auto-upgrade and node auto-repair disabled, and cannot be upgraded. They are also automatically deleted after 30 days.

Run the command:

gcloud container clusters describe [CLUSTER_NAME] \
  --zone [COMPUTE-ZONE] \
  --format json | jq '.enableKubernetesAlpha'

The output of the above command will return true if it is an Alpha cluster.

Upon creating a new cluster:

gcloud container clusters create [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE]

Do not use the --enable-kubernetes-alpha argument.

Ensure Pod Security Policy is Enabled and set as appropriate

Pod Security Policy should be used to prevent privileged containers where possible and enforce namespace and workload configurations.

A Pod Security Policy is a cluster-level resource that controls security sensitive aspects of the pod specification. A PodSecurityPolicy object defines a set of conditions that a pod must run with in order to be accepted into the system, as well as defaults for the related fields. When a request to create or update a Pod does not meet the conditions in the Pod Security Policy, that request is rejected and an error is returned. The Pod Security Policy admission controller validates requests against available Pod Security Policies.

Pod Security Policies specify a list of restrictions, requirements, and defaults for Pods created under the policy.

To check Pod Security Policy Admission Controller is enabled for an existing cluster, run the following command,

gcloud beta container clusters describe [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --format json | jq '.podSecurityPolicyConfig'

Ensure the output of the above command has JSON key attribute enabled set to true:

{
  "enabled": true
}

If Pod Security Policy is disabled, the above command output will return null ({ }).

To enable Pod Security Policy for an existing cluster, run the following command:

gcloud beta container clusters update [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --enable-pod-security-policy
Consider GKE Sandbox for running untrusted workloads

Use GKE Sandbox to restrict untrusted workloads as an additional layer of protection when running in a multi-tenant environment.

GKE Sandbox provides an extra layer of security to prevent untrusted code from affecting the host kernel on your cluster nodes.

When you enable GKE Sandbox on a Node pool, a sandbox is created for each Pod running on a node in that Node pool. In addition, nodes running sandboxed Pods are prevented from accessing other GCP services or cluster metadata. Each sandbox uses its own userspace kernel.

Multi-tenant clusters and clusters whose containers run untrusted workloads are more exposed to security vulnerabilities than other clusters. Examples include SaaS providers, web-hosting providers, or other organizations that allow their users to upload and run code. A flaw in the container runtime or in the host kernel could allow a process running within a container to 'escape' the container and affect the node's kernel, potentially bringing down the node.

The potential also exists for a malicious tenant to gain access to and exfiltrate another tenant's data in memory or on disk, by exploiting such a defect.

gcloud container node-pools describe [NODE_POOL_NAME] \
  --zone [COMPUTE-ZONE] \
  --cluster [CLUSTER_NAME] \
  --format json | jq '.config.sandboxConfig'

The output of the above command will return the following if the Node pool is running a sandbox:

{
  "sandboxType":"gvisor"
}

If there is no sandbox, the above command output will be null ({ }). The default node pool cannot use GKE Sandbox.

To enable GKE Sandbox on an existing cluster, a new Node pool must be created.

gcloud container node-pools create [NODE_POOL_NAME] \
  --zone=[COMPUTE-ZONE] \
  --cluster=[CLUSTER_NAME] \
  --image-type=cos_containerd \
  --sandbox type=gvisor
Ensure use of Binary Authorization

Binary Authorization helps to protect supply-chain security by only allowing images with verifiable cryptographically signed metadata into the cluster.

Binary Authorization provides software supply-chain security for images that you deploy to GKE from Google Container Registry (GCR) or another container image registry.

Binary Authorization requires images to be signed by trusted authorities during the development process. These signatures are then validated at deployment time. By enforcing validation, you can gain tighter control over your container environment by ensuring only verified images are integrated into the build-and-release process.

Firstly, check that Binary Authorization is enabled for the GKE cluster:

gcloud container clusters describe [CLUSTER_NAME] \
  --zone [COMPUTE_ZONE] \
  --format json | jq .binaryAuthorization

The above command output will be the following if Binary Authorization is enabled:

{
  "enabled": true
}

Then, assess the contents of the policy:

gcloud container binauthz policy export > current-policy.yaml

Ensure that the current policy is not configured to allow all images (evaluationMode: ALWAYS_ALLOW):

cat current-policy.yaml
...
defaultAdmissionRule:
  evaluationMode: ALWAYS_ALLOW

Firstly, update the cluster to enable Binary Authorization:

gcloud container cluster update [CLUSTER_NAME] \
  --zone [COMPUTE-ZONE] \
  --enable-binauthz

Create a Binary Authorization Policy using the Binary Authorization Policy Reference (https://cloud.google.com/binary-authorization/docs/policy-yaml-reference) for guidance.

Import the policy file into Binary Authorization:

gcloud container binauthz policy import [YAML_POLICY]
Enable Cloud Security Command Center (Cloud SCC)

Enable Cloud Security Command Center (Cloud SCC) to provide a centralized view of security for your GKE clusters.

Cloud Security Command Center (Cloud SCC) is the canonical security and data risk database for GCP. Cloud SCC enables you to understand your security and data attack surface by providing asset inventory, discovery, search, and management.

To determine if Cloud SCC is enabled and indexing assets, observe the output of the following command:

gcloud alpha scc assets list [PROJECT_ID]

If the output of the above command returns GKE assets, Cloud SCC is enabled and indexing GKE resources.

Managed services – AWS EKS

Image registry and image scanning

Ensure Image Vulnerability Scanning using Amazon ECR image scanning or a third party provider

Scan images being deployed to Amazon EKS for vulnerabilities.

Vulnerabilities in software packages can be exploited by hackers or malicious users to obtain unauthorized access to local cloud resources. Amazon ECR and other third party products allow images to be scanned for known vulnerabilities.

Please follow AWS ECS or your 3rd party image scanning provider's guidelines for enabling Image Scanning.

To utilize AWS ECR for Image scanning follow the steps below:

To create a repository configured for scan on push (AWS CLI)

aws ecr create-repository --repository-name $REPO_NAME --image-scanning-
configuration scanOnPush=true --region $REGION_CODE

To edit the settings of an existing repository (AWS CLI)

aws ecr put-image-scanning-configuration --repository-name $REPO_NAME --
image-scanning-configuration scanOnPush=true --region $REGION_CODE

Use the following steps to start a manual image scan using the AWS Management Console.

  1. Open the Amazon ECR console at https://console.aws.amazon.com/ecr/repositories.

  2. From the navigation bar, choose the Region to create your repository in.

  3. In the navigation pane, choose Repositories.

  4. On the Repositories page, choose the repository that contains the image to scan.

  5. On the Images page, select the image to scan and then choose Scan.

Minimize user access to Amazon ECR

Restrict user access to Amazon ECR, limiting interaction with build images to only authorized personnel and service accounts.

Weak access control to Amazon ECR may allow malicious users to replace built images with vulnerable containers.

Before you use IAM to manage access to Amazon ECR, you should understand what IAM features are available to use with Amazon ECR. To get a high-level view of how Amazon ECR and other AWS services work with IAM, see AWS Services That Work with IAM in the IAM User Guide.

Topics

  • Amazon ECR Identity-Based Policies
  • Amazon ECR Resource-Based Policies
  • Authorization Based on Amazon ECR Tags
  • Amazon ECR IAM Roles

Amazon ECR Identity-Based Policies

With IAM identity-based policies, you can specify allowed or denied actions and resources as well as the conditions under which actions are allowed or denied. Amazon ECR supports specific actions, resources, and condition keys. To learn about all of the elements that you use in a JSON policy, see IAM JSON Policy Elements Reference in the IAM User Guide.

Actions:

The Action element of an IAM identity-based policy describes the specific action or actions that will be allowed or denied by the policy. Policy actions usually have the same name as the associated AWS API operation. The action is used in a policy to grant permissions to perform the associated operation.

Policy actions in Amazon ECR use the following prefix before the action: ecr:. For example, to grant someone permission to create an Amazon ECR repository with the Amazon ECR CreateRepository API operation, you include the ecr:CreateRepository action in their policy. Policy statements must include either an Action or NotAction element. Amazon ECR defines its own set of actions that describe tasks that you can perform with this service.

To specify multiple actions in a single statement, separate them with commas as follows: "Action": [ "ecr:action1", "ecr:action2"

You can specify multiple actions using wildcards (). For example, to specify all actions that begin with the word Describe, include the following action: "Action": "ecr:Describe"

To see a list of Amazon ECR actions, see Actions, Resources, and Condition Keys for Amazon Elastic Container Registry in the IAM User Guide.

Resources:

The Resource element specifies the object or objects to which the action applies. Statements must include either a Resource or a NotResource element. You specify a resource using an ARN or using the wildcard (*) to indicate that the statement applies to all resources.

An Amazon ECR repository resource has the following ARN: arn:\({Partition}:ecr:\):\({Account}:repository/\) For more information about the format of ARNs, see Amazon Resource Names (ARNs) and AWS Service Namespaces.

For example, to specify the my-repo repository in the us-east-1 Region in your statement, use the following ARN: "Resource": "arn:aws:ecr:us-east-1:123456789012:repository/my-repo"

To specify all repositories that belong to a specific account, use the wildcard (): "Resource": "arn:aws:ecr:us-east-1:123456789012:repository/"

To specify multiple resources in a single statement, separate the ARNs with commas. "Resource": [ "resource1", "resource2"

To see a list of Amazon ECR resource types and their ARNs, see Resources Defined by Amazon Elastic Container Registry in the IAM User Guide. To learn with which actions you can specify the ARN of each resource, see Actions Defined by Amazon Elastic Container Registry.

Condition Keys:

The Condition element (or Condition block) lets you specify conditions in which a statement is in effect. The Condition element is optional. You can build conditional expressions that use condition operators, such as equals or less than, to match the condition in the policy with values in the request.

If you specify multiple Condition elements in a statement, or multiple keys in a single Condition element, AWS evaluates them using a logical AND operation. If you specify multiple values for a single condition key, AWS evaluates the condition using a logical OR operation. All of the conditions must be met before the statement's permissions are granted.

You can also use placeholder variables when you specify conditions. For example, you can grant an IAM user permission to access a resource only if it is tagged with their IAM user name. For more information, see IAM Policy Elements: Variables and Tags in the IAM User Guide.

Amazon ECR defines its own set of condition keys and also supports using some global condition keys. To see all AWS global condition keys, see AWS Global Condition Context Keys in the IAM User Guide.

Most Amazon ECR actions support the aws:ResourceTag and ecr:ResourceTag condition keys. For more information, see Using Tag-Based Access Control.

To see a list of Amazon ECR condition keys, see Condition Keys Defined by Amazon Elastic Container Registry in the IAM User Guide. To learn with which actions and resources you can use a condition key, see Actions Defined by Amazon Elastic Container Registry.

Minimize cluster access to read-only for Amazon ECR

Configure the Cluster Service Account with Storage Object Viewer Role to only allow read- only access to Amazon ECR.

The Cluster Service Account does not require administrative access to Amazon ECR, only requiring pull access to containers to deploy onto Amazon EKS. Restricting permissions follows the principles of least privilege and prevents credentials from being abused beyond the required role.

Review AWS ECS worker node IAM role (NodeInstanceRole) IAM Policy Permissions to Confirm they are set and the minimum required level.

If utilizing a 3rd-party tool to scan images utilize the minimum required permission level required to interact with the cluster – generally this should be read-only.

You can use your Amazon ECR images with Amazon EKS, but you need to satisfy the following prerequisites.

The Amazon EKS worker node IAM role (NodeInstanceRole) that you use with your worker nodes must possess the following IAM policy permissions for Amazon ECR.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:BatchGetImage",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetAuthorizationToken"
            ],
            "Resource": "*"
        }
    ]
}
Minimize Container Registries to only those approved

Allowing unrestricted access to external container registries provides the opportunity for malicious or unapproved containers to be deployed into the cluster. Allowlisting only approved container registries reduces this risk.

All container images to be deployed to the cluster must be hosted within an approved container image registry.

Identity and access management (IAM)

Prefer using dedicated EKS Service Accounts

Kubernetes workloads should not use cluster node service accounts to authenticate to Amazon EKS APIs. Each Kubernetes workload that needs to authenticate to other AWS services using AWS IAM should be provisioned with a dedicated Service account.

Manual approaches for authenticating Kubernetes workloads running on Amazon EKS against AWS APIs are: storing service account keys as a Kubernetes secret (which introduces manual key rotation and potential for key compromise); or use of the underlying nodes' IAM Service account, which violates the principle of least privilege on a multi-tenanted node, when one pod needs to have access to a service, but every other pod on the node that uses the Service account does not.

For each namespace in the cluster, review the rights assigned to the default service account and ensure that it has no roles or cluster roles bound to it apart from the defaults. Additionally ensure that the automountServiceAccountToken: false setting is in place for each default service account.

With IAM roles for service accounts on Amazon EKS clusters, you can associate an IAM role with a Kubernetes service account. This service account can then provide AWS permissions to the containers in any pod that uses that service account. With this feature, you no longer need to provide extended permissions to the worker node IAM role so that pods on that node can call AWS APIs.

Applications must sign their AWS API requests with AWS credentials. This feature provides a strategy for managing credentials for your applications, similar to the way that Amazon EC2 instance profiles provide credentials to Amazon EC2 instances. Instead of creating and distributing your AWS credentials to the containers or using the Amazon EC2 instance’s role, you can associate an IAM role with a Kubernetes service account. The applications in the pod’s containers can then use an AWS SDK or the AWS CLI to make API requests to authorized AWS services.

The IAM roles for service accounts feature provides the following benefits:

Least privilege

By using the IAM roles for service accounts feature, you no longer need to provide extended permissions to the worker node IAM role so that pods on that node can call AWS APIs. You can scope IAM permissions to a service account, and only pods that use that service account have access to those permissions. This feature also eliminates the need for third-party solutions such as kiam or kube2iam.

Credential isolation

A container can only retrieve credentials for the IAM role that is associated with the service account to which it belongs. A container never has access to credentials that are intended for another container that belongs to another pod.

Auditability

Access and event logging is available through CloudTrail to help ensure retrospective auditing.

AWS key management service (KMS)

Ensure Kubernetes Secrets are encrypted using Customer Master Keys (CMKs) managed in AWS KMS

Encrypt Kubernetes secrets, stored in etcd, using secrets encryption feature during Amazon EKS cluster creation.

Kubernetes can store secrets that pods can access via a mounted volume. Today, Kubernetes secrets are stored with Base64 encoding, but encrypting is the recommended approach. Amazon EKS clusters version 1.13 and higher support the capability of encrypting your Kubernetes secrets using AWS Key Management Service (KMS) Customer Managed Keys (CMK). The only requirement is to enable the encryption provider support during EKS cluster creation.

Use AWS Key Management Service (KMS) keys to provide envelope encryption of Kubernetes secrets stored in Amazon EKS. Implementing envelope encryption is considered a security best practice for applications that store sensitive data and is part of a defense in depth security strategy.

Application-layer Secrets Encryption provides an additional layer of security for sensitive data, such as user defined Secrets and Secrets required for the operation of the cluster, such as service account keys, which are all stored in etcd.

Using this functionality, you can use a key, that you manage in AWS KMS, to encrypt data at the application layer. This protects against attackers in the event that they manage to gain access to etcd.

For Amazon EKS clusters with Secrets Encryption enabled, look for 'encryptionConfig' configuration when you run:

aws eks describe-cluster --name="<cluster-name>"

Enable 'Secrets Encryption' during Amazon EKS cluster creation.

Creating an Amazon EKS cluster
Encrypting Kubernetes secrets

Cluster networking

Restrict access to the control plane endpoint

Enable Endpoint Private Access to restrict access to the cluster’s control plane to only an allowlist of authorized IPs.

Authorized networks are a way of specifying a restricted range of IP addresses that are permitted to access your cluster’s control plane. Kubernetes Engine uses both Transport Layer Security (TLS) and authentication to provide secure access to your cluster’s control plane from the public internet. This provides you the flexibility to administer your cluster from anywhere; however, you might want to further restrict access to a set of IP addresses that you control. You can set this restriction by specifying an authorized network.

Restricting access to an authorized network can provide additional security benefits for your container cluster, including:

Better protection from outsider attacks

Authorized networks provide an additional layer of security by limiting external access to a specific set of addresses you designate, such as those that originate from your premises. This helps protect access to your cluster in the case of a vulnerability in the cluster’s authentication or authorization mechanism.

Better protection from insider attacks

Authorized networks help protect your cluster from accidental leaks of master certificates from your company’s premises. Leaked certificates used from outside Amazon EC2 and outside the authorized IP ranges (for example, from addresses outside your company) are still denied access.

Input:

aws eks describe-cluster \
    --region <region> \
    --name <clustername>

Output:

...
"endpointPublicAccess": false,
"endpointPrivateAccess": true,
"publicAccessCidrs": [
    "203.0.113.5/32"
]
...

Complete the following steps using the AWS CLI version 1.18.10 or later. You can check your current version with aws --version. To install or upgrade the AWS CLI, see Installing the AWS CLI.

Update your cluster API server endpoint access with the following AWS CLI command. Substitute your cluster name and desired endpoint access values. If you set endpointPublicAccess=true, then you can (optionally) enter single CIDR block, or a comma- separated list of CIDR blocks for publicAccessCidrs. The blocks cannot include reserved addresses. If you specify CIDR blocks, then the public API server endpoint will only receive requests from the listed blocks. There is a maximum number of CIDR blocks that you can specify. For more information, see Amazon EKS Service Quotas. If you restrict access to your public endpoint using CIDR blocks, it is recommended that you also enable private endpoint access so that worker nodes and Fargate pods (if you use them) can communicate with the cluster. Without the private endpoint enabled, your public access endpoint CIDR sources must include the egress sources from your VPC. For example, if you have a worker node in a private subnet that communicates to the internet through a NAT Gateway, you will need to add the outbound IP address of the NAT gateway as part of a whitelisted CIDR block on your public endpoint. If you specify no CIDR blocks, then the public API server endpoint receives requests from all (0.0.0.0/0) IP addresses.

The following command enables private access and public access from a single IP address for the API server endpoint. Replace 203.0.113.5/32 with a single CIDR block, or a comma- separated list of CIDR blocks that you want to restrict network access to.

Example command:

aws eks update-cluster-config \
    --region region-code \
    --name dev \
    --resources-vpc-config \
    endpointPublicAccess=true, \
    publicAccessCidrs="203.0.113.5/32",\
    endpointPrivateAccess=true

Output:

{
    "update": {
        "id": "e6f0905f-a5d4-4a2a-8c49-EXAMPLE00000",
        "status": "InProgress",
        "type": "EndpointAccessUpdate",
        "params": [
            {
              "type": "EndpointPublicAccess",
              "value": "true"
            },
            {
              "type": "EndpointPrivateAccess",
              "value": "true"
            },
            {
              "type": "publicAccessCidrs",
              "value": "[\203.0.113.5/32\"]"
            }
        ],
        "createdAt": 1576874258.137,
        "errors": []
}
Ensure clusters are created with Private Endpoint Enabled and Public Access Disabled

Disable access to the Kubernetes API from outside the node network if it is not required.

In a private cluster, the master node has two endpoints, a private and public endpoint. The private endpoint is the internal IP address of the master, behind an internal load balancer in the master’s VPC network. Nodes communicate with the master using the private endpoint. The public endpoint enables the Kubernetes API to be accessed from outside the master’s VPC network.

Although Kubernetes API requires an authorized token to perform sensitive actions, a vulnerability could potentially expose the Kubernetes publically with unrestricted access. Additionally, an attacker may be able to identify the current cluster and Kubernetes API version and determine whether it is vulnerable to an attack. Unless required, disabling public endpoint will help prevent such threats, and require the attacker to be on the master’s VPC network to perform any attack on the Kubernetes API.

Configure the EKS cluster endpoint to be private.

Amazon EKS cluster endpoint access control