Handling Kubernetes vulnerabilities with CIS Benchmark

CIS (Center for Internet Security) is an independent, nonprofit organisation responsible for creating worldwide recognised security best practices, benchmarks, and controls. Regarding Kubernetes, they have benchmarks for securing Kubernetes clusters in the major cloud providers like AWS and on standalone installations.

In this article, I will use a tool called kube-bench to evaluate a Kubernetes cluster for vulnerabilities covered in the CIS benchmark. I will show how to access the kube-bench report and fix some vulnerabilities.

Generate the vulnerability report

To generate the report using the command-line, install the kube-bench binary according to your operating system:

> curl -L https://github.com/aquasecurity/kube-bench/releases/download/v0.7.3/kube-bench_0.7.3_linux_amd64.deb -o kube-bench_0.7.3_linux_amd64.deb

> sudo apt install ./kube-bench_0.7.3_linux_amd64.deb -f

To generate the vulnerability report for the Kubernetes cluster, execute kube-bench run:

>kube-bench run

[[INFO] 1 Master Node Security Configuration
[INFO] 1.1 Master Node Configuration Files
[PASS] 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated)
[PASS] 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated)
[PASS] 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated)
...
== Remediations node
4.2.6 If using a Kubelet config file, edit the file to set protectKernelDefaults: true.
If using command line arguments, edit the kubelet service file
/lib/systemd/system/kubelet.service on each worker node and
set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable.
--protect-kernel-defaults=true
Based on your system, restart the kubelet service. For example:
systemctl daemon-reload
systemctl restart kubelet.service
...
== Summary total ==
68 checks PASS
12 checks FAIL
43 checks WARN
0 checks INFO

Another installation option is to run kube-bench as a Job in the Kubernetes cluster. For this, create a local file based on the the job.yaml file from https://github.com/aquasecurity/kube-bench/blob/main/job.yaml and apply it to you cluster:

> kubectl apply -f job.yamljob.batch/kube-bench created

A Pod will be created and generate the kube-bench report. To visualize the report, access the Pod’s logs:

> kubectl logs kube-bench-wddkj

The output will be the same as running with the command-line. The report has five parts: master, etcd, control plane, worker (node), and policies. The report's summary indicates that 12 checks have failed and 43 have triggered warnings. We will now address some of these issues.

How to fix CIS vulnerabilities

The kube-bench report provides us with recommendation to fix the vulnerabilities. In this subsection, we will solve two vulnerabilities. One for the control plane and another one for a worker node.

Control Plane

In this example, we will solve the vulnerability 1.1.12:

[FAIL] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated)

In the report, you can see the recommendation on how to fix this vulnerability:

1.1.12 On the etcd server node, get the etcd data directory, passed as an argument --data-dir, from the below command:

ps -ef | grep etcd

Run the below command (based on the etcd data directory found above).

For example, chown etcd:etcd /var/lib/etcd

Execute the instructions as directed. First, in the control plane, locate the data-dir directory for etcd:

> ps -ef | grep etcd

root        1594    1296  2 09:24 ?        00:01:05 etcd --advertise-client-urls=https://172.31.119.105:2379 --cert-file=/etc/kubernetes/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd 

Change the permissions in the /var/lib/etcd and execute kube-bench and filter by the vulnerability 1.1.12:

>chown etcd:etcd /var/lib/etcd

>kube-bench run | grep -i 1.1.12[PASS] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated)

Now, the vulnerability for the etcd data-dir directory ownership is resolved.

Worker node

For the worker, the vulnerability 4.2.10 must be solved:

[FAIL] 4.2.10 Ensure that the --rotate-certificates argument is not set to false (Automated)

Reviewing the recommendations:

4.2.10 If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or remove it altogether to use the default value.

If using command line arguments, edit the kubelet service file/lib/systemd/system/kubelet.service on each worker node andremove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS variable.

Based on your system, restart the kubelet service. For example,

systemctl daemon-reload
systemctl restart kubelet.service

To fix this we need to log in into the worker node:

> ssh <user>@<node_name/ip address>

Once logged in the worker node, find the kubelet config file:

> ps -auxwww | grep -i kubeletroot        4016  1.3  4.2 2140228 83988 ?       Ssl  19:00   0:14 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9

Open the config file in the editor:

> sudo vi /var/lib/kubelet/config.yaml

Change the --rotate-certificates=false parameter to true:

rotateCertificates: false  #change from false to true

Restart the kubelet service:

> sudo systemctl daemon-reload
> sudo systemctl restart kubelet

Return to the control plane and generate a new report to validate the changes. To do that with the Kubernetes job, delete the current job and apply it again:

> kubectl delete jobs.batch kube-bench
job.batch "kube-bench" deleted

> kubectl apply -f job.yaml
job.batch/kube-bench created

In the Pod’s log, search for the 4.2.10 vulnerability:

> kubectl logs kube-bench-n52f8 | grep -i 4.2.10
[PASS] 4.2.10 Ensure that the --rotate-certificates argument is not set to false (Automated)

One more vulnerability successfully resolved!

Summary

kube-bench is an easy-to-use and straightforward tool that helps administrators determine the current vulnerabilities of your Kubernetes cluster and define the priorities for tackling these vulnerabilities. The tool provides the necessary information on how to solve the vulnerabilities. Also, you can access the CIS Kubernetes Benchmarks reports and download a PDF complete with all the instructions.

Give kube-bench a chance and improve the security of your cluster today!