Every Solution You Can Imagine – and More
What cybersecurity solution do you need? From Zero Trust to ADR, IAM, risk/privacy, data protection, AppSec and threat, securing digital transformation, to resiliency and remediation, we can build the right program to help solve your challenges.
A Single Partner for Everything You Need
Optiv works with more than 450 world-class security technology partners. By putting you at the center of our unmatched ecosystem of people, products, partners and programs, we accelerate business progress like no other company can.
We Are Optiv
Greatness is every team working toward a common goal. Winning in spite of cyber threats and overcoming challenges in spite of them. It’s building for a future that only you can create or simply coming home in time for dinner.
However you define greatness, Optiv is in your corner. We manage cyber risk so you can secure your full potential.
Container Runtime Protection
October 29, 2021
Are you asking the right questions when it comes to selecting container security tools?
Over the last two years several endpoint security vendors have started offering coverage for container environments. Some of these vendors have internally developed container security features, while others have made acquisitions to get a jump on the space. How mature are these products? And how do they stack up to a purpose-built solution like Palo Alto Networks Prisma Cloud?
To get an understanding of the current state of these offerings, Optiv devised a quick runtime protection test plan. This included a Kubernetes environment for each endpoint security product tested. These clusters were built in AWS using EKS, and each EKS cluster was running the Weaveworks microservices application Sock Shop.
Note: Not all pods for the Sock Shop microservices application are displayed in the image.
Each product evaluated was installed on its own independent EKS cluster comprised of three ec2 nodes running Amazon Linux 2. Optiv used these environments to understand how far endpoint security vendors have come in providing protection and visibility into container runtime environments. Optiv also provisioned a cluster for Prisma Cloud. During the evaluation we were looking to see if each solution could provide detail around:
Runtime detection/prevention policy – out of the box
To make the attack scenario as level across all products as possible, Optiv focused on container runtime protection. We also took a few liberties when purposely misconfiguring one of the pods for the initial compromise. Note: while many container security products also focus on web application security, Optiv saw it as unfair to perform a web-based attack on the application, given the feature sets available from endpoint security tools.
In past blog posts, we used a well-documented container breakout that leverages cgroups. The cgroups exploit allows the attacker to move from the container to the host node. This time around we took a simpler approach and ran the container as root. Why is that simpler? If a container is compromised when it’s running as root the attacker can mount the host drive as a temp directory. Yep – that’s all it takes. It’s understood that there are multiple ways to stop this scenario – don’t run the pod as root, scan the image in the registry before deploying it, etc. But for this project we focused on an easy-to-replicate runtime protection use case.
Listed below are the exact attack commands that are executed through the initial c2 channel on the pod once the reverse shell executes at runtime.
mount /dev/nvme0n1p1 /mnt/demo
cp /usr/src/app/reverse_shell.elf /mnt/demo/bin/
echo "3 * * * * root /bin/reverse_shell.elf" >> /mnt/demo/etc/crontab
Note: After the commands were executed the attacker receives a connection from the EKS node in three minutes.
There have been two different approaches when it comes to gaining visibility and providing protections for containers running in Kubernetes environments. Each approach has benefits and downsides.
Option 1 DaemonSet
A DaemonSet deployment guarantees that the specified pod runs on all of the nodes that are part of the Kubernetes cluster. Benefits include ec2 autoscaling scenarios, node metric insight and node log collection. One of the biggest downsides is that DaemonSets will not work in managed container environments (like AWS Fargate).
Option 2 Sidecar
A sidecar deployment injects itself into each pod on the host. This is different because a DaemonSet deployment has a single deployment per host, regardless of the number of pods running on it. With the sidecar approach, injecting into every pod can cause additional overhead.
There was no common deployment theme. The evaluated endpoint vendors employed both approaches, with one even using a DaemonSet with a sidecar. Interestingly, each endpoint vendor evaluated required Helm for deployment; Palo Alto Networks has it as an option, but it isn’t necessary.
The policy differences between the runtime protection offered by endpoint products versus Prisma Cloud were quite obvious. For example, Optiv found that policies to prevent containers from running as root or with certain allowed capabilities were nonexistent in the endpoint security products tested. The endpoint security products have largely repurposed their existing Linux policies for container environments.
One product evaluated had limited prevention settings for their Linux policy. These preventions were repurposed for their container protections. Attempting to use minimal Linux preventions in container environments only further highlights the endpoint industry’s avoidance of Linux security. Legacy-focused policies don’t always meet the security runtime requirements necessary to compete with advanced attacks in container environments. This high-level evaluation also didn’t dive into infrastructure policy checks, which would only further highlight the problems with an endpoint approach to container security.
The endpoint products evaluated offered the ability to create custom rules for Linux systems, but there are limitations when attempting to port these over to a container environment. For example, these custom policies couldn’t be made namespace-specific. While there are many elements of endpoint security in containerized environments, treating containers as just another endpoint is the wrong approach.
Prisma Cloud and one of the endpoint products evaluated detected the malicious elf executing within the pod at runtime. A second endpoint product logged the commands but didn’t alert on the file execution or the c2 connection. We were able to locate the reverse_shell.elf execution along with the ip address of the c2 server in the logs across all the products tested.
Image: Prisma Cloud reverse_shell.elf detection.
As part of the scenario, the attacker mounts the ec2 node root drive as a temporary drive within the container. None of the products evaluated created an alert when this happened, but the commands were recorded in the logs of each product tested. It is also important to understand that custom detections can be created within each of the products to alert on this behavior.
File Copy and Crontab Edit
Once c2 was established, and the ec2 node’s root drive was mounted, the attack scenario copied the elf file to /bin directory. At this time the contab file of the ec2 node was edited. This edit happened through the container entrypoint to add a cronjob that executed the file as root every three minutes, offering the adversary persistence to the ec2 node. None of the products alerted to the file copy or that the crontab file was edited. Optiv was able to discover that the actions took place, along with the ip address of the c2 server in the logs for each product.
Note: After each product evaluation was completed Optiv shared the findings with the vendor. If any deployment or policy misconfigurations were made during the initial round of testing Optiv worked with each of the vendors to make corrections and retest.
The endpoint products we tested offered no visual depiction of cluster nodes, namespaces or the relationship between pods within a namespace. Again, this view was only found in Prisma Cloud. Optiv views this graphical depiction as a valuable resource.
Image: Prisma Cloud Radars/Container view of Optiv’s sock-shop namespace.
We were also surprised to discover that each endpoint product lacked a default timeline view of container events – an additional timesaver when investigating a compelling event. Instead, a specific log search would need to be performed to produce a similar view.
Image: Prisma Cloud forensic view of a single lab node.
Prisma was also the only solution to include an ATT&CK for Containers alignment. This demonstrates more alignment with container security compared to endpoint security vendors who have entered the container security space.
Image: Prisma Cloud ATT&CK for containers view.
While the endpoint market has begun providing products designed to provide insight and protection in the container space, there’s quite a bit of room for maturation before they can be considered container security solutions. Lack of policy separation between container and host, ability to create policies based on namespace and visualizations are just some of the areas that traditional endpoint vendors need to improve on.
For now, organizations grappling with container security should look toward those solutions that were purpose-built to protect container environments.
Want to know more? Talk to us.
September 16, 2020
Palo Alto Networks Prisma Compute can help minimize the risk of deploying untrusted images.
February 20, 2020
Container technologies allow developers to assemble code and associated dependencies into a single package or container image.
March 31, 2020
This container compromise scenario is difficult, but very plausible.
Let us know what you need, and we will have an Optiv professional contact you shortly.