January 06, 2020
Gaining Visibility into NIST SP 800-190, Part Three
In Part 2 of this series I discussed the hybrid cloud lab environment we built to evaluate container security solutions. Now that the base outline of the lab can be visualized let’s focus on what native AWS tools and third-party solutions are available to address the risks identified in NIST SP 800-190 Application Container Security Guide. This post will focus on section 3.1 (Image Risks) and 4.1 (Image Countermeasures).
3.1.1 Image Vulnerabilities: Images are collections of files which can be used to perform a function or even comprise an entire operating system. Like traditional operating systems, images used by containers may have vulnerabilities that pose a risk to an organization. NIST provides the following guidance on risks and countermeasures:
3.1.1 Example Risks
- An image may be missing critical security updates or are otherwise outdated
- Newly discovered vulnerabilities can pose issues to the image used at deployment
- Visibility into vulnerabilities at all layers of the image
- Policy enforcement at each stage of build and deployment
- Preventing the use of images with known vulnerabilities
We used EKS and EC2 instances for the worker nodes. Each node had FluentD as a DaemonSet to send logs to CloudWatch. After investigating the telemetry present in these logs, we found that the log stream captured operational activities within the cluster and did not contain image security details.
Prisma Cloud’s Vulnerabilities view allows users to quickly asses the images it has discovered by the number of vulnerabilities found and the risk each image poses to the organization. Shown in the screen capture below, the view details items such as the registry the image is located, its tag, which hosts the image can be found on and risk scores.
Further drill-down allows the administrator to view the type of vulnerability, component version, CVE, severity and the layer where the vulnerability exists.
To prevent vulnerable images from being deployed, Prisma Cloud Defender can be used to inspect images and enforce policies that would block images with vulnerabilities. As seen in the screenshot below, vulnerabilities rules can get extremely granular. The administrator has the ability to create severity-based actions, by resource, with additional conditions.
3.1.2 Image Configuration Defects: Beyond vulnerabilities, images can also contain configuration defects. These can be the result of a rushed job or lack of understanding on the implications of allowing images to run certain processes. NIST provides the following guidance on risks and countermeasures:
3.1.2 Example Risks
- Image running with excessive privileges
- Images contain sensitive information
- Image running unneeded SSH daemon
- Validation of image settings
- Continuous reporting and monitoring of image compliance state
- Use of base layers from trusted sources
- Use of remote tools is not enabled on images
Similar to 4.1.1, I didn’t find an AWS tool to provide countermeasures for image configuration defects.
Using the Compliance view under the Monitor section Prisma Cloud continuously reports the overall compliance state for all images. As shown in the screen capture below, Prisma Cloud will assess images against CIS benchmarks and report on any configuration defects found.
Users can drill deeper into each image to identify configuration defects. The screen capture shown below details a configuration risk defined by the CIS Docker benchmark.
Users can also define custom compliance rules. Ignore, alert, or block actions can be taken on events and resources specified by the administrator. As an example, containers running as root can be blocked for only specific containers.
3.1.3 Embedded malware: Since an image is just a collection of files packed together, it’s important to be able to determine if any of the files are malicious. Opportunistic attacks have been found on Docker Hub and other images repositories, where the malicious actor embedded malware on a base image in hopes of compromising systems through unsuspecting developers. NIST provides the following guidance on risks and countermeasures:
3.1.3 Example Risks
- Backdoor hidden inside MYSQL
- Backdoor hidden inside Tomcat
- Python Reverse shell
- Signature analysis
- Behavioral analysis
At this time AWS relies on third-party software to identify malware running containers.
There are several ways in which Prisma Cloud can identify images containing malware. The intelligence stream combined with the image scanner can detect malware within container images without any additional configuration. If malware is found it will log the information in the image scan report.
3.1.4 Embedded clear text secrets: Some applications require the use of secrets to communicate between containers. This communication is often secured with SSH keys or X.509 keys. Sometimes the private keys needed are stored on the image, which can put a company at risk for having the keys stolen. NIST provides the following guidance on risks and countermeasures:
3.1.4 Example Risks
- SSH private keys stored in image
- X.509 private keys stored in image
- Store keys outside of images
- Use of an orchestration tool to dynamically provide keys at runtime
- Encrypt secrets at rest and in transit
AWS Secrets Manager can be used to rotate, manage and retrieve secrets.
Prisma Cloud’s Compliance view can inform administrators when private keys or unencrypted secrets in environmental variables are stored with in an image. Users can then take steps to remove the keys and deploy a new image.
Prisma Cloud can also integrate with multiple secrets managers and inject secrets as variables into containers as needed.
3.1.5 Use of untrusted images: The use of untrusted images is a reckless practice that could place an organization at risk. More often than not these images aren’t validated and can contain misconfigured software, vulnerabilities and even malware, creating unwanted data exposure. NIST provides the following guidance on risks and countermeasures:
3.1.5 Example Risks
- Images from unknown or 3rd party repositories
- Private or trusted registries
- Repository monitoring
- Image validation
- Enforcement of host being able to run only certain images
Amazon Elastic Container Registry was not used as a part of this project. Both private and public registries were used to build and deploy images. No information could be found within AWS to let us know know if the images used were trusted or untrusted.
Prisma Cloud defines three image trust levels: Trusted, Partially Trusted and Untrusted. There are multiple ways to create a policy to trust an image. These can include registry origin, the image ID and base layer. A partially trusted image refers to an image that’s referenced in the console, but it isn’t explicitly trusted.
In our next post we’ll cover registry risks and countermeasures.