Container Risks Home Insights Blog Container Risks February 20, 2020 Container Risks Gaining Visibility into NIST SP 800-190, Part Six In part five of this series, I described how native AWS tools and third-party solutions can address orchestrator risks identified in section 3.3 of the NIST SP 800-190 Application Container Security Guide. In this post we’ll explore section 3.4 of the guide: Container Risks (3.4) and Container Countermeasures (4.4). We will again leverage our lab environment, which utilizes AWS EKS and Palo Alto Networks Prisma Cloud (formerly Twistlock). Container technologies, such as Docker, LXC and Heroku, allow developers to assemble code and associated dependencies into a single package or container image. Container images can be run within their respective engines (like Docker Engine) and deployed quickly using a container orchestration technology, such as Kubernetes, Docker Swarm or Nomad. Managing container security is key to the DevOps process. Areas like runtime software vulnerabilities, unbounded network access from containers, insecure container runtime configurations, application vulnerabilities and rogue containers should be thoroughly reviewed when planning a DevOps lifecycle involving container technology. 3.4.1 Vulnerabilities Within the Runtime Software In the containers context, vulnerabilities with runtime software present a lower risk profile. However, they can be especially damaging when factoring in the potential for a threat actor to use the vulnerability to escape the container and attack other resources. Damaging modifications or the planting of a trojan are tactics that should be accounted for. NIST SP 800-190 offers this guidance on Vulnerabilities within the Runtime Software: 3.4.1 Example Risks Vulnerabilities within the runtime software are particularly dangerous if they allow “container escape” scenarios in which malicious software can attack resources in other containers and the host OS itself. An attacker may also be able to exploit vulnerabilities to compromise the runtime software itself, and then alter that software so it allows the attacker to access other containers, monitor container-to-container communications, etc. 4.4.1 Countermeasures Initially assess all binaries within containers for vulnerabilities when they’re first uploaded into a container registry Continuously assess and re-assess containers for new vulnerabilities Integrate vulnerability scanning process with bug fix/ticketing system for efficiency and manageability Scanning containers for vulnerabilities addresses the issue of vulnerable runtime binaries. Utilizing policy, Palo Alto Networks’ Prisma Cloud can assess newly created containers before they’ve been instantiated. Any alert raised immediately blocks the container from being deployed. Prisma Cloud can also be configured to continually reassess containers for vulnerabilities at a predetermined scan interval, with the default being 24 hours. Figure 1 - Container Vulnerabilities - Runtime Compliance Figure 2 - Figure 2 – Monitor/Compliance (Containers) 3.4.2 Unbounded Network Access from Containers In most environments, communication between containers and the host OS is unrestricted. In the event of a compromised container, the lack of network segmentation could potentially expose other containers in the environment to attack. An example of this could be reconnaissance, exploitation and subsequent lateral movement to another host within the cluster. NIST SP 800-190 offers this guidance on Unbounded Network Access from Containers: 3.4.2 Example Risks By default, in most container runtimes, individual containers are able to access each other and the host OS over the network. If a container is compromised and acting maliciously, allowing this network traffic may expose other resources in the environment to risk. Egress network access is more complex to manage in a containerized environment because so much of the connection is virtualized between containers. Thus, traffic from one container to another may appear simply as encapsulated packets on the network without directly indicating the ultimate source, destination, or payload. Tools and operational processes that are not container aware are not able to inspect this traffic or determine whether it represents a threat. 4.4.2 Countermeasures Monitor and control egress traffic at network borders, ensuring containers are not able to send traffic across networks of differing sensitivity levels (e.g. trusted network to Internet, etc.), similar to how traditional networks are architected. Due to potential traffic encapsulation, employ technologies that are app aware, can filter container network connections and generate reporting /metrics for inbound network traffic and process-port bindings. Prisma Cloud’s Runtime Defense technology can build predictive models for each image in an environment. These network models can identify malicious ingress activity, like containers that create unexpected network listeners, as well as identifying malicious egress traffic connections to unexpected and known malicious destinations. Figure 3 - Runtime Defense Technology - Monitor/Runtime/Incident Explorer In addition, the built-in Cloud Native Network Firewall (CNNF) uses machine learning (ML) to model observed network traffic between containers. The CNNF uses this traffic observation to whitelist known good traffic. Visibility and inspection into east-west traffic minimizes the potential damage of a container breach by limiting the attacker’s ability to move laterally. Figure 4 - Runtime Defense Technology - Monitor/Events (CNNF) 3.4.3 Insecure Container Runtime Configurations Insecure container runtime configurations can impact an organization’s threat profile. Typically, the set of system calls is limited to only those required for safe operation. If the list is expanded, containers or possibly even the host OS can be at risk. Containers running in privileged mode, with its access to all devices on a host, can increase risk by exposing the host OS and all containers within. NIST 800-190 3.4.3 also addresses the risk from insecure containers’ ability to mount sensitive directories on the host OS. In an ideal setting, containers should be abstracted from the host OS and rarely, if ever, should be able to make changes to the host OS file system. They should also never be able to make change to locations that control a system’s basic functionality. In Linux containers, this would be /boot and in Windows containers, this would be C:\Windows. NIST SP 800-190 offers this guidance on Insecure Container Runtime Configurations: 3.4.3 Example Risks Containers runtimes with configurations that are overly permissive can increase risks to the host OS and other containers. Containers that have direct access to a host OS file system (e.g. mounting) is highly discouraged and can also raise the risk profile. 4.4.3 Countermeasures Using a tool like Prisma Cloud or Tenable, assess the container’s runtime configurations against the Center for Internet Security’s Docker Benchmark. Implement native mandatory access controls like SELinux and AppArmor to effectively raise the barrier around the container OS by controlling access to binaries, libraries and system calls. One way to measure compliance for runtime standards is to assess the container’s runtime configurations against the Center for Internet Security’s Docker Benchmark. The CIS Docker benchmark, used by Prisma Cloud, provides details on container options and recommended settings. Automation of such container runtime vulnerability assessments is recommended by NIST to help scale the assessment process. Within Prisma Cloud, Critical and High checks are enabled by default so security practitioners can focus on the most serious issues in their environment. Figure 5 - Monitor/Compliance (Containers) Native mandatory access controls within the OS, like SELinux and AppArmor, can provide additional security controls for containers running Linux OSs. These technologies help harden the container by constructing a zero-trust segmentation around the permitted binary and associated processes, allowing only specific file paths, processes and network sockets. Secure computing profiles (seccomp) are another method of constraining the permitted resources and protecting the container at runtime. As noted in NIST SP 800-190, “common container runtimes, like Docker, include default seccomp profiles that drop unsafe system calls that are unrequired for container operations.” Prisma Cloud Labs provides curated custom seccomp profiles for widely used containerized apps, like Apache and MongoDB. Figure 6 - Monitor/Runtime (Container Models) 3.4.4 App Vulnerabilities While the recommendations in the NIST SP 800-190 guide provide some coverage in addressing potential container risks, vulnerabilities in the applications that the containers run still add risks of compromise. When a container is compromised in this fashion, the countermeasures mentioned throughout the series become much more critical. 3.4.4 Example Risks Vulnerable applications within containers could lead to a compromise of the container. Compromising a vulnerable application within a container can help an attacker establish a foothold to other containers, sensitive information or even the host OS. 4.4.4 Countermeasures Deploy container-aware vulnerability assessment solutions designed to operate at scale. Solutions should be able to automatically profile containerized apps using behavioral and machine learning. Containers should be run with root filesystems in read-only mode, making the container more resilient to compromise since the filesystem is static and easily separated from the rest of the application. Prisma Cloud Runtime Defense helps secure containers by modeling each container image and then using the models to detect abnormal activity. Runtime Defense is enabled by default. Models have four dimensions: processes, networking, file system and system calls. Runtime Defense sensors can protect against all of the events outlined above: Invalid or unexpected process execution Invalid or unexpected system calls Changes to protected configuration files and binaries — writes to unexpected locations and file types Creation of unexpected network listeners Traffic sent to unexpected network destinations Malware storage or execution Figure 7 - Defend/Runtime (Container Policy) Prisma Cloud ships with CIS Docker Benchmark compliance check 5.12 automatically enabled. Wth the CIS Docker Compliance check in place, an alert will be raised if a container mounts its root filesystem with any permission other than read-only. 3.4.5 Rogue Containers Due to the dynamic nature of DevOps, where new systems and services are constantly being spun up and down, unsanctioned containers can be common within the development environment. These rogue containers, potentially vulnerable to attack, could be inadvertently introduced into production environments. 3.4.5 Example Risks Rogue containers are unplanned or unsanctioned containers in an environment. Commonly found in development environments, used as a means to test code. With lack of vulnerability scanning, they open organizations to additional risk. 4.4.5 Countermeasures Organizations should tightly segment their environments by purpose, specifically for development, staging/test and production. Container creation should be associated with individual user identities and logged to provide a clear audit trail of activity. Utilize tools that enforce baselines for vulnerability management and compliance prior to allowing them to run. Prisma Cloud addresses the concept of rogue containers with a number of controls. It’s role-based access controls allows for policies limiting who can access resources in an environment. Rules allow Prisma Cloud users to target specific hosts, images, containers and labels with pattern-matching expressions. Figure 8 - Defend/Access (Default Deny-All Policy) To provide additional visibility, Prisma Cloud logs all Docker and Kubernetes commands, along with the user who ran them. It also assesses containers for vulnerabilities and compliance issue before any container is run. For policy violations resulting in a raised alert, the container is blocked from operating. The NIST SP 800-190 Application Container Security Guide is an invaluable tool for any cybersecurity professionals looking to obtain greater visibility into container security risks. In the seventh and final blog in this series, we’ll consider the challenges surrounding host OS risks and countermeasures. By: Rob Brooks Senior Research Scientist Rob Brooks has been involved in Information Security for 20 years and has served as a CISO, Senior Architect, Sysadmin and Engineer along the way. Rob currently works as a Sr. Research Scientist in Optiv's R&D group, managing the company’s private cloud and helping research security products. Share: NIST Containers CyberOps NIST Series How Can We Help? Let us know what you need, and we will have an Optiv professional contact you shortly.