Home Insights Source Zero The Evolution of Cloud Forensics and Incident Response February 01, 2021 The Evolution of Cloud Forensics and Incident Response With the adoption of cloud infrastructure rapidly changing our attack landscape, we find ourselves with SOC and IR teams asking for some of the same visibility they had in the data center in the cloud. In some scenarios, the traditional SOC/IR tools are hardly practical or cost-effective. Due to the ephemeral nature of cloud infrastructure, a lot of the traditional security operations motions need to evolve to replicate the high-fidelity forensics that have given security teams the peace of mind to know what happened, on what host and what traffic traversed the cloud. In this blog we will leverage several native and third-party tools to capture malicious activities that can be performed on a cloud host to provide guidance on cloud forensics and the evolution of security operations and incident response. Overview of Infrastructure For this blog series, we will be focused on AWS and EC2 hosts, but many of the tools will be public and private cloud-agnostic. The initial visibility we would want aligns with tools traditionally used for configuration management. Who modified the firewall? What rules were changed? What were the permissions given to an IAM role? For this visibility, we will leverage Palo Alto Networks Prisma Cloud to replicate monitoring of an environment that an SOC would want in a centralized dashboard. Image Figure 1 - AWS Environment with Prisma Cloud Defender and Cortex XDR Prevent Agent Moving further into the visibility of the cloud, we will deploy a lightweight service that runs on the actual cloud host that leverages a four-dimensional behavioral model that looks at process activity, system calls, file system and outgoing network activity. This will all talk to a single console and incidents will be recorded from the behavioral model. In addition to some static and behavioral analysis we will see what a baseline is for “normal.” We will also be given host, container and image vulnerability data that will enable us to understand the attack surface even further in the cloud. We can also get visibility into CIS benchmarks and several regulatory compliance violations. The Prisma Cloud Defender is deployed at the host level with a simple curl script and can be deployed using automation with minimal administrative overhead. Image Figure 2 - [Defender Radar Pop Out] Much of the aforementioned functionality will be from a sole product set but as we have seen repeatedly, there is no silver bullet for the requirements that come from multiple business units, security, infrastructure, DevOps and any other stakeholders who want to protect a company’s assets. For some higher fidelity endpoint detection and response, we will leverage the Palo Alto Cortex XDR Prevent agent to get more granular visibility of processes and child processes spawned as well as doing more in-depth host-level threat hunting on our AWS EC2 instance. This will also allow for terminal access from the Cortex XDR dashboard to perform remediation if necessary. Image Figure 3 - [Screenshot of Cortex XDR Dashboard Showing Parent/Child Process Chain on an EC Host] The beauty of the product set that we will be leveraging for this blog series is that there is very little lift to get the administrative consoles set up. Both are offered as a SaaS and neither requires any modifications to an on-prem or cloud environment. In a matter of minutes, we can onboard a cloud account to ingest the metadata of the configurations and flow logs of AWS, GCP, Azure and Alibaba. This will provide a large set of features that can be mapped back to security operations workflows. A short list is provided below: Security Operations dashboard Cloud asset inventory Investigation capabilities to search flow logs, user activities and configuration changes Compliance dashboard mapping cloud to over 17 different standards Alert triage and notifications Identification of malicious activity Attack Scenario Leveraging MITRE ATT&CK® Matrix for Enterprise Cloud MITRE released this attack matrix for Enterprise Cloud that I feel is a good reference for walking through some of the aspects of an example enterprise cloud attack that we can identify once we have Prisma Cloud deployed. For this blog series, I’ll be covering techniques in initial access, persistence, privilege escalation and defense evasion tactic phases. Image Figure 4 – MITRE Att@ck Matrix for Enterprise Cloud MITRE ATT&CK® – Initial Access – Valid Accounts Imagine a scenario where a developer has cloud account credentials compromised. Without setting up two-factor authentication in AWS, Azure or GCP, a login from an anomaly IP address can go undetected if alarms aren’t set up in the native console or some type of UEBA solution. The credentials are used to log into the AWS console from a Ukrainian IP address. At this point, Prisma Cloud will capture CloudTrail logs and correlate the source IP and country of administrative tasks performed. The user’s console login will be flagged as a location-based anomaly and an “Account Hijacking Attempt” alert will be generated in the Prisma Cloud alerts table. At this point, you could send those Prisma Cloud alerts to a ticketing system like ServiceNOW or JIRA or notify security personnel via Microsoft Teams, Pager Duty or Slack. But for the purposes of this blog, we will allow the attacker to continue down the path of malicious activity. Along the way, we will demonstrate visibility across the attack chain. In a future blog, we will create a finely tuned policy. Image Figure 5 - [Compromised AWS User Ukraine Login] Account Hijack Attempt Alert Investigation (Prisma Cloud Enterprise -> Alerts -> Account Hijack Attempt Alert Image Figure 5a - [Compromised AWS User Ukraine Login With Map] Account Hijack Attempt Alert Investigation (Prisma Cloud Enterprise -> Alerts -> Account Hijack Attempt Alert -> Investigation Map View MITRE ATT&CK® - Persistence – Account Manipulation – Create Accounts – Valid Accounts Once attacker uses the stolen credentials to log in into the AWS console, he or she has administrative access of the cloud infrastructure and can start to look through the EC2 dashboard for potential hosts to compromise. At this point, the attacker notices an EC2 instance of interest and wants to get root access to create a new admin user and connect via SSH. The AWS credentials used have access to attach an IAM policy to the EC2 instance that allows root access through the web UI of AWS via the Amazon SSM agent service that is enabled on the EC2 host and allows command line access through AWS Systems Session Manager. At this point the attacker now has full root access to the EC2 instance and will start to conduct activities on the cloud host level. Prior to having a Defender deployed on the EC2 host, we were only leveraging the CSPM features of Prisma Cloud. The Defender adds host- and container-level visibility. This allows us to see every SSH login event and sudo command performed. The screenshot below shows that the attacker ran as root via the Amazon System Manager Agent process to create a new user “admin.” Prisma Cloud has activity monitoring capabilities via the runtime host protection functionality. Runtime host protection is designed to continuously report an up-to-date context for the EC2. You can set alerts for filesystem, process, network, log file events and more. Some events, such as process executions, can be blocked. A screenshot of the Host Observations tab in the Prisma Cloud GUI is provided in the figure below showing the ssm-agent-worker process running commands as root to create the admin user. Image Figure 6 - Host Observations (Compute -> Monitor -> Runtime -> Host Observations->Select EC2 Host With Defender Deployed->App tab) MITRE ATT&CK® – Privileged Escalation – Valid Accounts In addition to creating the new user, attackers generate their own SSH keys on the EC2 instance and can copy those private keys via the web GUI console so the SSH keys never leave the compromised cloud host. The keys are left stored on the instance so the only way we can identify that someone generated their own SSH keys is by viewing the Prisma Cloud Compute event logs, which are recording every command performed on the EC2 instance. Image Figure 7 - [Secrets Modified Event] (Compute -> Monitor -> Events -> Host Activities It should be noted that you can enable Session Manager logging in the native AWS tools for commands run through the web console, but this is not enabled by default: it was the attacker who enabled this connectivity, not the AWS admin. The fact the attacker was able to assign the IAM role for SSM quick setup to this device emphasizes that overly permissive IAM privlages continue to be a challenge, as it was on prem, but it’s even more amplified in the cloud. Image Figure 8 - Enable AWS System Manager Session Logs –AWS Systems Manager->Session Manager->Preferences Tab->Enable CloudWatch Logging Image Figure 9 - Host Observations (Compute -> Monitor -> Runtime -> Host Observations->Select EC2 Host With Defender Deployed-SSH Events Tab) Now the attacker has a new privileged account local to the EC2 and can SSH with the newly-generated private key. But upon attempting to login, the attacker realizes the security group needs the IP address allowed on port 22 inbound from the Ukraine. Since they can modify security groups with the AWS console web credentials, we can see the attacker making that change to the security group as part of the audit trail view in Prisma Cloud. This will show that a valid account made this change so it will not flag an alert on Prisma Cloud, but we are able to do an investigation to show when the change was made, to which security group and by which admin. Image Figure 10 - Audit Trail of Modifications to a Security Group by Compromised Cloud Admin In part two of this series, we will begin to exfiltrate data from the EC2 host and identify how Prisma Cloud ingests flow logs and can identify large volumes of traffic traversing the public cloud to the attacker’s public internet IP. Additional malicious activity will also be performed on the cloud host. By: Ramy F. Rahman Partner Architect - Cloud | Optiv Ramy Rahman is a Partner Architect specializing in cloud research on Optiv’s partner research and strategy team. In this role he’s responsible for use case development and the vetting of cloud native security platforms and cloud workload protection platforms. He educates SOC and IR teams on how to prepare to handle incidents as enterprises move to the cloud. Share: Blue Team Source Zero Threat Cloud Security Incident Response Data Forensics Copyright © 2021 Optiv Security Inc. All rights reserved. No license, express or implied, to any intellectual property or other content is granted or intended hereby. This blog is provided to you for information purposes only. While the information contained in this site has been obtained from sources believed to be reliable, Optiv disclaims all warranties as to the accuracy, completeness or adequacy of such information. Links to third party sites are provided for your convenience and do not constitute an endorsement by Optiv. These sites may not have the same privacy, security or accessibility standards. Complaints / questions should be directed to Legal@optiv.com Related Insights Image Infrastructure as Code: Terraform, AWS EKS, Gitlab & Prisma Cloud June 11, 2020 A how to guide using Gitlab, Hashicorp Terraform Cloud, AWS EKS and Palo Alto Networks Prisma Cloud to build and secure a Kubernetes cluster. See Details Blog Image Anatomy of a Kubernetes Attack - How Untrusted Docker Images Fail Us July 22, 2020 An attacker could use a poisoned docker image to break out of a container. See Details Blog Image Defending Against Container Threats With Palo Alto Prisma Cloud September 16, 2020 Palo Alto Networks Prisma Compute can help minimize the risk of deploying untrusted images. See Details Blog How Can We Help? Let us know what you need, and we will have an Optiv professional contact you shortly.