What Lurks in Your Network? Finding and Combating Undetected Malware

By Aaron Goldstein ·

How to Combat the Threats of Undetected Malware

For the past 19 months, I have been in charge of the Incident Management (IM) team for FishNet Security, handling digital investigations and proactive services relating to the same. Most investigations the team is involved with center around incident response activities.

It should be no surprise to anyone at this point that there has been an escalating trend of successful hacks at companies of all types and sizes across the United States. After all, the who’s who of large enterprises, retailers and government entities have all suffered catastrophic breaches in the last few years, and nobody seems immune to it.

But, the trends and statistics are not nearly as interesting as what lurks beneath the surface of most networks in the esoteric world of malware and espionage and the details of the digital war we are currently embroiled in.

What is often a surprise to affected parties is the sophistication and timeline of attacks and how long targeted environments have actually been exposed vs. knowing they were exposed and the depth of penetration and persistence attackers have within the computing infrastructure. Often times the date of breach and date of detection have a differential of months to years.

It is not uncommon to identify that the bad guys have gained admin access to an affected environment, installed backdoors and rootkits and have been quietly snooping around, gathering intel, mapping out networks and applications, understanding data flows and siphoning off sensitive information long before their activities are detected.

Possibly the longest case I have encountered was a 5 ½ year window between compromise and detection of a company’s employee/customer database, involving PII for all parties. During this time, the attacker milked the system for new data and defended it from other attackers. If not for another attacker successfully compromising the system and using it noisily to attack other systems on the Internet, it might still be undetected today.

What’s worse, even if organizations have the technology available to them to detect the malicious activity, it often does not provide value for a variety of reasons:

  • It is usually not properly configured or fine-tuned enough to filter out the noise to facilitate detection.
  • Security personnel lack the necessary training and strategy to successfully identify and respond to current and evolving threats.
  • Detection systems are simply not monitored at all.

In fact, most often, malicious activity is reported by outsiders, such as card brands, law enforcement entities, complaining customers or other businesses being attacked by computing resources within the affected environment. Since companies generally are not in the business of being hacked or conducting incident response investigations, they seldom have the proficiency in house to be able to adequately respond.

Quite simply put, the ability of bad guys to hack environments and establish persistence mechanisms has far outpaced most organizations’ abilities to detect and respond to the same.

Contributing to the problems, security applications (such as antivirus and IDS/IPS) intended to protect computing environments from these types of threats have been outmatched at their own game by the evolution of attacks and often provide little to no value. There is a digital arms race that is occurring and the aggressors are clearly winning.

This should be no surprise, considering the volume of money behind the attackers in what has been deemed the largest transfer of wealth in human history. There is no shortage of funding for the attackers to develop new and innovative approaches to compromise and maintain persistence within target organizations. And, as new defensive tools and methodologies are pioneered and rolled out on a large scale, the attacks quickly evolve to stay ahead of them and avoid detection.

However, in most cases, exotic evasion techniques are not needed. Attackers can generally run undetected in most environments using commoditized, high-grade malware that has become available for free or at nominal cost over the last few years.

Generally speaking, off-the-shelf malware is sufficient to evade most detection mechanisms, because of several factors outlined below, some of which have been observed by FishNet Security across scores of customer environments within the last year:

  • Waterhole or spearphishing attacks employing multi-layered, automated attack strategies targeting dozens of commonly installed applications in turn in an attempt to exploit vulnerable apps, often delivered from legitimate sites over SSL and frustrating detection from IDS/IPS and network malware monitoring solutions.
  • Polymorphic malware per machine changing most aspects of its footprint (hash, size, name, path) and evading AV signatures.
  • Fragmented applications where no single part seems particularly malicious or ranks high in a threat score,  but when functioning as a whole provide comprehensive backdoor capabilities and evade AV and host based IDS/IPS.
  • Creation of hundreds or thousands of unique and randomly generated domain names for command and control, bypassing reputation blocking methods and IDS/IPS.
  • Encrypted command and control communication streams evading IDS/IPS signatures and network malware monitoring solutions.
  • Timestamp modification masking malware file install dates to look like legitimate operating system install date or patch install dates, frustrating basic forensic analysis.
  • Sophisticated crimeware suites with remote administration capabilities permitting updates, rolling out of new modules, and holistic changes to installed code and communication channels inside and outside of the target environment allowing for rapid deployment of desired changes and complicating detection using any method.
  • Memory resident, non-persistent binaries leaving no footprint on the target file system and only operational for random periods of time, complicating forensic analysis.

Over the last year, some security vendors have recognized significant gaps in these areas and have devised various solutions that have begun to address portions of them. However, there currently is no solution that addresses all of the gaps. Manual methodologies must be continually employed to identify malicious behavior, especially newly evolving threats.

To facilitate successful investigations in the evolving threat landscape across diverse customer environments, our IM team employs investigative techniques that combine numerous solutions from various vendors (as needed per the customer environment) combined with an adaptive methodology that is geared toward identifying the latest and greatest penetration and persistence methodologies.

First we take a high-level look at the entire evidence universe - file systems, system memory, process behavior, log data, network communications, etc. - using the Breach Discovery methodology. Data is monitored and collected using non-intrusive means and correlated with threat intelligence sources from years of consulting engagements in order to identify the breadth and depth of an intrusion.

Not only is this typically the first phase of any breach investigation, but often provided on its own in a proactive manner to identify if a customer has suffered a breach, what nodes may be involved and what data is potentially being exfiltrated from their environment. Further, most (if not all) of this first phase can be completed remotely, using cloud-based forensics and targeted log/network traffic collections.

Note: To date, utilizing the Breach Discovery methodology, NO network of significant size has been identified free of malware intrusion. All investigated networks contain some level of malicious command and control in varying degrees.

Once the initial scope of intrusion and the associated hosts have been identified, the IM team is ready to take a deeper dive. This can be done in a non-invasive way by utilizing a combination of network, local, and cloud-based forensics, log correlation and network traffic analysis. This allows the team to identify the timeline of events, including exploitation methods, malicious applications, backdoors, persistence mechanisms and data that has been exfiltrated from the customer environment. We’ve even been able to identify and respond to malware on nodes on remote sites and roaming users’ laptops through cloud-based forensics technology as long as the end user or IT staff installs the appropriate agent.

In conjunction with the in-depth forensic process, it is vital to perform containment and surgical remediation of affected systems, while continuing to monitor for new infections to prevent further spread of malware and persistence mechanisms throughout the environment, until a more permanent remediation process can begin.

In general this is performed using a variety of mechanisms, including:

  • Host-based agents capable of blocking known malicious processes and derivatives from executing and/or making network connections.
  • Host-based agents capable of killing malicious processes and associated files and persistence mechanisms.
  • Network based restrictions, including segmenting affected hosts and security appliances capable or configurable to stop identified malicious network signatures and remote hosts.
  • Active directory restrictions to prevent the spread of malware via network shares.
  • Custom antivirus signatures to eliminate known variants within an organization.

By utilizing surgical remediation and live containment methodologies, it is possible to allow a partially contaminated network to function and facilitate semi-normal business processes while the investigation and cleanup runs in tandem. However, it is always recommended that a complete remediation be performed on all affected nodes as soon as possible, involving reimaging affected machines or rebuilding from install media, depending on the situation.

Employing a combination of best-of-breed tools and proactive methodologies, it is possible to identify and root out next-generation threats that may lurk silently und go undetected for months or years, prior to bursting forth into the headlines as the next large and embarrassing attack affecting millions of customers.

We’ve seen examples in the headlines lately of large companies with deep pockets come under attack. The ultimate cost and impact of these breaches is incalculable. But taking a proactive approach to incident response rather than waiting to find out the hard way that you’ve been breached, can minimize the associated costs and impact.