Everything, Everywhere, All at Once: The Key to IT Crisis Management Effective Incident Response

September 28, 2023

To maintain CyberOps and ITOps resilience, a game plan is good. But muscle memory is better.


As any professional athlete will tell you, resilience is key to success and the only way to attain it is to train, train and train some more. Such is the case for incident response (IR) teams which have to remember their game plan and have everything, everywhere working in harmony. That’s the way to weather the storm and recover as quickly as possible.



Uphill battle

Today’s IT ops and security teams too often find themselves in an uphill battle. Attackers may not always carry the element of surprise or unleash sophisticated attacks, but their persistence and stealth can be unnerving. They are relentless, ever ready to exploit a vulnerability, use a stolen credential, or take advantage of a slow reaction or lapse in focus. Plus, they have a thriving underground economy offering all the tools, services, and knowledge they need to launch attacks. One security vendor saw cyber threats jump tenfold in just the first six months of this year. Yet many more sneak in under the radar.


On the defensive end, network protectors face a relentless tide of challenges. There’s a persistent and chronic skills gap, with an estimated shortfall of cybersecurity professionals globally in excess of three million, including more than 410,000 in the U.S. And thanks to pandemic-era investments in digital infrastructure and a shift to hybrid work, corporate attack surfaces have never been more expansive. Two-fifths of global organizations think these are “spiraling out of control.”


Against this backdrop, breaches and security incidents are inevitable. And they’re having a critical impact on enterprises. The average cost of a data breach globally is $4.35M, and in the U.S., it is $9.44M.


The speed of incident response is also a top security operations (SecOps) challenge. Why? Because too many organizations are laboring with multiple tools and siloed data, which give conflicted signals that are difficult for SecOps teams to prioritize. The data they do get to work with is often stale, incomplete, and riddled with false positives. Systems management solutions are unable to provide the rapid visibility analysts need to detect and contain emerging threats.



Practice, practice, practice

So how can organizations get back on track? In short, careful planning and repeated practice. First, decide what your corporate crown jewels are. It may be a customer transaction database, or some closely guarded intellectual property. What can the organization not live without? Then it’s about war gaming to find where the gaps in security posture are that need filling: an unpatched server here, a misconfigured access point there. Know what you look like to an attacker. Use red teams to simulate real-world attacks covering an exhaustive range of scenarios. This is about building resilience to minimize the chance of a successful breach. An ounce of prevention is worth a ton of cure.


But, as discussed, prevention can only get you so far. That’s why it’s critical to test those incident response plans. This is the opportunity to get everything and everyone working together, all at once. Get your critical IR stakeholders in a room twice a year to run and test those foundational processes around change management, problem management, communication structure, disaster recovery plans, business continuity and more. This is the only way you’re going to find those single points of failure that won’t appear during normal operations.


The key here is to keep those operations running as smoothly as possible even during an incident. It’s about resisting the human urge to panic and try something new when a crisis hits. You want to get to a stage where when a crisis comes along it’s not a crisis at all but just second nature for everyone involved. Everything, everywhere whirrs into action because you planned and trained for this. In short, you know what to do when you get hit in the mouth.



When seconds count

Emergency responders do this. When lives are on the line, they need to prepare for every eventuality. Military units do it too. And of course, professional athletes practice their critical plays until it comes as naturally as breathing. But analogies aside, what do incident response teams actually need to maintain business operations, recover rapidly, and minimize the damage to the organization?


First, they require real-time visibility into the entire IT environment. IR teams need granular, accurate, and rich data to scope an incident, then investigate efficiently and quickly. And they need instant control, to take corrective action when necessary. That means working from a single source of truth – a platform that can scale to hundreds of thousands of endpoints without breaking a sweat. That kind of visibility and control won’t just allow IR teams to fix any issues unearthed in war-gaming tests, it will also enable real-time threat hunting, as well as incident investigation and rapid remediation, if a worst-case scenario becomes a reality.


When seconds count, speed and precision are critical. But so too are cool heads and well-trained teams.

Tim Morris
Financial Services Strategist | Tanium
Tim joined Tanium in May 2021, after retiring from Wells Fargo, where he spent 21 years. He led the Cyber Threat Engineering and Research teams within Information & Cyber Security for the bank.

Tim has worked with almost every facet of computer and network technologies. Concentration has been with endpoint detection & response, systems & patch management, and vulnerability assessment. He has built teams that manage: endpoint security, platform engineering, incident response, digital forensics, and offensive security, i.e., "red team".

Tim was first introduced to Tanium in 2008. However, he didn't begin working with it fully until 2013. Tim was privileged to have the opportunity to be one of the first to deploy & manage Tanium at a large scale on 500K endpoints. At the same time, he was able to build one of the best cyber security engineering teams in the industry. Their effectiveness and efficiency were due in large part to Tanium - The best incident response and system management tool in the industry.

Optiv Security: Secure greatness.®

Optiv is the cyber advisory and solutions leader, delivering strategic and technical expertise to nearly 6,000 companies across every major industry. We partner with organizations to advise, deploy and operate complete cybersecurity programs from strategy and managed security services to risk, integration and technology solutions. With clients at the center of our unmatched ecosystem of people, products, partners and programs, we accelerate business progress like no other company can. At Optiv, we manage cyber risk so you can secure your full potential. For more information, visit www.optiv.com.