Accelerating Vulnerability Remediation with Automation

One way Optiv helps its clients is by speeding up their vulnerability management process. In a recent project we sought to reduce mean time to remediate (MTTR) by accelerating the processes from vulnerability discovery to the deployment of a corrective system update.

 

The vulnerability management process for user workstation and on-premise servers is ripe for revision. Taking cues from current systems management and application development processes, legacy vulnerability management can be transformed and accelerated by adopting two components:

 

  • patch unit testing and
  • automated patch identification and deployment.

 

The technologies required to incorporate these components into the vulnerability management process are robotic process automation (RPA) and systems management tools that offer the orchestration of vulnerability scan data, patch correlation and rules-based deployment through patch management solutions.

 

During our research we found that there’s no standardized method to model and analyze vulnerability management workflows. To have a better method to model these workflows Optiv created a “vulnerability management pipeline” concept. The pipeline is based on a set of building blocks that align with the major and minor phases organizations perform during the vulnerability management process.

 

In order to improve remediation times, an organization will first need to understand what factors are contributing to the overall time to remediate. Rather than building on a dated model, the goal of the pipeline is to break the process into better defined and measurable phases. This post will focus on how Optiv achieved automated end-to-end discovery to remediation for Windows workstations. An example of this pipeline is shown below.

 

Image
Accelerating Vulnerability Remediation img1

Figure 1 – Windows workstation remediation pipeline

 

 

Lab Architecture

 

Image
Accelerating Vulnerability Remediation img2

 

 

Lab Environment

 

  • SaaS
    • Tenable.io
    • Vulcan Cyber
  • Azure
    • Power Automate Cloud flows
    • Azure Automation runbooks
  • ESXi
    • Active Directory
    • System Center Configuration Manager (SCCM)
    • Windows 2016 servers
    • Windows 10 hosts

 

 

Integrating Vulcan Cyber Into the Remediation Pipeline

 

Image
Accelerating Vulnerability Remediation img3

Figure 2 – Adding Vulcan Cyber to a basic vulnerability management configuration

 

 

Vulcan’s solution was added to the remediation pipeline as a method to correlate Tenable.io vulnerability scan data with the appropriate corrective action. The initial design is shown above. Using the API integrations between Tenable.io, Vulcan Cyber and Microsoft SCCM, the flow of the pipeline is as follows:

 

  1. Tenable scans the Windows 10 desktops and Windows servers for vulnerabilities.
  2. Tenable.io sends vulnerability scan data to Vulcan. SCCM also sends host information to Vulcan.
  3. Vulcan sends a remediation package to SCCM.
  4. The Vulcan remediation package creates a device collection and a software update group in SCCM.

 

At this stage an operator can initiate patch deployment from SCCM to the device collection.

 

An example Vulcan runbook used in this proof of concept is shown in Figure 3 below. When Vulcan receives vulnerabilities with fixes from Tenable.io matching asset names that begin with PRS-KS, it creates an SCCM remediation action.

 

Image
Accelerating Vulnerability Remediation img4

 

 

Image
Accelerating Vulnerability Remediation img5

Figure 3 – Example Vulcan runbook

 

 

At this stage we have not reached a fully automated remediation pipeline. In this environment the “last mile” of remediation requires a human to perform four steps:

 

  1. Create a deployment package.
  2. Download the software for the deployment package.
  3. Assign a distribution point for the deployment package.
  4. Start the deployment update.

 

 

Deployment Automation

In order to automate the “last mile” of the Windows remediation pipeline Optiv used the automation functionality provided in Microsoft’s Power Automate and Azure Automation. In figure 4 the additional components of the remediation pipeline are highlighted in green.

 

Image
Accelerating Vulnerability Remediation img6

Figure 4 – Additions to the vulnerability management configuration

 

 

An action is added to the Vulcan lightweight automation runbook that sends an email with all of the aggregated information to a service email account. The Playbook note, which is optional, was added as a way for Power Automate to down select only certain emails as a trigger. This will be explained further in the steps below.

Image
Accelerating Vulnerability Remediation img7

Figure 5a – Kicking off a Vulcan runbook

 

 

A Power Automate flow was created to assist with the automation. The flow is triggered by an email from hello@vulcan[.]io that has attachments.

 

Image
Accelerating Vulnerability Remediation img8

Figure 5b – Power Automate flow trigger from Vulcan runbook

 

 

The next step in the flow is to narrow the scope of the emails with attachments by performing a regular expression search for a particular note in the body of the email. This information was added in the Vulcan action shown above in figure 5b.

 

Image
Accelerating Vulnerability Remediation img9

Figure 6 – Conditional step

 

 

If the condition is met when the body of the email matches the regular expression the flow continues on to the next step, which is an Azure Automation runbook.

Image
Accelerating Vulnerability Remediation img10

Figure 7 – Azure Automation runbook

 

 

The details of this Azure Automation runbook are shown in the PowerShell code below. When the code is executed on the hybrid-worker, it will perform the “last mile” tasks that were still manual in the first iteration of the remediation pipeline.

 

  1. Calls the Automation account credential store to use the account named service for the RunAs account
  2. Logs into the SCCM server from the hybrid-worker
  3. Installs the Configuration Manager PowerShell cmdlets
  4. Checks the current time vs. the time that SUG-RPA software update group was last modified; if the time was modified within the last 10 minutes, SCCM is instructed to:

    1. Create a deployment package
    2. Download the software for the deployment package
    3. Assign a distribution point for the deployment package
    4. Start the deployment update

 

#Get credentials to perform RunAs from the automation crendtial store.
$Credential = Get-AutomationPSCredential -Name 'service'
$computername = 'PRS-SCCM01'
$ScriptBlock = {
$DriveName = 'P01'
$Root = 'PRS-SCCM01.optivtest.com'


#Importing the Configuration Manager PowerShell cmdlets
cd "C:\Program Files (x86)\Microsoft Configuration Manager\AdminConsole\bin\ConfigurationManager\" -verbose;
Import-Module .\ConfigurationManager.psd1
New-PSDrive -Name $DriveName -PSProvider "AdminUI.PS.Provider\CMSite" -Root $Root -Description "Primary site";
Set-Location P01:
#Ignore error message “A drive with the name ‘P01’ already exists.”

#Time Difference Check
$DateTimeNow = Get-Date
$CollectionModifiedTime = Get-CMSoftwareUpdateGroup -Name SUG-RPA
$DateLastModified = $CollectionModifiedTime.DateLastModified
$Difference = New-TimeSpan -start $DateLastModified -End $DateTimeNow
$Minutes = $Difference.TotalMinutes
$Minutes
if($Difference
-le 10){
"New Deployment Modification Time"

#Create Deployment Package
#Note: If the Deployment Package Name exisits then it will throw an error message and continue with the next command.
New-CMSoftwareUpdateDeploymentPackage -Name "RPA" -Path "\\PRS-SCCM01\Sources\SCCM\PackageUpdates\RPA"

#Download Required Updates
#Note: If the Required Updates are already downloaded then it will throw an error message and continue with the next command.
Get-CMSoftwareUpdateGroup -Name SUG-RPA | Save-CMSoftwareUpdate -DeploymentPackageName RPA

#Create Distribution Point
#Note: If the Distribution Point for the package already exisits then it will throw an error message and continue with the next command.
Start-CMContentDistribution -DeploymentPackageName "RPA" -DistributionPointName "PRS-sccm01.optivtest.com"

#Start Deployment Update
Start-CMSoftwareUpdateDeployment `
-SoftwareUpdateGroupName "SUG-RPA" `
-CollectionName "DC-RPA" `
-DeploymentName "Automation" `
-Description "Push from Automation Pipeline" `
-DeploymentType Required `
-VerbosityLevel AllMessages `
-TimeBasedOn UTC `
-UserNotification DisplayAll `
-PersistOnWriteFilterDevice $False `
-DisableOperationsManagerAlert $True `
-GenerateOperationsManagerAlert $True `
-ProtectedType RemoteDistributionPoint `
-UnprotectedType NoInstall `
-UseBranchCache $False `
-DownloadFromMicrosoftUpdate $False

} else {
"Deployment Modification Time Not New"
}
}
Invoke-Command -ScriptBlock $ScriptBlock -Credential $Credential -computername $computername

 

 

This particular runbook is rather static, as it lists a specific software update group, device collection and deployment name, but it can be altered to include others. The script is shown as the first action step highlighted in red in the flow below.

 

Image
Accelerating Vulnerability Remediation img11

Figure 8 – Power Automate Cloud Flow

 

 

Patch Unit Testing with Robotic Process Automation

Optiv recently surveyed a sample set of clients regarding vulnerability remediation. All of the clients surveyed responded they would allow for end-to-end automated remediation, with no human intervention, if user testing was completed. Optiv has shown that user acceptance testing, post update, can be automated with the use of robotic process automation. Not only can the testing be automated, but it can be an integrated step of the pipeline.

 

While this post is not intended to provide detailed guidance on the use of RPA, it is important that once user workflows are captured, they are tested repeatably to ensure a consistent and expected outcome. Once confident in the RPA’s expected outcome of the workflow(s), the execution of the workflows can be added as a step in the pipeline.

 

Adding unit testing:

 

Image
Accelerating Vulnerability Remediation img12

Figure 9 – Adding unit testing to the vulnerability management configuration

 

 

In the final iteration of the pipeline, Optiv used several Azure Automation runbooks and a Power Automate Desktop flow as additional actions that are added on to the previous Power Automate cloud flow. In order for RPA to be used as a workflow test, Optiv needed to ensure the software updates were applied to the test host. Optiv created an Azure Automation runbook to force the test client to check for updates:

 

#Get credentials to perform RunAs from the automation crendtial store.
$Credential = Get-AutomationPSCredential -Name 'service'
$computername = 'PRS-SCCM01'
$ScriptBlock = {
$DriveName = 'P01'
$Root = 'PRS-SCCM01.optivtest.com'

#Importing the Configuration Manager PowerShell cmdlets
cd "C:\Program Files (x86)\Microsoft Configuration Manager\AdminConsole\bin\ConfigurationManager\" -verbose;
Import-Module .\ConfigurationManager.psd1
New-PSDrive -Name $DriveName -PSProvider "AdminUI.PS.Provider\CMSite" -Root $Root -Description "Primary site";
Set-Location P01:

#Ignore error message “A drive with the name ‘P01’ already exists.”
#Ignore error message “A drive with the name ‘P01’ already exists.”

Invoke-CMClientAction `
-DeviceName PRS-KSRPA `
-ActionType ClientNotificationSUMDeplEvalNow
}
Invoke-Command -ScriptBlock $ScriptBlock -Credential $Credential -computername $computername

 

 

This is what this action looks like in the Power Automate flow.

 

Image
Accelerating Vulnerability Remediation img13

Figure 10 – Power Automate Flow

 

 

After the host is “forced” to run the update a delay timer is executed. The delay timer is intended to provide the host enough time to install the updated. After the time has expired an Azure Automation runbook is used to query that the update was installed.

 

Image
Accelerating Vulnerability Remediation img14

Figure 11 – Power Automate Flow details

 

 

The third Azure Automation runbook that was used is listed below. The runbook checks if the deployment to the host was successful by running the Get-CMDeployment command and checking the NumberSuccess object.

 

#Get credentials to perform RunAs from the automation crendtial store.
$Credential = Get-AutomationPSCredential -Name 'service'
$computername = 'PRS-SCCM01'
$ScriptBlock = {
$DriveName = 'P01'
$Root = 'PRS-SCCM01.optivtest.com'

#Importing the Configuration Manager PowerShell cmdlets
cd "C:\Program Files (x86)\Microsoft Configuration Manager\AdminConsole\bin\ConfigurationManager\" -verbose;
Import-Module .\ConfigurationManager.psd1
New-PSDrive -Name $DriveName -PSProvider "AdminUI.PS.Provider\CMSite" -Root $Root -Description "Primary site";
Set-Location P01:

#Ignore error message “A drive with the name ‘P01’ already exists.”
#Ignore error message “A drive with the name ‘P01’ already exists.”

$NS = Get-CMDeployment -CollectionName "DC-RPA" -FeatureType SoftwareUpdate | ForEach-Object NumberSuccess
if($NS -contains 1){
"Host is up to date"
} else {
"Host has not been updated"
}
}
Invoke-Command -ScriptBlock $ScriptBlock -Credential $Credential -computername $computername

 

If the deployment was successful the host will execute the user workflow test(s) using Microsoft’s Power Automate Desktop flow.

 

Image
Accelerating Vulnerability Remediation img15

Figure 12 – Power Automate Desktop Flow

 

 

Using a Power Automate Desktop flow, a simple user workflow is executed after the host is patched to ensure that the software or security update does not disrupt any workflow or business process. In Optiv’s proof of concept a single user workflow was tested in the remediation pipeline, but multiple local and web application workflows can be added to this flow to ensure all possible user tasks are tested before scheduling an update for the entire organization.

 

 

Automated Remediation Pipeline

With the assistance of Microsoft Desktop flow, user workflow testing is complete. The last step in the automated remediation pipeline is to deploy the software update to a larger device collection in SCCM. Optiv was able to do this using an additional Azure Automation runbook. The runbook below assigns the software update group to an existing device collection that contains additional Windows 10 hosts. These hosts are running the same version of Windows 10 and have the same applications installed as the one tested in the RPA action of the Power Automate flow.

 

#Get credentials to perform RunAs from the automation crendtial store.
$Credential = Get-AutomationPSCredential -Name 'service'
$computername = 'PRS-SCCM01'
$ScriptBlock = {
$DriveName = 'P01'
$Root = 'PRS-SCCM01.optivtest.com'

#Importing the Configuration Manager PowerShell cmdlets
cd "C:\Program Files (x86)\Microsoft Configuration Manager\AdminConsole\bin\ConfigurationManager\" -verbose;
Import-Module .\ConfigurationManager.psd1
New-PSDrive -Name $DriveName -PSProvider "AdminUI.PS.Provider\CMSite" -Root $Root -Description "Primary site";
Set-Location P01:
#Ignore error message “A drive with the name ‘P01’ already exists.”

#Start Deployment Update
Start-CMSoftwareUpdateDeployment `
-SoftwareUpdateGroupName "SUG-RPA" `
-CollectionName "WindowsWorkstations-Win10" `
-DeploymentName "Automation" `
-Description "Push from Automation Pipeline - Post RPA Test(s)" `
-DeploymentType Required `
-VerbosityLevel AllMessages `
-TimeBasedOn UTC `
-DeploymentAvailableDay $DateTimeNow `
-DeploymentAvailableTime $DateTimeNow `
-UserNotification DisplayAll `
-PersistOnWriteFilterDevice $False `
-DisableOperationsManagerAlert $True `
-GenerateOperationsManagerAlert $True `
-ProtectedType RemoteDistributionPoint `
-UnprotectedType NoInstall `
-UseBranchCache $False `
-DownloadFromMicrosoftUpdate $False
}

Invoke-Command -ScriptBlock $ScriptBlock -Credential $Credential -computername $computername

 

The code above is in the action highlighted in red in the flow below.

 

Image
Accelerating Vulnerability Remediation img16

Figure 13 – Power Automate Desktop Flow details

 

 

Complete Flow:

Image
Accelerating Vulnerability Remediation img17

Figure 14 – Complete flow

 

 

  1. Power Automate checks the service email account for new email from Vulcan.io
  2. Email matches regex criteria
  3. Azure Automation Runbook
    1. Creates a deployment package
    2. Downloads the software included in the software update group for the deployment package
    3. Assigns a distribution point for the deployment package
    4. Deployment update is started
  4. Azure Automation Runbook
    1. Force the test client to check SCCM for new software updates and download
  5. Sleep timer used to wait 1 hour until to ensure the software update is installed
  6. Azure Automation Runbook
    1. Checks to see if the deployment package was successfully installed on the test host
  7. Power Automate flow checks to see that the previous set was successful
  8. Power Automate Desktop flow is executed to simulate a user workflow
  9. Azure Automation Runbook
    1. The software update is applied to a larger device collection in SCCM
  10. An email notifies the admin that the software update has been tested and pushed to a larger device collection

 

This pipeline shows that workstation vulnerability management can be fully automated, including user workflow testing, with no human interaction. In a previous project, Optiv validated that endpoint remediation can be tested using a similar flow with Mandiant Security Validation. This research was completed in 2020 and Optiv was able to create an Azure host, install the MSV agent and perform testing with no human interaction.

 

In my next Source Zero post I will add the ability to provide efficacy testing, through Mandiant Security Validation, to the existing remediation pipeline, further automating validation of the remediation action.

Dan Kiraly
Senior Research Analyst
Dan Kiraly is senior research analyst on Optiv’s partner research and strategy team. In this role he responsible for use case development and the vetting of security products for Optiv.