Sigma Rule Translation and Automatic Queries

May 26, 2022

Sigma Rule Translation and Automatic Queries

There are many different SIEM and EDR tools which, for the most part, use their own query language for information retrieval. When security team members cycle out of an organization their tribal knowledge and the specialized understanding of each query language is lost. Our industry is plagued by numerous query languages which hinders our collective ability to execute defenses effectively. But: what if queries could be written in one language and then translated into over 20 others? Such a development would reduce the learning curve to request specific information from various security tools and enhance detection efficacy.

An emerging standardized language does exist: Sigma. Sigma is a generic rule format, written in YAML, to express search queries on log data. The project can be found here: https://github.com/SigmaHQ.

How does this help and what does this mean? It means rules are formatted in a way that they can be translated between various query languages. As depicted in the diagram below, a Sigma rule can be translated to search/hunt across several SIEM and EDR platforms without the analyst having to know the query syntax of each platform.

An example Sigma rule from the official GitHub project is shown below.

Example: sysmon_accessing_winapi_in_powershell_credentials_dumping.yml

title: Accessing WinAPI in PowerShell for Credentials Dumping
id: 3f07b9d1-2082-4c56-9277-613a621983cc
description: Detects Accessing to lsass.exe by Powershell
status: experimental
author: oscd.community, Natalia Shornikova
date: 2020/10/06
modified: 2021/05/24
references:
- https://speakerdeck.com/heirhabarov/hunting-for-powershell-abuse
tags:
- attack.credential_access
- attack.t1003.001
logsource:
product: windows
service: sysmon
detection:
selection:
EventID:
- 8
- 10
SourceImage|endswith: '\powershell.exe'
TargetImage|endswith: '\lsass.exe'
condition: selection
falsepositives:
- Unknown
level: high

I found these resources helpful for additional information on creating Sigma rules:

Sigma has been around for a few years and there are various contributors to the official project, as well as additional projects that serve as repos for Sigma rules. One such repo I found is listed here: https://github.com/mdecrevoisier/SIGMA-detection-rules.

I found the rule translation process and the ability to potentially automate queries across platforms intriguing. Specifically, how could the translation outputs be used as queries with little to no human interaction? To answer that, let’s begin with how rules can be translated.

The Sigma project contains a Python script called sigmac. Sigmac is a tool that preforms the translation between Sigma rules and various security query formats using a YAML configuration file that map Sigma fields to each vendors’ query language.

SOCPrime provides a web translation tool that can be found here: https://uncoder.io/.

For a proof-of-concept I wanted to see if I could automate the use of sigmac to convert rules to a specific query format. Once translated, I wondered if I could automatically run a query against the security tool for any hits?

My initial thoughts were to use Microsoft robotic process automation with uncoder.io to copy and paste rules or Power Automate (I’m a big fan of Power Automate) and sigmac to tackle the individual tasks necessary to perform the conversion and query. I feel like I’ve spent enough time highlighting Microsoft’s Automation tools in past posts that I wanted a new challenge. So, this idea would need to be distilled down to smaller steps to successfully execute my proof-of-concept.

I decided Python would be my tool of choice for this task. Trying my hand at something new provides me with an appreciation for the complexities that are encountered in other technical roles.

Breaking the original idea down into smaller tasks provided me with the outline below:

Make the project shareable
Clone the Sigma repo
Convert a subset of Sigma rules to Devo queries
Write the translated rules to separate files for later use
Pass the converted queries to an API request

To make the proof-of-concept sharable via Docker seemed an obvious choice. Docker is freely available, easy to share, widely used, and there is a convenient extension in VS Code in which I could prove all of this out. Within the Docker file the Sigma repo is cloned. The Docker file below uses a Python image, installs the necessary dependencies and runs several scripts, which I’ll walk through, to achieve the goal of automating the translation and API queries.

Dockerfile

FROM python:3.8-slim-buster

# setup dependencies
RUN apt-get update
RUN apt-get install xz-utils
RUN apt-get -y install curl git
RUN pip install pyyaml
RUN pip install requests
RUN pip install python-dateutil

WORKDIR /usr/src/app/

COPY . .

RUN git clone https://github.com/SigmaHQ/sigma.git
RUN cp devo-win.yml sigma/tools/config/
RUN mkdir Translations
RUN python translate.py
RUN mv Devo_* Translations
RUN python main.py

CMD ["sh"]

With the dependencies installed, the container clones the Sigma repo. With the repo cloned the next task was to run sigmac (rule translation script) against all the files in a directory. The translate.py code below does this.

translate.py

# import required modules
import cmd
import os
from re import sub

rootdir = '/usr/src/app/sigma/rules/windows/wmi_event'
cmd = './sigma/tools/sigmac -t devo -c sigma/tools/config/devo-win.yml '

for subdir, dirs, files in os.walk(rootdir):
for file in files:
ruleoutput = 'Devo_'+ file
devorule= ruleoutput.replace(".yml","")
cmdLine = cmd +subdir+'/'+file + ' -o '+ devorule
os.system(cmdLine)

For this proof-of-concept, the script defines a root directory. In this case, it is the path of the Sigma rules for windows wmi events, which is a sub-directory. Within the folder there are three Sigma rules. Sigmac will translate each of the three rules in this folder into a Devo query. The script will also run correctly if pointed at the Windows parent directory. If Windows is listed as the parent directory sigmac would then translate all rules in every subfolder below windows parent directory. Note, this can take some time.

The script also creates a new file for each translated Devo rule, appending Devo_ to the original filename. One thing to note when the rules are translated with sigmac or uncode.io all attributes such as title, author, MITRE IDs, etc. are removed and you are left with the raw query. After translation is complete, each new file is moved into the Translations folder. This action is defined in the docker file, line 19. Taking a look at the contents of the Translations folder we can see that there are three new files beginning with “Devo_”.

The screenshot above also shows the contents of the file Devo_sysmon_wmi_event_subscription. Here we see a properly translated Devo query that aligns with original Sigma rule shown below.

One item to note is that the proper translation of Sigma rules to Devo_windows queries are not working correctly without a little finessing of the devo_windows.yml config file. I have opened and issue regarding this: https://github.com/SigmaHQ/sigma/issues/2941. The issue has to do with a single field on line 125. Changing the line to Type:eventType as shown in the second screen shot below seemed to work for me. The edited file can be found here.

Image: devo-windows.yml from https://github.com/SigmaHQ/sigma/blob/master/tools/config/devo-windows.yml

Image: Corrected line Note: There is a devo-win.yml file on my github project that corrects this.

After the Sigma rules were properly translated, the last step was to send an API query request to Devo. Devo does have a SDK, but I wanted to make the request without using it, so I created the devo_request.py script listed below. The script requires API keys for authentication. Once the keys are provided, the only argument required for the script to run correctly is a properly formatted Devo query. Devo API queries require that a start date/time and end date/time are defined as a string. The script automatically calculates a thirty-day window, in the proper format, from the time of script execution and will search over the last thirty days for whatever query it is provided. The devo_request.py is used to run individual queries. This means I would need to call the script as many times as there are new Devo translated queries within the Translations folder.

devo_request.py

from inspect import signature
import time
import datetime
import dateutil.relativedelta
import hmac
import hashlib
import json
import requests
import sys

url = "https://apiv2-us.devo.com/search/query"
#note: hard coding keys is a really bad practice and this is just poc code.
api_key = 'YOUR-KEY'
api_secret = 'YOU-SECRET'
query = sys.argv[1]
curr_dt = datetime.datetime.now()
now = int(round(curr_dt.timestamp()))
start_dt = (curr_dt + dateutil.relativedelta.relativedelta(days=-30))
start = int(round(start_dt.timestamp()))
payload = json.dumps({
"query": query,
"from": start,
"to": now,
"limit": 10,
"mode": {
"type": "json/compact"
}
})
a = bytes(api_secret, 'utf-8')
timestamp = str(int(time.time()) * 1000)
data = payload
message = api_key + data + timestamp
b = bytes(message, 'utf-8')
sign = hmac.new(a, b, digestmod=hashlib.sha256).hexdigest()
headers = {
'Content-Type': 'application/json',
'x-logtrust-apikey': api_key,
'x-logtrust-sign': sign,
'x-logtrust-timestamp': timestamp
}
response = requests.request("POST", url, data=payload,
headers=headers)
#print(response.text)
#print(response.json())
json_object = json.dumps(response.json(), indent = 4)
with open('response.txt', 'a') as outfile:
outfile.write("\n"+ "Query results from the last 30 Days for query " + query + json_object +"\n")

The request script can write the contents to screen or it can write the query results, in json format, to a file, response.txt. Each query result will be appended to the file so past queries are not overwritten. I could have used something like pandas to present the results to the user in a better format, but that was above and beyond my original goal.

With the devo_request.py script written, I needed to find a way to open all the files within a folder, read the contents of each and pass the contents as the query argument in the devo_request.py script. The main.py script below does just that. With the Translations folder defined, the program walks the folder, opens each file, reads the contents and passes the text as the query parameter in the devo_request.py script.

main.py

from pathlib import Path
import sys
import cmd
import os
from re import sub
import cmd

cmd = 'python devo_request.py '

folder = "/usr/src/app/Translations/"
for file in os.listdir(folder):
filepath = os.path.join(folder, file)
f = open(filepath, 'r')
r = f.read()
cmdLine = cmd +("\'"+r+"\'")
print('\n')
print("Query results from the last 30 Days for "+ r)
f.close()
os.system(cmdLine)

Opening the response.txt file shows the exact Devo API query made, as well results or response of each query. This can be seen in the partial screenshot of the response.txt file.

Stepping back from the code the entire flow looks like this:

The docker image will drop you into a shell so you can poke around. There is a ton of room to expand on these ideas and improve my scripts. I would encourage anyone interested in content engineering or API queries to take a look.

Tools referenced in this blog

https://github.com/SigmaHQ
https://github.com/SigmaHQ/sigma/wiki/Converter-Tool-Sigmac
https://github.com/kiraly15/sigma_blog

By:

Dan Kiraly

Senior Research Scientist | Optiv

Dan Kiraly is senior research scientist on Optiv’s R&D team. In this role he's responsible for use case development and the vetting of security products for Optiv.

Optiv Security: Secure greatness.^®

Optiv is the cyber advisory and solutions leader, delivering strategic and technical expertise to nearly 6,000 companies across every major industry. We partner with organizations to advise, deploy and operate complete cybersecurity programs from strategy and managed security services to risk, integration and technology solutions. With clients at the center of our unmatched ecosystem of people, products, partners and programs, we accelerate business progress like no other company can. At Optiv, we manage cyber risk so you can secure your full potential. For more information, visit www.optiv.com.

Related Insights

Mastering the Hunt: Threat Hunting with Optiv and Carbon Black

Even though corporate endpoints made up the top five assets involved in breaches last year, many enterprises still focus only on securing their network.

Cyber Incident Readiness Services

Cybersecurity incident readiness services measure your incident response capability against the current threat landscape and industry best practices. Learn more!

Threat Intelligence Services

Cybersecurity success hinges on transforming data into actionable intel. Gain resources to repel advanced cyber threats with Optiv’s Threat Intelligence services.

How Can We Help?

Let us know what you need, and we will have an Optiv professional contact you shortly.