Getting Started with STIX Shifter

Posted by:

If you are reading this blog post via a 3rd party source it is very likely that many parts of it will not render correctly. Please view the post on signalscorps.com for the full interactive viewing experience.

In this post I will show you how to turn STIX Patterns into common threat detection languages.

My last post described standardising writing detection using STIX 2.1 Patterns and Objects (and how to converted existing Sigma Rules to STIX 2.1 Patterns/Objects).

Whilst STIX 2.1 makes it easier to create, manage, and collaborate on detection content, it does not solve the actioning problem – that is; how to use STIX Patterns in downstream detection tools (e.g. SIEMs).

That is where STIX-Shifter comes in.

STIX-shifter is an open source python library allowing software to connect to products that house data repositories by using STIX Patterning, and return results as STIX Observations.

STIX-Shifter;

takes STIX 2.x Patterns as input
converts them to target rule formats
sends the converted rule to the downstream tool
detects data that matches the patterns inside downstream tools (e.g. SIEMs, EDRs, etc)
transforms the output (the detection) into STIX 2.x Observed Data Objects.

Here is the flow modelled in a slide presented by the IBM team in this webinar;

STIX-Shifter flow

This flow means all of the intelligence and security incidents, regardless of the source, is modelled in the same way – in STIX 2.x.

STIX-Shifter Connectors

STIX-Shifter is based around the concept of Connectors.

A STIX-Shifter connector is a module inside the STIX-Shifter library that implements an interface for:

data source query and result set translation
data source communication

Each Connector supports a set of STIX objects and properties as defined in the connector’s mapping files.

There are about 30 Connectors that currently exist, detailed here.

Let me demonstrate this concept using some examples.

Installing STIX-Shifter

STIX Shifter can be used as a command line utility or as a Python library.

To install STIX-Shifter in both ways;

git clone https://github.com/opencybersecurityalliance/stix-shifter
cd stix-shifter
python3 -m venv tutorial_env
source tutorial_env/bin/activate
pip3 install stix-shifter
pip3 install stix-shifter-utils
stix-shifter -h

STIX-Shifter core functions

Stix-Shifter provides three core functions;

translate: The translate command converts STIX patterns into data source queries (in whatever query language the data source might use) and translates data source results (in JSON format) into bundled STIX observation objects.
transmit: The transmit command allows stix-shifter to connect with products that house repositories of cybersecurity data. Connection and authentication credentials are passed to the data source APIs where stix-shifter can make calls to ping the data source, make queries, delete queries, check query status, and fetch query results.
execute: The translation and transmission functions can work in sequence by using the execute command from the CLI.

Converting STIX Patterns to target formats

To use a connector (to translate a STIX pattern), you must first install it. You can do this using pip as follows;

pip3 install stix-shifter-modules-<CONNECTOR NAME>

For example, to install the Splunk Connector;

pip3 install stix-shifter-modules-splunk

The translate command line argument takes the form;

stix-shifter translate <CONNECTOR NAME> query "<STIX IDENTITY OBJECT>" "<STIX PATTERN>" "<OPTIONS>"

Therefore to convert the STIX Pattern [url:value = 'http://www.testaddress.com'] OR [ipv4-addr:value = '192.168.122.84'] using the newly installed Splunk Connector I can run;

stix-shifter translate splunk query "{}" "[url:value = 'http://www.testaddress.com'] OR [ipv4-addr:value = '192.168.122.84']"

Note, I passed an empty

Prints the converted Splunk query in a JSON response;

{
    "queries": [
        "search (url = \"http://www.testaddress.com\") OR ((src_ip = \"192.168.122.84\") OR (dest_ip = \"192.168.122.84\")) earliest=\"-5minutes\" | head 10000 | fields src_ip, src_port, src_mac, src_ipv6, dest_ip, dest_port, dest_mac, dest_ipv6, file_hash, user, url, protocol, host, source, DeviceType, Direction, severity, EventID, EventName, ss_name, TacticId, Tactic, TechniqueId, Technique, process, process_id, process_name, process_exec, process_path, process_hash, parent_process, parent_process_id, parent_process_name, parent_process_exec, description, result, signature, signature_id, query, answer"
    ]
}

You will notice the Splunk search (nested in the queries field). The key part of the search is;

(url = \"http://www.testaddress.com\") OR ((src_ip = \"192.168.122.84\") OR (dest_ip = \"192.168.122.84\"))

STIX Shifter has converted the STIX fields url:value into url and ipv4-addr:value into both src_ip and dest_ip fields (as the STIX pattern could refer to either).

You read a description of the logic performed by the Splunk Connector to perform the translation here.

Splunk data (assumed to be in the Common Information Model (CIM) standard) to STIX mapping is defined in the Splunk modules to_stix_map.json file.

Also notice how the search ends with earliest=\"-5minutes\" | head 10000 | fields src_ip,.... These are added by default in the conversion and are not converted from the STIX Pattern. In short these Splunk commands:

earliest=\"-5minutes\" is defining the time range to look back
head is limiting the number of results returned (to first 10,000) and
fields specifies which fields to keep or remove from the search results

The purpose of including these commands is to limit to scope of the search and ensure all fields are present when matches are found, when STIX-Shifter is used to created Observed Data Object (more on that to follow). If you just want to use STIX-Shifter for conversion to Splunk format, I would remove this from the output as it’s not really useful.

Lets try another conversion, this time using the Elastic ECS Connector on the same STIX Pattern;

pip3 install stix-shifter-modules-elastic_ecs

The Elastic Common Schema (ECS) is Elastics own standard, similar to the Splunk CIM.

stix-shifter translate elastic_ecs query "{}" "[url:value = 'http://www.testaddress.com'] OR [ipv4-addr:value = '192.168.122.84']"

{
    "queries": [
        "(url.original : \"http://www.testaddress.com\") OR ((source.ip : \"192.168.122.84\" OR destination.ip : \"192.168.122.84\" OR client.ip : \"192.168.122.84\" OR server.ip : \"192.168.122.84\" OR host.ip : \"192.168.122.84\" OR dns.resolved_ip : \"192.168.122.84\")) AND (@timestamp:[\"2022-08-23T06:28:44.754Z\" TO \"2022-08-23T06:33:44.754Z\"])"
    ]
}

You can see the STIX Pattern to Elastic ECS conversion logic here.

STIX Shifter has converted the STIX fields url:value into url.original and ipv4-addr:value into source.ip, server.ip, host.ip, and dns.resolved_ip.

Like with Splunk, the query also included a 5 minute time window @timestamp:[\"2022-08-23T06:28:44.754Z\" TO \"2022-08-23T06:33:44.754Z\"]).

Creating STIX Observed Data from Detections

In this post I won’t cover transmit, where STIX-Shifter can authenticate to downstream products via a Connector which can be used to push rules. However, imagine my converted rules were sent down to Splunk to look for matching log lines.

I will show a simulated example of a match being detected and written into a STIX 2.1 Observed Data Object.

Lets assume the downstream tool (Splunk) detects a match between a converted STIX Pattern ([ipv4-addr:value = '1.1.1.1'] -> src_ip=1.1.1.1 OR dest_ip=1.1.1.1) and a log line that contains src_ip=1.1.1.1. In addition to the matching field, the log line has the following fields (this is where the fields command in the the Splunk STIX-Shifter output is important) modelled in json;

[
    {
        "src_ip": "1.1.1.1",
        "dest_ip": "2.2.2.2",
        "url": "www.testaddress.com"
    }
]

It is vital that the fields match those defined in the STIX-Shifter Connector so that the can be mapped to the correct STIX Cyber Observable Object (e.g. IPv4 STIX Cyber Observable Object) during the translation. The STIX-Shifter Splunk Connector expects CIM compliant fields (src_ip, dest_ip and url are all CIM compliant).

This time the translate query takes a slightly different form to create STIX 2.1 Observed Data and Cyber Observable Objects from the detection (using result instead of query);

stix-shifter translate <MODULE NAME> results '<STIX IDENTITY OBJECT>' '<LIST OF JSON RESULTS>'

Unlike before, a STIX Identity Object is required to be used in the command to attribute the Observed Data Objects to someone. I will use a demo Identity as follows;

        {
          "type": "identity",
          "spec_version": "2.1",
          "id": "identity--a3e71e61-2e8a-4bca-acd6-d34a36e6c47a",
          "created": "2016-01-01T00:00:00.000Z",
          "modified": "2016-01-01T00:00:00.000Z",
          "name": "Signals Corps Demos",
          "description": "Just for testing",
          "identity_class": "organization",
          "sectors": "technology"
        }

Which written out into an entire translate query gives;

python main.py translate splunk results \
 '{"type":"identity","spec_version":"2.1","id":"identity--a3e71e61-2e8a-4bca-acd6-d34a36e6c47a","created":"2016-01-01T00:00:00.000Z","modified":"2016-01-01T00:00:00.000Z","name":"Signals Corps Demos","description":"Just for testing","identity_class":"organization","sectors":"technology"}' \
 '[{"src_ip":"1.1.1.1","dest_ip":"2.2.2.2","url":"www.testaddress.com"}]' \
 '{"stix_2.1": true}'

By default, JSON results are translated into STIX 2.0. To return STIX 2.1 results include {"stix_2.1": true} in the options part (last part) of the CLI command.

This command prints a JSON bundle with a STIX 2.1 Observed Data Object (covering the entire log line representing the match), and four STIX 2.1 Cyber Observable Objects representing each field type in the log line, src_ip, dest_ip, and url. Note, there are four results as the single url in my log (www.testaddress.com) is converted by the Splunk STIX-Shifter Connector into STIX 2.1 Cyber Observable types URL and Domain Name.

Here is the bundle output modelled as a graph (you can export it to see the raw STIX 2.1 bundle);

Now let me highlight why the fields printed in the log data, must match those expected by the Connector.

This time I will use the Elastic ECS Connector on the same log line. Elastic ECS does not use the CIM field name standard used by Splunk. For example, as shown in the example translate conversion from STIX Pattern to Elastic ECS, IPs are captured in the field name source.ip (in Splunk the CIM compliant field is src_ip).

Demonstrating using the same command as I did for Splunk, the only difference being the connector used (this time elastic_ecs);

python main.py translate elastic_ecs results \
 '{"type":"identity","spec_version":"2.1","id":"identity--a3e71e61-2e8a-4bca-acd6-d34a36e6c47a","created":"2016-01-01T00:00:00.000Z","modified":"2016-01-01T00:00:00.000Z","name":"Signals Corps Demos","description":"Just for testing","identity_class":"organization","sectors":"technology"}' \
 '[{"src_ip":"1.1.1.1","dest_ip":"2.2.2.2","url":"www.testaddress.com"}]' \
 '{"stix_2.1": true}'

See how an Observed Data Object is created, but STIX-Shifter cannot convert any Cyber Observable Data Objects from the input because the field names in the log are not mapped in the Elastic ECS Connector configuration.

It is important to understand that when field mappings are incorrect, STIX-Shifter can product inconsistent results.

Let me demonstrate using the QRadar Connector;

pip3 install stix-shifter-modules-qradar

python main.py translate qradar results \
 '{"type":"identity","spec_version":"2.1","id":"identity--a3e71e61-2e8a-4bca-acd6-d34a36e6c47a","created":"2016-01-01T00:00:00.000Z","modified":"2016-01-01T00:00:00.000Z","name":"Signals Corps Demos","description":"Just for testing","identity_class":"organization","sectors":"technology"}' \
 '[{"src_ip":"1.1.1.1","dest_ip":"2.2.2.2","url":"www.testaddress.com"}]' \
 '{"stix_2.1": true}'

QRadar does use the url field name, so this is mapped correctly to a STIX URL Cyber Observable Object (note, this is different behaviour to the Splunk Connector which creates a URL and Domain Observable for this record).

However, for the unrecognised fields (src_ip and dest_ip) the QRadar Connector creates a custom STIX 2.1 Cyber Observable Object ("type": "x-Signals Corps Demos"), which contains the properties "src_ip": "1.1.1.1" and "dest_ip": "2.2.2.2" for these unrecognised fields.

The point being, be careful with field mappings, because if the Connector does not support the fields in the log, the results from STIX-Shifter can be unexpected. This is the age old problem of SIEMs – normalising fields between logs being ingested and normalising fields across SIEMs.

STIX-Shifter Resources

Hopefully this post has given you a good starting point to get up-and-running with STIX-Shifter. Here are some useful resources I used to create this post that will provide useful references as you progress with STIX-Shifter;

Discuss this post

JOIN US