If you are reading this blog post via a 3rd party source it is very likely that many parts of it will not render correctly. Please view the post on signalscorps.com for the full interactive viewing experience.
In this post I will show intelligence producers how to add and remove STIX 2.1 Objects using a TAXII Server API.
Note: this post is written for OASIS TAXII version 2.1. The concepts discussed are not always correct for earlier versions of OASIS TAXII. This post also talks about some specifics related to the Signal Corps implementation of a TAXII 2.1 Server, namely the exclusive use of STIX 2.1 content.
Most users will simply want to consume intelligence from a TAXII Server. However, someone needs to import the intelligence for consumption from the server.
In the last post I showed the response of an API Root;
curl -i https://MY_HOST/api/v1/group1/collections/ \
-H "Accept: application/taxii+json;version=2.1" \
-H "Authorization: Basic <BASE64 ENCODED CREDENTIALS>" \
-A "Signals Corps Curl Demo Client/1.0"
Which returns each Collection inside of the API Root;
HTTP/1.1 200 OK
Content-Length: 637
Content-Type: application/taxii+json;version=2.1
Connection: close
{
"collections": [
{
"id": "91a7b528-80eb-42ed-a74d-c6fbd5a26116",
"title": "High Value Indicator Collection",
"description": "This data collection contains high value IOCs",
"can_read": true,
"can_write": false,
"media_types": [
"application/stix+json;version=2.1"
]
},
{
"id": "52892447-4d7e-4f70-b94d-d7f22742ff63",
"title": "Another Collection",
"description": "This data collection is for collecting current IOCs",
"can_read": true,
"can_write": true,
"media_types": [
"application/stix+json;version=2.1"
]
}
]
}
Note how one collection returns "can_write": true for the authenticated user.
When true clients can POST and DELETE Objects using the objects endpoint (/{api-root}/collections/{id}/objects/).
Lets start by adding a new Object to this Collection using the Add Objects endpoint.
The request takes the form; HOST/<API_ROOT>/collections/<COLLECTION_ID>/objects/, with a JSON envelope payload in the body.
Here is the body of the response I will pass in the body (-d);
{
"objects": [
{
"type": "indicator",
"spec_version": "2.1",
"id": "indicator--9ef5baeb-ad55-4c2c-aa89-ea5515dacf61",
"created": "2022-01-01T01:00:00.000Z",
"modified": "2022-01-01T01:00:00.000Z",
"name": "Malicious site hosting downloader",
"description": "This organized threat actor group operates to create profit from all types of crime.",
"indicator_types": [
"malicious-activity"
],
"pattern": "[url:value = 'http://x4z9arb.cn/4712/']",
"pattern_type": "stix",
"valid_from": "2022-01-01T01:00:00.000Z"
}
]
}
In the request, I pass it minified with json esacpes (\) like so;
curl -i -X POST https://MY_HOST/api/v1/group1/collections/52892447-4d7e-4f70-b94d-d7f22742ff63/objects/ \
-H "Accept: application/taxii+json;version=2.1" \
-H "Content-Type: application/taxii+json;version=2.1" \
-H "Authorization: Basic <BASE64 ENCODED CREDENTIALS>" \
-A "Signals Corps Curl Demo Client/1.0" \
-d "{\"objects\":[{\"type\":\"indicator\",\"spec_version\":\"2.1\",\"id\":\"indicator--9ef5baeb-ad55-4c2c-aa89-ea5515dacf61\",\"created\":\"2022-01-01T01:00:00.000Z\",\"modified\":\"2022-01-01T01:00:00.000Z\",\"name\":\"Malicious site hosting downloader\",\"description\":\"This organized threat actor group operates to create profit from all types of crime.\",\"indicator_types\":[\"malicious-activity\"],\"pattern\":\"[url:value = 'http:\/\/x4z9arb.cn\/4712\/']\",\"pattern_type\":\"stix\",\"valid_from\":\"2022-01-01T01:00:00.000Z\"}]}"
Which returns a detailed response with lots of information about the status of the request
HTTP/1.1 202 ACCEPTED
Content-Length: 307
Content-Type: application/taxii+json;version=2.1
Connection: close
{
"id": "2d086da7-4bdc-4f91-900e-d77486753710",
"status": "complete",
"request_timestamp": "2022-01-01T01:00:00.001Z",
"total_count": 1,
"success_count": 1,
"successes": [
{
"id": "indicator--9ef5baeb-ad55-4c2c-aa89-ea5515dacf61"
}
],
"failure_count": 0,
"pending_count": 0
}
When adding multiple Objects, the status property can equal pending as the records are added. You can view more information about the Status resource here.
To update an existing Object, I can use the same endpoint and post an updated Object. Essentially this is a minor STIX update, where the id and created properties of the STIX Object remains the same, but other properties updated, including updating the STIX Objects modified time.
For example, if I wanted to update the title of the object I just created (indicator--9ef5baeb-ad55-4c2c-aa89-ea5515dacf61), I would need to create a new version for it.
The new version is as follows;
{
"objects": [
{
"type": "indicator",
"spec_version": "2.1",
"id": "indicator--9ef5baeb-ad55-4c2c-aa89-ea5515dacf61",
"created": "2022-01-01T01:00:00.000Z",
"modified": "2022-08-02T03:33:33.000Z",
"name": "UPDATED TITLE Malicious site hosting downloader",
"description": "This organized threat actor group operates to create profit from all types of crime.",
"indicator_types": [
"malicious-activity"
],
"pattern": "[url:value = 'http://x4z9arb.cn/4712/']",
"pattern_type": "stix",
"valid_from": "2022-01-01T01:00:00.000Z"
}
]
}
Note how the title and modified properties are updated. All other properties remain the same.
The request to create a new version is the same as when creating a new Object, except for the modified properties in the payload (-d);
curl -i -X POST https://MY_HOST/api/v1/group1/collections/52892447-4d7e-4f70-b94d-d7f22742ff63/objects/ \
-H "Accept: application/taxii+json;version=2.1" \
-H "Content-Type: application/taxii+json;version=2.1" \
-H "Authorization: Basic <BASE64 ENCODED CREDENTIALS>" \
-A "Signals Corps Curl Demo Client/1.0" \
-d "{\"objects\":[{\"type\":\"indicator\",\"spec_version\":\"2.1\",\"id\":\"indicator--9ef5baeb-ad55-4c2c-aa89-ea5515dacf61\",\"created\":\"2022-01-01T01:00:00.000Z\",\"modified\":\"2022-08-02T03:33:33.000Z\",\"name\":\"UPDATED TITLE Malicious site hosting downloader\",\"description\":\"This organized threat actor group operates to create profit from all types of crime.\",\"indicator_types\":[\"malicious-activity\"],\"pattern\":\"[url:value = 'http:\/\/x4z9arb.cn\/4712\/']\",\"pattern_type\":\"stix\",\"valid_from\":\"2022-01-01T01:00:00.000Z\"}]}"
Now when any of the Object endpoints are queried, multiple versions of it will be returned (assuming they are not filtered out). For example,
curl -i https://MY_HOST/api/v1/group1/collections/52892447-4d7e-4f70-b94d-d7f22742ff63/objects/indicator--9ef5baeb-ad55-4c2c-aa89-ea5515dacf61/versions/ \
-H "Accept: application/taxii+json;version=2.1" \
-H "Authorization: Basic <BASE64 ENCODED CREDENTIALS>" \
-A "Signals Corps Curl Demo Client/1.0"
Which returns both versions;
HTTP/1.1 200 OK
Content-Length: 116
Content-Type: application/taxii+json;version=2.1
X-TAXII-Date-Added-First: 2022-01-01T01:00:00.000Z
X-TAXII-Date-Added-Last: 2022-01-01T01:00:00.000Z
Connection: close
{
"more": false,
"next": 0,
"versions": [
"2022-01-01T01:00:00.000Z",
"2022-08-02T03:33:33.000Z"
]
}
In many cases the status of a POST requests will need to be tracked – usually when many Objects are being added and the initial response returns "status": "pending" as it takes the server time to process them all.
For this the Get Status endpoint can be used for 24 hours after the request was made.
This endpoint takes the request in the form; HOST/<API_ROOT>/status/<STATUS_ID>/objects/.
The STATUS_ID passed in the following request is returned in the POST response to Add Object(s) endpoint’
curl -i https://MY_HOST/api/v1/group1/status/2d086da7-4bdc-4f91-900e-d77486753710 \
-H "Accept: application/taxii+json;version=2.1" \
-H "Authorization: Basic <BASE64 ENCODED CREDENTIALS>" \
-A "Signals Corps Curl Demo Client/1.0" \
Which returns a response that will include status information on successes, failures and pendings Object additions;
HTTP/1.1 200 OK
Content-Length: 800
Content-Type: application/taxii+json;version=2.1
Connection: close
{
"id": "2d086da7-4bdc-4f91-900e-d77486753710",
"status": "pending",
"request_timestamp": "2016-11-02T12:34:34.12345Z",
"total_count": 4,
"success_count": 1,
"successes": [
{
"id": "indicator--c410e480-e42b-47d1-9476-85307c12bcbf" ,
"version": "2018-05-27T12:02:41.312Z"
}
],
"failure_count": 1,
"failures": [
{
"id": "malware--664fa29d-bf65-4f28-a667-bdb76f29ec98",
"version": "2018-05-28T14:03:42.543Z",
"message": "Unable to process object"
}
],
"pending_count": 2,
"pendings": [
{
"id": "indicator--252c7c11-daf2-42bd-843b-be65edca9f61",
"version": "2018-05-18T20:16:21.148Z"
},
{
"id": "relationship--045585ad-a22f-4333-af33-bfd503a683b5",
"version": "2018-05-15T10:13:32.579Z"
}
]
}
You can see there was 1 failure (malware--664fa29d-bf65-4f28-a667-bdb76f29ec98) where the server was Unable to process object. This shows the importance of the status endpoint, as it is likely I would now want to resolve the error by debugging it and reuploading the Malware STIX 2.1 Object.
If I wanted to delete both versions of the Objects I previously created (indicator--9ef5baeb-ad55-4c2c-aa89-ea5515dacf61), I could run a delete query to the Delete Objects endpoint.
This endpoint takes the request in the form; HOST/<API_ROOT>/collections/<COLLECTION_ID>/objects/<OBJECT_ID>.
Here is how I would Delete all versions of the Object;
curl -i -X "DELETE" https://MY_HOST/api/v1/group1/collections/52892447-4d7e-4f70-b94d-d7f22742ff63/objects/indicator--9ef5baeb-ad55-4c2c-aa89-ea5515dacf61/ \
-H "Accept: application/taxii+json;version=2.1" \
-H "Authorization: Basic <BASE64 ENCODED CREDENTIALS>" \
-A "Signals Corps Curl Demo Client/1.0"
Which, if successful, returns;
HTTP/1.1 200 OK
Content-Length: 0
Content-Type: application/taxii+json;version=2.1
Connection: close
There is no status ID generated for delete requests. If a 200 is returned the request was successful, else any other error means the delete failed for the specified reason.
Any subsequent attempts to fetch the object/version (in this case all versions of indicator--9ef5baeb-ad55-4c2c-aa89-ea5515dacf61) using the Get Objects Endpoint will return a 404 (Not Found) response.
The delete endpoint also allows for filtering to delete specific versions or spec_versions of an Object.
In the following example I am deleting the 2nd version of the Object indicator--9ef5baeb-ad55-4c2c-aa89-ea5515dacf61 (2022-08-02T03:33:33.000Z);
curl -i -X "DELETE" https://MY_HOST/api/v1/group1/52892447-4d7e-4f70-b94d-d7f22742ff63/objects/indicator--9ef5baeb-ad55-4c2c-aa89-ea5515dacf61/?match[version]=2022-08-02T03:33:33.000Z \
-H "Accept: application/taxii+json;version=2.1" \
-H "Authorization: Basic <BASE64 ENCODED CREDENTIALS>" \
-A "Signals Corps Curl Demo Client/1.0"
A quick note on Obstracts approach to deletes
In our TAXII server implementation, deletes do not actually delete the record, it updates the STIX Object with the property "revoked": true as per the Common Properties of STIX 2.1 Objects and at the same time updates the modified timestamp of the Object. Any Objects with "revoked": true are still returned via the TAXII server, but can be identified in request to the TAXII API using the revoked and modified properties. Note, this does go against the official TAXII 2.1 Specification which requires deleted Objects to return 404s.
Next up: Out-of-the-box TAXII Servers
Now you know the basics, I imagine you are turning to Google to find some public TAXII 2.1 servers you can play with.
Sadly, many are private at the time of writing (here is a list of public TAXII servers listed on the wiki of OASIS CTI Technical Committee).
However, Medallion (cti-taxii-server), is an open-source TAXII 2.1 Server Library Written in Python that makes for a great local playground. In the next tutorial post I will be writing a Medallion quick-start guide to get you up and running.
TAXII 2.1 Certification (Virtual and In Person)
The content used in this post is a small subset of our full training material used in our TAXII 2.1 training.
If you want to join a select group of certified TAXII 2.1 professionals, subscribe to our newsletter below to be notified of new course dates.
Discuss this post
Never miss an update
Sign up to receive new articles in your inbox as they published.