If you are reading this blog post via a 3rd party source it is very likely that many parts of it will not render correctly. Please view the post on signalscorps.com for the full interactive viewing experience.
In this post I will show you a quick start guide to using Oasis’s open-source TAXII Client, cti-taxii-client.
Note: this post is written for OASIS TAXII version 2.1. The concepts discussed are not always correct for earlier versions of OASIS TAXII. This post also talks about some specifics related to the Signal Corps implementation of a TAXII 2.1 Server, namely the exclusive use of STIX 2.1 content.
A growing number of cyber security products are introducing native TAXII 2.1 Client functionality to consume cyber threat intelligence fro TAXII 2.1 Servers.
In the previous posts I showed examples of interacting with TAXII 2.1 APIs manually using curl requests. It is of course possible to use this knowledge in addition to the TAXII specification to build your own client.
However, before you decide to build out all the logic required for a TAXII 2.1 Client into your own product, here are some existing open-source options you might want to consider, or at least to use for inspiration.
cti-taxii-client
cti-taxii-client is a client developed by Oasis, and it is probably the best template to use if you plan to build your own TAXII 2.1 Client.
cti-taxii-client is a minimal client implementation for the TAXII 2.X server. It supports the following TAXII 2.0 and 2.1 API services:
- Server Discovery
- Get API Root Information
- Get Status
- Get Collections
- Get a Collection
- Get Objects
- Add Objects
- Get an Object
- Delete an Object (2.1 only)
- Get Object Manifests
- Get Object Versions (2.1 only)
The easiest way to install the TAXII client is with pip:
git clone https://github.com/signalscorps/tutorials
cd tutorials/cti-taxii-client-tutorial
python3 -m venv tutorial_env
source tutorial_env/bin/activate
pip3 install taxii2-client
If you are wondering why I cloned the tutorials repository, it is to have the sample scripts locally that are used in this tutorial.
To test out the TAXII 2.1 Client, you will need a running TAXII 2.1 Server. For this post I will use the same TAXII 2.1 Server (medallion) I demonstrated last week.
Unlike last week when I was manually creating curl requests that required me to add the full logic in each request to interact with the TAXII 2.1 Server. Using a TAXII Client, in this case cti-taxii-client, is much simpler as these already contain the logic for the TAXII Client / TAXII Server interaction.
cti-taxii-client supports both TAXII 2.0 and TAXII 2.1 in two distinct sub-modules. For this tutorial I will use TAXII 2.1.
To begin with I have written a simple Python script that uses the cti-taxii-client module functions to query the discovery endpoint to find out what is on the server.
From the taxii2client.v21 package I need to import the Server Class and define how to authenticate to the server (user/password).
Here is a script I wrote to simply print all the properties for the Server object: https://github.com/signalscorps/tutorials/blob/main/cti-taxii-client-tutorial/examples/01-server-discovery.py;
python3 01-server-discovery.py
Which prints;
server.title : Some TAXII Server
server.description : This TAXII Server contains a listing of
server.contact : string containing contact information
server.default.url : http://localhost:5000/trustgroup1/
server.custom_properties : {}
server.api_roots : ['http://localhost:5000/api1/', 'http://localhost:5000/api2/', 'http://localhost:5000/trustgroup1/']
Three server.api_roots are printed (thus accessible to this user). I can now identify what Collections exist in one of the API Roots (I will use http://localhost:5000/trustgroup1)
This time importing the ApiRoot class, here is a script I wrote to print all the Collections belonging to the API Root URL http://localhost:5000/trustgroup1 using ApiRoot: https://github.com/signalscorps/tutorials/blob/main/cti-taxii-client-tutorial/examples/02-list-collections-in-api-root.py;
python3 02-list-collections-in-api-root.py
Which prints;
Collection 1
collection.title: This data collection is for testing querying across collections
collection.description: None
collection.id: 472c94ae-3113-4e3e-a4dd-a9f4ac7471d4
collection.custom_properties: {}
collection.can_read: False
collection.can_write: True
collection.media_types: ['application/stix+json;version=2.1']
Collection 2
collection.title: This data collection is for testing adding objects
collection.description: None
collection.id: 365fed99-08fa-fdcd-a1b3-fb247eb41d01
collection.custom_properties: {}
collection.can_read: True
collection.can_write: True
collection.media_types: ['application/stix+json;version=2.1']
Collection 3
collection.title: High Value Indicator Collection
collection.description: This data collection is for collecting high value IOCs
collection.id: 91a7b528-80eb-42ed-a74d-c6fbd5a26116
collection.custom_properties: {}
collection.can_read: True
collection.can_write: True
collection.media_types: ['application/stix+json;version=2.0', 'application/stix+json;version=2.1']
...
Now I can start to discover the Objects held by each of these Collections. I will use 91a7b528-80eb-42ed-a74d-c6fbd5a26116 to demonstrate.
Here is a script I wrote that grabs the first page of Objects and prints it: https://github.com/signalscorps/tutorials/blob/main/cti-taxii-client-tutorial/examples/03-print-all-objects-in-collection-no-pagination.py;
python3 03-print-all-objects-in-collection-no-pagination.py
Which returns;
{'more': False, 'objects': [{'created': '2014-05-08T09:00:00.000Z', 'id': 'relationship--2f9a9aa9-108a-4333-83e2-4fb25add0463', 'modified': '2014-05-08T09:00:00.000Z', 'relationship_type': 'indicates', 'source_ref': 'indicator--cd981c25-8042-4166-8945-51178443bdac', 'spec_version': '2.1', 'target_ref': 'malware--c0931cc6-c75e-47e5-9036-78fabc95d4ec', 'type': 'relationship'}, {'created': '2014-05-08T09:00:00.000Z', 'id': 'indicator--cd981c25-8042-4166-8945-51178443bdac', 'indicator_types': ['file-hash-watchlist'], 'modified': '2014-05-08T09:00:00.000Z', 'name': 'File hash for Poison Ivy variant', 'pattern': "[file:hashes.'SHA-256' = 'ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c']", 'pattern_type': 'stix', 'spec_version': '2.1', 'type': 'indicator', 'valid_from': '2014-05-08T09:00:00.000000Z'}, {'created': '2017-01-20T00:00:00.000Z', 'definition': {'tlp': 'green'}, 'definition_type': 'tlp', 'id': 'marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da', 'name': 'TLP:GREEN', 'spec_version': '2.1', 'type': 'marking-definition'}, {'created': '2017-01-27T13:49:53.997Z', 'description': 'Poison Ivy', 'id': 'malware--c0931cc6-c75e-47e5-9036-78fabc95d4ec', 'is_family': True, 'malware_types': ['remote-access-trojan'], 'modified': '2017-01-27T13:49:53.997Z', 'name': 'Poison Ivy', 'spec_version': '2.1', 'type': 'malware'}, {'created': '2016-11-03T12:30:59.000Z', 'description': 'Accessing this url will infect your machine with malware. This is the last updated indicator', 'id': 'indicator--6770298f-0fd8-471a-ab8c-1c658a46574e', 'indicator_types': ['url-watchlist'], 'modified': '2017-01-27T13:49:53.935Z', 'name': 'Malicious site hosting downloader', 'pattern': "[url:value = 'http://x4z9arb.cn/4712']", 'pattern_type': 'stix', 'spec_version': '2.1', 'type': 'indicator', 'valid_from': '2016-11-03T12:30:59.000Z'}]}
There are 5 Objects print in this Collection (relationship--2f9a9aa9-108a-4333-83e2-4fb25add0463, indicator--cd981c25-8042-4166-8945-51178443bdac, marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da, malware--c0931cc6-c75e-47e5-9036-78fabc95d4ec, and indicator--6770298f-0fd8-471a-ab8c-1c658a46574e). In this case I know there are no more pages (and thus Objects) because 'more': False.
However, in many cases, the Collection will return Objects over multiple pages.
If you remember back to last week, the Object printed per page is set in the medallion server config file. To demonstrate how the library deals with pagination I can change the setting to one Object per page like so;
"taxii": {
"max_page_size": 1
}
With this now set, for the same script as before I would expect 5 pages when this Collection is returned – 1 page for each Object.
If I run the same script again (03-print-all-objects-in-collection-no-pagination.py), you will see this;
{'more': True, 'next': '563dd8a5-7f48-43e7-8306-65fab8803a77', 'objects': [{'created': '2014-05-08T09:00:00.000Z', 'id': 'relationship--2f9a9aa9-108a-4333-83e2-4fb25add0463', 'modified': '2014-05-08T09:00:00.000Z', 'relationship_type': 'indicates', 'source_ref': 'indicator--cd981c25-8042-4166-8945-51178443bdac', 'spec_version': '2.1', 'target_ref': 'malware--c0931cc6-c75e-47e5-9036-78fabc95d4ec', 'type': 'relationship'}]}
This time only one Object is printed (because one Object per page).
As such, I need to introduce some pagination logic into the script. The TAXII Client ships with a Class (as_pages) for TAXII 2.1 endpoints that support pagination.
Here is an example script that pages through each page and prints the result of each page: https://github.com/signalscorps/tutorials/blob/main/cti-taxii-client-tutorial/examples/04-print-all-objects-in-collection-with-pagination.py;
python3 04-print-all-objects-in-collection-with-pagination.py
Which prints the following (note some lines removed ... for brevity);
Page # 1
{'more': True, 'next': '989ba757-07ea-4828-b42e-e2b53f7aab3a', 'objects': [{'created': '2014-05-08T09:00:00.000Z', 'id': 'relationship--2f9a9aa9-108a-4333-83e2-4fb25add0463', 'modified': '2014-05-08T09:00:00.000Z', 'relationship_type': 'indicates', 'source_ref': 'indicator--cd981c25-8042-4166-8945-51178443bdac', 'spec_version': '2.1', 'target_ref': 'malware--c0931cc6-c75e-47e5-9036-78fabc95d4ec', 'type': 'relationship'}]}
...
Page # 5
{'more': False, 'objects': [{'created': '2016-11-03T12:30:59.000Z', 'description': 'Accessing this url will infect your machine with malware. This is the last updated indicator', 'id': 'indicator--6770298f-0fd8-471a-ab8c-1c658a46574e', 'indicator_types': ['url-watchlist'], 'modified': '2017-01-27T13:49:53.935Z', 'name': 'Malicious site hosting downloader', 'pattern': "[url:value = 'http://x4z9arb.cn/4712']", 'pattern_type': 'stix', 'spec_version': '2.1', 'type': 'indicator', 'valid_from': '2016-11-03T12:30:59.000Z'}]}
I can also get a specific Object, using its ID. Using indicator--cd981c25-8042-4166-8945-51178443bdac returned in the last responses, I created a script to get this specific Object and print it; https://github.com/signalscorps/tutorials/blob/main/cti-taxii-client-tutorial/examples/05-get-specific-object-no-pagination.py;
python3 05-get-specific-object-no-pagination.py
Which prints the Object;
{'more': False, 'objects': [{'created': '2014-05-08T09:00:00.000Z', 'id': 'indicator--cd981c25-8042-4166-8945-51178443bdac', 'indicator_types': ['file-hash-watchlist'], 'modified': '2014-05-08T09:00:00.000Z', 'name': 'File hash for Poison Ivy variant', 'pattern': "[file:hashes.'SHA-256' = 'ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c']", 'pattern_type': 'stix', 'spec_version': '2.1', 'type': 'indicator', 'valid_from': '2014-05-08T09:00:00.000000Z'}]}
You can see 'more': False indicating no more pages, however, it is entirely possible for this request to return more than one version of the same Object ID (and thus more than one page). Whilst my script does not handle pagination, you should account for dealing with more than one version of an Object in TAXII responses.
It is also possible filter Object responses by object_versions to be more specific with what Objects are returned. For example, in the following script I get the specific object_version=2014-05-08T09:00:00.000Z of Object indicator--cd981c25-8042-4166-8945-51178443bdac and print it; https://github.com/signalscorps/tutorials/blob/main/cti-taxii-client-tutorial/examples/06-filter-objects-by-version.py;
06-filter-objects-by-version.py
Which prints
{'more': False, 'objects': [{'created': '2014-05-08T09:00:00.000Z', 'id': 'indicator--cd981c25-8042-4166-8945-51178443bdac', 'indicator_types': ['file-hash-watchlist'], 'modified': '2014-05-08T09:00:00.000Z', 'name': 'File hash for Poison Ivy variant', 'pattern': "[file:hashes.'SHA-256' = 'ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c']", 'pattern_type': 'stix', 'spec_version': '2.1', 'type': 'indicator', 'valid_from': '2014-05-08T09:00:00.000000Z'}]}
cti-taxii-client also supports the publishing of Objects (add_objects). You can pass the STIX Objects using .add_objects in a JSON escaped STIX 2.1 Object (in the same way I showed with the API on the last tutorial).
Here is a script I wrote to create an Indicator SDO Object with the ID indicator--9ef5baeb-ad55-4c2c-aa89-ea5515dacf62 in the Collection ID 91a7b528-80eb-42ed-a74d-c6fbd5a26116. The script prints the status of the request (success, failure, or pending); https://github.com/signalscorps/tutorials/blob/main/cti-taxii-client-tutorial/examples/07-create-an-object.py;
python3 07-create-an-object.py
Which prints;
status: complete
id: 082ce7d7-7de0-42ca-bafa-098b5b5b0847
failure_count: 0
pending_count: 0
success_count: 1
Finally, delete operations are also covered by cti-taxii-client. In this script, I delete the Indicator previously created (indicator--9ef5baeb-ad55-4c2c-aa89-ea5515dacf62); https://github.com/signalscorps/tutorials/blob/main/cti-taxii-client-tutorial/examples/08-delete-an-object.py;
python3 08-delete-an-object.py
Which prints;
Successfully deleted
Hopefully some of these demo scripts have given you a brief overview of what is possible using cti-taxii-client. They are not designed to be perfect, and the scripts are far from production. Nor have I covered all its features.
As cti-taxii-client minimal implementation, there are some functions missing. That said, it is still a great starting point to build off or to use for testing the responses from a TAXII Server.
Other TAXII 2.1 Client options
There is another other open-source option out there (uncovered using a GitHub search);
- CyTAXII: CYTAXII2 is an Open Source offering from Cyware that provides developers with the support for interacting with the TAXII server using a Python library. It implements all TAXII services according to TAXII 2.X specifications.
If you know of any others, please do share them with me and I can include in this post.
CONGRATULATIONS!
You have made it to the end of this short course.
Hopefully the last few weeks have given you a deeper understanding of sharing and consuming cyber threat intelligence over TAXII 2.1.
Here are some useful links to bookmark following this course, some I have covered, some I have not, that you I find useful when working with TAXII 2.1:
- The TAXII 2.1 Specification will be your number one resource for looking information up
- The OASIS TAXII website offers a central point to discover additional content
- OASIS TAXII Server implementation (Medallion)
- OASIS TAXII Client implementation
If you have any questions about the content in this tutorial, please do not hesitate to drop us a message on Slack.
TAXII 2.1 Certification (Virtual and In Person)
The content used in this post is a small subset of our full training material used in our TAXII 2.1 training.
If you want to join a select group of certified TAXII 2.1 professionals, subscribe to our newsletter below to be notified of new course dates.
Discuss this post
Never miss an update
Sign up to receive new articles in your inbox as they published.