TAXII 2.1 102: Consuming Objects in Collections

Posted by:

If you are reading this blog post via a 3rd party source it is very likely that many parts of it will not render correctly. Please view the post on signalscorps.com for the full interactive viewing experience.

In this post I will walk through how TAXII Clients logically consume cyber threat intelligence from Collections using a TAXII 2.1 API.

Note: this post is written for OASIS TAXII version 2.1. The concepts discussed are not always correct for earlier versions of OASIS TAXII. This post also talks about some specifics related to the Signal Corps implementation of a TAXII 2.1 Server, namely the exclusive use of STIX 2.1 content.

A TAXII Collection is a logical grouping of threat intelligence that enables the exchange of information between a TAXII Client and a TAXII Server via a TAXII API in a request-response manner.

The following flows demonstrates the logic of how TAXII Clients generally interact with a TAXII Servers API to get STIX 2.1 structured cyber threat intelligence.

First, the TAXII Client needs to determine available Collections hosted under the API Root. For this the Get Collections endpoint can be used.

The GET request for the endpoint takes the form HOST/<API_ROOT>/collections/.

For example;

curl -i https://MY_HOST/api/v1/group1/collections/ \
 -H "Accept: application/taxii+json;version=2.1" \
 -H "Authorization: Basic <BASE64 ENCODED CREDENTIALS>"  \
 -A "Signals Corps Curl Demo Client/1.0"

Which returns each Collection under this API root;

HTTP/1.1 200 OK
Content-Length: 637
Content-Type: application/taxii+json;version=2.1
Connection: close
{
  "collections": [
    {
      "id": "91a7b528-80eb-42ed-a74d-c6fbd5a26116",
      "title": "High Value Indicator Collection",
      "description": "This data collection contains high value IOCs",
      "can_read": true,
      "can_write": false,
      "media_types": [
        "application/stix+json;version=2.1"
      ]
    },
    {
      "id": "52892447-4d7e-4f70-b94d-d7f22742ff63",
      "title": "Another Collection",
      "description": "This data collection is for collecting current IOCs",
      "can_read": true,
      "can_write": true,
      "media_types": [
        "application/stix+json;version=2.1"
      ]
    }
  ]
}

You should note the endpoint could return Collections the authenticated user cannot read ("can_read": false). This highlights how users permissions can be set in two places;

to access the API Root (covered in the last post), and
to access the Collections under the API Root (as shown above).

You can also request a specific Collection inside the API root using the Collection id from the last request in a request against the Get a Collection Endpoint.

The GET request for the endpoint takes the form HOST/<API_ROOT>/collections/<COLLECTION_ID>/.

For example, to request the Collection 91a7b528-80eb-42ed-a74d-c6fbd5a26116;

curl -i https://MY_HOST/api/v1/group1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/ \
 -H "Accept: application/taxii+json;version=2.1" \
 -H "Authorization: Basic <BASE64 ENCODED CREDENTIALS>"  \
 -A "Signals Corps Curl Demo Client/1.0"

This time, only one Collection will be returned;

HTTP/1.1 200 OK
Content-Length: 268
Content-Type: application/taxii+json;version=2.1
Connection: close
{
  "id": "91a7b528-80eb-42ed-a74d-c6fbd5a26116",
  "title": "High Value Indicator Collection",
  "description": "This data collection contains high value IOCs",
  "can_read": true,
  "can_write": false,
  "media_types": [
    "application/stix+json;version=2.1"
  ]
}

The body of the response is identical to the Get Collections endpoint, it just prints the details of a single Collection.

Get Object Manifests

To get an overview of what is inside a Collection (the cyber threat intelligence Objects), I can request the Object Manifests using the Get Object Manifests endpoint.

The GET request for the endpoint takes the form HOST/<API_ROOT>/collections/<ID>/manifest/?<FILTERS>.

For example;

curl -i https://MY_HOST/api/v1/group1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/manifest/ \
 -H "Accept: application/taxii+json;version=2.1" \
 -H "Authorization: Basic <BASE64 ENCODED CREDENTIALS>"  \
 -A "Signals Corps Curl Demo Client/1.0" \

Which returns a summary of the Objects (in this case STIX 2.1);

HTTP/1.1 200 OK
Content-Length: 726
Content-Type: application/taxii+json;version=2.1
X-TAXII-Date-Added-First: 2016-11-04T03:04:05.000Z
X-TAXII-Date-Added-Last: 2016-11-04T10:29:06.000Z
Connection: close
{
  "more": false,
  "next": 0,
  "objects": [
    {
      "id": "indicator--29aba82c-5393-42a8-9edb-6a2cb1df070b",
      "date_added": "2016-11-04T03:04:05.000Z",
      "version": "2016-11-03T12:30:59.000Z",
      "media_type": "application/stix+json;version=2.1"
    },
    {
      "id": "indicator--ef0b28e1-308c-4a30-8770-9b4851b260a5",
      "date_added": "2016-11-04T10:29:06.000Z",
      "version": "2016-11-04T10:29:06.000Z",
      "media_type": "application/stix+json;version=2.1"
    },
    {
      "id": "indicator--ef0b28e1-308c-4a30-8770-9b4851b260a5",
      "date_added": "2016-11-04T10:29:06.000Z",
      "version": "2017-01-22T00:00:00.000Z",
      "media_type": "application/stix+json;version=2.1"
    }
  ]
}

For Object and Manifest Endpoints, the Objects are returned sorted in ascending order by the date_added property. Meaning, the most recently added object is last in the list.

This endpoint returns two additional headers, X-TAXII-Date-Added-First and X-TAXII-Date-Added-Last. These headers define the earliest and latest date/time values for the Objects in the response.

It is also important to point out the difference between date_added and version properties in the returned response. date_added refers to the time the object was added to the TAXII server. The added_after parameter is not in any way related to dates or times in a STIX Object or any other CTI Object. Whereas version equals the modified property of the STIX Object. This means the Object and Manifest Endpoints might return multiple versions of the same Object id (unless version is explicitly specified using a filter).

The following filtering parameters can be used with the Get Object Manifests endpoint;

added_after: e.g. 2015-01-01T00:00:00.000Z
limit: e.g. 100
next: e.g. 2
match[id]: e.g. indicator--29aba82c-5393-42a8-9edb-6a2cb1df070b,indicator--ef0b28e1-308c-4a30-8770-9b4851b260a5
match[type]: e.g. indicator,sighting
match[version]: e.g. first,2018-03-02T01:01:01.123Z,last
match[spec_version]: e.g. 2.0,2.1

Note, multiple values passed to fields that support match values can be passed with a comma ‘,’ which is treated as a logical OR. For example, ?match[spec_version]=2.0,2.1 returns object with versions both 2.0 OR 2.1.

For example I can filter the results to only include a certain type of STIX Object (match[type]=indicator) with a specific ID match[id]=indicator--ef0b28e1-308c-4a30-8770-9b4851b260a5 and a specific version of that Object (match[version]=2016-11-04T10:29:06.000Z);

curl -i https://MY_HOST/api/v1/group1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/manifest/?match[version]=2016-11-04T10:29:06.000Z&match[type]=indicator&match[id]=indicator--ef0b28e1-308c-4a30-8770-9b4851b260a5 \
 -H "Accept: application/taxii+json;version=2.1" \
 -H "Authorization: Basic <BASE64 ENCODED CREDENTIALS>"  \
 -A "Signals Corps Curl Demo Client/1.0"

Returns;

HTTP/1.1 200 OK
Content-Length: 276
Content-Type: application/taxii+json;version=2.1
X-TAXII-Date-Added-First: 2016-11-04T10:29:06.000Z
X-TAXII-Date-Added-Last: 2016-11-04T10:29:06.000Z
Connection: close
{
  "more": false,
  "next": 0,
  "objects": [
    {
      "id": "indicator--ef0b28e1-308c-4a30-8770-9b4851b260a5",
      "date_added": "2016-11-04T10:29:06.000Z",
      "version": "2016-11-04T10:29:06.000Z",
      "media_type": "application/stix+json;version=2.1"
    }
  ]
}

You can see now only one version (2016-11-04T10:29:06.000Z) of the object (indicator--ef0b28e1-308c-4a30-8770-9b4851b260a5) is returned due to the filters applied.

If "more": true the "next" property would indicate there is another page of results (e.g. 1). I could then paginate through results using the next parameter until "more": false and "next": 0 – telling the client it had reached the last page. Here is an example of using the next parameter to go to page 1 (note, pages in our TAXII server implementation start at page 0).

curl -i https://MY_HOST/api/v1/group1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/manifest/?next=1 \
 -H "Accept: application/taxii+json;version=2.1" \
 -H "Authorization: Basic <BASE64 ENCODED CREDENTIALS>"  \
 -A "Signals Corps Curl Demo Client/1.0"

This endpoint is paticularly useful for a client in identifying new or updated Objects between each poll. For example, the added_after property could be specified with each request with the timestamp equal to the last poll time. That way, with each request new or updated Objects (including deleted Objects) could be easily identified by the TAXII Client. The TAXII Client could subsequetly get these Objects.

Get Objects

To get the full STIX 2.1 Objects, the Get Objects Endpoint can be used.

This Endpoint retrieves the full Objects from a Collection. Clients can search for Objects in the Collection, retrieve all Objects in a Collection, or paginate through Objects in the Collection.

The same filters used on the Get Object Manifests endpoint can be applied to the Objects endpoint.

The GET request for the endpoint takes the form HOST/<API_ROOT>/collections/<COLLECTION_ID>/objects/?<FILTERS>.

For example, filtering for a specific Object ID;

curl -i https://MY_HOST/api/v1/group1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/?match[type]=indicator&match[id]=indicator--ef0b28e1-308c-4a30-8770-9b4851b260a5 \
 -H "Accept: application/taxii+json;version=2.1" \
 -H "Authorization: Basic <BASE64 ENCODED CREDENTIALS>"  \
 -A "Signals Corps Curl Demo Client/1.0"

Which returns the full content of each matching STIX 2.1 Object;

HTTP/1.1 200 OK
Content-Length: 1293
Content-Type: application/taxii+json;version=2.1
X-TAXII-Date-Added-First: 2016-11-04T10:29:06.000Z
X-TAXII-Date-Added-Last: 2016-11-04T10:29:06.000Z
Connection: close
{
  "more": false,
  "next": 0,
  "objects": [
    {
        "type": "indicator",
        "spec_version": "2.1",
        "id": "indicator--ef0b28e1-308c-4a30-8770-9b4851b260a5",
        "created": "2016-11-04T10:29:06.000Z",
        "modified": "2016-11-04T10:29:06.000Z",
        "name": "Malicious site hosting downloader",
        "description": "This organized threat actor group operates to create profit from all types of crime.",
        "indicator_types": [
            "malicious-activity"
        ],
        "pattern": "[url:value = 'http://x4z9arb.cn/4712/']",
        "pattern_type": "stix",
        "valid_from": "2016-11-04T10:29:06.000Z"
    },
    {
        "type": "indicator",
        "spec_version": "2.1",
        "id": "indicator--ef0b28e1-308c-4a30-8770-9b4851b260a5",
        "created": "2016-11-04T10:29:06.000Z",
        "modified": "2017-01-22T00:00:00.000Z",
        "name": "UPDATED Malicious site hosting downloader",
        "description": "UPDATED This organized threat actor group operates to create profit from all types of crime.",
        "indicator_types": [
            "malicious-activity"
        ],
        "pattern": "[url:value = 'http://x4z9arb.cn/4712/']",
        "pattern_type": "stix",
        "valid_from": "2016-11-04T10:29:06.000Z"
    }
  ]
}

Get an Object

In the last example I filtered Objects by id. It is also possible to return all versions of an Object (or specific versions if filtered) using the Get an Object endpoint too. Of course, the use of this endpoint assumes you know the Objects ID.

The GET request for the endpoint takes the form HOST/<API_ROOT>/collections/<COLLECTION_ID>/objects/<OBJECT_ID>/?<FILTERS>.

For example;

curl -i https://MY_HOST/api/v1/group1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/indicator--ef0b28e1-308c-4a30-8770-9b4851b260a5/ \
 -H "Accept: application/taxii+json;version=2.1" \
 -H "Authorization: Basic <BASE64 ENCODED CREDENTIALS>" \
 -A "Signals Corps Curl Demo Client/1.0"

Which returns the same response as the earlier request;

HTTP/1.1 200 OK
Content-Length: 1293
Content-Type: application/taxii+json;version=2.1
X-TAXII-Date-Added-First: 2016-11-04T10:29:06.000Z
X-TAXII-Date-Added-Last: 2016-11-04T10:29:06.000Z
Connection: close
{
  "more": false,
  "next": 0,
  "objects": [
    {
        "type": "indicator",
        "spec_version": "2.1",
        "id": "indicator--ef0b28e1-308c-4a30-8770-9b4851b260a5",
        "created": "2016-11-04T10:29:06.000Z",
        "modified": "2016-11-04T10:29:06.000Z",
        "name": "Malicious site hosting downloader",
        "description": "This organized threat actor group operates to create profit from all types of crime.",
        "indicator_types": [
            "malicious-activity"
        ],
        "pattern": "[url:value = 'http://x4z9arb.cn/4712/']",
        "pattern_type": "stix",
        "valid_from": "2016-11-04T10:29:06.000Z"
    },
    {
        "type": "indicator",
        "spec_version": "2.1",
        "id": "indicator--ef0b28e1-308c-4a30-8770-9b4851b260a5",
        "created": "2016-11-04T10:29:06.000Z",
        "modified": "2017-01-22T00:00:00.000Z",
        "name": "UPDATED Malicious site hosting downloader",
        "description": "UPDATED This organized threat actor group operates to create profit from all types of crime.",
        "indicator_types": [
            "malicious-activity"
        ],
        "pattern": "[url:value = 'http://x4z9arb.cn/4712/']",
        "pattern_type": "stix",
        "valid_from": "2016-11-04T10:29:06.000Z"
    }
  ]
}

This endpoint allows for the following query parameters; added_after, limit, next, match[version], and match[spec_version], allowing you to filter out versions of the Object.

Get Object Versions

It is also possible to find all versions of an Object (or specific versions if filtered) without getting the entire Object using the Get Object Versions endpoint.

The GET request for the endpoint takes the form HOST/<API_ROOT>/collections/<COLLECTION_ID>/objects/<OBJECT_ID>/versions/?<FILTERS>.

For example;

curl -i https://MY_HOST/api/v1/group1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/indicator--ef0b28e1-308c-4a30-8770-9b4851b260a5/versions/ \
 -H "Accept: application/taxii+json;version=2.1" \
 -H "Authorization: Basic <BASE64 ENCODED CREDENTIALS>" \
 -A "Signals Corps Curl Demo Client/1.0" 

Which returns;

HTTP/1.1 200 OK
Content-Length: 115
Content-Type: application/taxii+json;version=2.1
X-TAXII-Date-Added-First: 2016-11-04T10:29:06.000Z
X-TAXII-Date-Added-Last: 2016-11-04T10:29:06.000Z
Connection: close
{
  "more": false,
  "next": 0,
  "versions": [
  	"2016-11-04T10:29:06.000Z",
    "2017-01-22T00:00:00.000Z"
  ]
}

This endpoint allows for the following query parameters; added_after, limit, next, and match[spec_version], allowing you to filter out versions of the Object.

A note on custom properties

The TAXII 2.1 also accounts for custom properties in the response body. They are used in a similar way to STIX 2.1 custom properties with the property name prefixed x_.

Custom properties can used anywhere in the TAXII Data Types.

For example, inside the envelope;

HTTP/1.1 200 OK
Content-Length: 138
Content-Type: application/taxii+json;version=2.1
Connection: close
{
  "more": false,
  "next": 0,
  "x_total_pages": 5,
  "versions": [
    "2016-11-04T10:29:06.000Z",
    "2017-01-22T00:00:00.000Z"
  ]
}

Or inside a Collection;

HTTP/1.1 200 OK
Content-Length: 305
Content-Type: application/taxii+json;version=2.1
Connection: close
{
  "id": "91a7b528-80eb-42ed-a74d-c6fbd5a26116",
  "title": "High Value Indicator Collection",
  "description": "This data collection contains high value IOCs",
  "can_read": true,
  "can_write": false,
  "media_types": [
    "application/stix+json;version=2.1"
  ],
  "x_tlp": {
    "tlp": "green"
  }
}

It should be pointed out though these are only useful when building your own TAXII Clients that can be used to parse these custom properties. Generic TAXII Servers will likely silently ignore any custom properties. It might also result in the TAXII server refusing to process the response.

Next Up: Adding and Deleting Objects

If you are an intelligence producer you will not only be consuming cyber threat intelligence Objects as shown in this post, you will also be creating them.

A TAXII 2.1 Servers API can be used to add new Objects, add new versions of Objects, or delete them. I will show you how to manage the intelligence inside a Collection in the next post.

TAXII 2.1 Certification (Virtual and In Person)

The content used in this post is a small subset of our full training material used in our TAXII 2.1 training.

If you want to join a select group of certified TAXII 2.1 professionals, subscribe to our newsletter below to be notified of new course dates.

Discuss this post

JOIN US