Posted by:

David Greenwood

David Greenwood, Chief of Signal

If you are reading this blog post via a 3rd party source it is very likely that many parts of it will not render correctly. Please view the post on signalscorps.com for the full interactive viewing experience.

In this post I will introduce the important concepts of CVEs and CPEs and explain how to work with them using the NVD API.

Note: Vulmatch is an ever evolving product therefore the concepts described in this post might not always be relevant for the latest version.

Vulmatch is a web app that allows users to be immediately alerted when a product they use has a vulnerability discovered.

When building Vulmatch I was keen to utilise many knowledge-bases for threat intelligence that exist to enrich CVEs and help analysts triage a vulnerabilities potential impact.

Over the next few posts, I am going to lift the lid on Vulmatch and explain exactly how it works and some of the design decisions made along the way so that you can build it yourself. Of course, you can sign up for free right now too.

CPEs

One of the key features of Vulmatch allows users to select the products they are using (so that they could be notified of vulnerabilities in them).

For that I needed a standardised way of describing products. That is where Common Platform Enumerations (CPEs) came in;

CPE is a structured naming scheme for information technology systems, software, and packages. Based upon the generic syntax for Uniform Resource Identifiers (URI), CPE includes a formal name format, a method for checking names against a system, and a description format for binding text and tests to a name.

CPEs were originally managed by MITRE but ownership has since been transferred to the US National Institute of Standard of Technology (NIST).

The most important part of a CPE is the URI – a computer readable format to describe the details of operating systems, software applications, or hardware devices.

Here is the CPE 2.3 URI schema;

cpe:2.3:<part>:<vendor>:<product>:
<version>:<update>:<edition>:<language>:<sw_edition>:<target_sw:>:<target_hw>:<other>

Where:

  • cpe: always cpe
  • 2.3: the cpe version (currently latest is 2.3)
  • <part>: The part attribute SHALL have one of these three string values:
    • a for applications,
    • o for operating systems,
    • h for hardware devices
  • <vendor>: described or identifies the person or organisation that manufactured or created the product
  • <product>: describes or identifies the most common and recognisable title or name of the product
  • <version>: vendor-specific alphanumeric strings characterising the particular release version of the product
  • <update>: vendor-specific alphanumeric strings characterising the particular update, service pack, or point release of the product.
  • <edition> assigned the logical value ANY (*) except where required for backward compatibility with version 2.2 of the CPE specification
  • <language>: valid language tags as defined by [RFC5646]
  • <sw_edition>: characterises how the product is tailored to a particular market or class of end users.
  • <target_sw>: characterises the software computing environment within which the product operates.
  • <target_hw>: characterises the instruction set architecture (e.g., x86) on which the product being described or identified operates
  • <other>: capture any other general descriptive or identifying information which is vendor- or product-specific and which does not logically fit in any other attribute value

Here is an example of a CPE URI (for Apple Quicktime v7.71.80.42);

cpe:2.3:a:apple:quicktime:7.71.80.42:*:*:*:*:*:*:*

Where;

  • part: a (application)
  • vendor: apple
  • product: quicktime
  • version: 7.71.80.42
  • update: *
  • edition: *
  • language: *
  • sw_edition: *
  • target_sw: *
  • target_hw: *
  • other: *

Where * equals ANY specified or unspecified option.

You can browse CPEs here. Here is the record shown in the example above: https://nvd.nist.gov/products/cpe/detail/165622

The NVD CPE data can be also accessed via their APIs.

To start using the NVD APIs you will need to request an API key here.

Once you have your API key, you can start making requests. To help you get started quickly, you can use the Postman Collection I have created.

There is only once CPE endpoint;

GET https://services.nvd.nist.gov/rest/json/cpes/1.0/

It has a range of parameters to filter the results to find products.

For example, I can use a full or partial cpeMatchString parameter (the CPE URI) to query the API for it;

GET https://services.nvd.nist.gov/rest/json/cpes/1.0/?apiKey=<<APIKEY>>&cpeMatchString=cpe:2.3:a:apple:quicktime:7.71.80.42:*:*:*:*:*:*:*

The response shows the friendly name of the software ("title": "Apple Quicktime 7.71.80.42") among other fields.

The response also shows the lastModifiedDate of the CPE, which is usually close to the release date of the product (although not always).

You do not need to pass all parts of a cpeMatchString (URI). cpeMatchString=cpe:2.3:a:apple is exactly the same as cpeMatchString=cpe:2.3:a:apple::: which is exactly the same as cpe:2.3:a:apple:*:*:*:*:*:*:*:*:*

If I wanted a list of all Apple (vendor=apple) applications (part=a) last updated in the first 3 months of this year (2022) I could run the following query;

GET https://services.nvd.nist.gov/rest/json/cpes/1.0/?apiKey=<<APIKEY>>&modStartDate=2022-01-01T00:00:00:001%20Z&modEndDate=2022-04-01T00:00:00:000%20Z&cpeMatchString=cpe:2.3:a:apple

Which currently returns information for 58 Apple (vendor) applications (part);

{
    "resultsPerPage": 20,
    "startIndex": 0,
    "totalResults": 58,
    "result": {

I can also get the response to print any CVEs IDs impacting the returned software;

GET https://services.nvd.nist.gov/rest/json/cpes/1.0/?apiKey=<<APIKEY>>&modStartDate=2022-01-01T00:00:00:000%20Z&modEndDate=2022-04-01T00:00:00:000%20Z&cpeMatchString=cpe:2.3:a:apple:swiftnio_http\\/2:1.19.1:*:*:*:*:swift:*:*

You will see there are escape characters in the match string. This is important when writing match strings by hand. Here two backslashes \\ escape the forward slash / present in the version string. For full information about when escaping is needed, read the CPE 2.3 naming spec document here.

In the response I can see the CPE, "cpe:2.3:a:apple:swiftnio_http\\/2:1.19.1:*:*:*:*:swift:*:*" (Apple SwiftNIO HTTP/2 1.19.1 for Swift), is affected by the following vulnerabilities; CVE-2022-0618, CVE-2022-24666, CVE-2022-24667, and CVE-2022-24668.

However, for our use-case it is better to perform the lookup the other way around (matching CVEs to CPEs) because CVEs have more detailed software information in software configurations that need to be observed (more on that later).

It is also important to check the deprecated field. By default, deprecated CPEs will be omitted unless you pass the argument includeDeprecated=true in your request.

CPE URIs can be deprecated for multiple reasons, but it ultimately means that the unique string is no longer considered accurate and instead a different string should be used.

The following request will return some deprecated records. Note the addition of the keyword arguement, used to retrieve records where a word or phrase is found in the CPE title or reference link fields.

GET https://services.nvd.nist.gov/rest/json/cpes/1.0/?apiKey=<<APIKEY>>&cpeMatchString=cpe:2.3:a:microsoft&includeDeprecated=true&keyword=visio
    {
        "deprecated": true,
        "cpe23Uri": "cpe:2.3:a:microsoft:visio_2003-:sp3:*:*:*:*:*:*:*",
        "lastModifiedDate": "2009-03-05T17:19Z",
        "titles": [
            {
                "title": "Microsoft Office Visio 2003 SP3",
                "lang": "en_US"
             }
        ],
        "refs": [],
        "deprecatedBy": [
            "cpe:2.3:a:microsoft:visio:2003:sp3:*:*:*:*:*:*"
        ],
        "vulnerabilities": []
    },

You can see when "deprecated": true a deprecatedBy value is present, pointing the up-to-date CPE record for this software.

For our use-case, identifying when future records become deprecated over time is important so that I can ensure 1) users are selecting the active CPE record and 2) that records selected by users that have since become deprecated point to the most recent CPE record.

On the initial backfill of data for Vulmatch this is not required as at the point of app install I only want currently active products.

The backfill can be carried out by paging through all results using startIndex (and creating longer pages using resultsPerPage (max is 2000)).

Request 1;

GET https://services.nvd.nist.gov/rest/json/cpes/1.0/?apiKey=<<APIKEY>>&startIndex=0&resultsPerPage=500&includeDeprecated=true
{
    "resultsPerPage": 500,
    "startIndex": 0,
    "totalResults": 861412,

Request 2;

GET https://services.nvd.nist.gov/rest/json/cpes/1.0/?apiKey=<<APIKEY>>&startIndex=1&resultsPerPage=1000&includeDeprecated=true
{
    "resultsPerPage": 500,
    "startIndex": 1,
    "totalResults": 861412,

And so on until resultsPerPage * (startIndex + 1) is >= (is greater than or equal to) totalResults.

Note. (startIndex + 1) is to account for the zero rated index.

There are a few other things to be mindful of when backfilling data.

Firstly, the NVD APIs can be unstable at times. Even more so the larger the response size (never use resultsPerPage=2000). 500 resultsPerPage seems to be a good trade-off.

NVD also recommend your application sleeps for several seconds between requests so that legitimate requests are not denied by the server (which implements various controls to stop abuse).

Once the backfill is complete, adding new or update CPE records (remembering to include includeDeprecated=true) can be achieved using the modStartDate and modEndDate parameters where modStartDate equals the last update check request time and modEndDate equals time the request is executed.

I look for updates daily because the CPE database is fairly static;

GET https://services.nvd.nist.gov/rest/json/cpes/1.0/?apiKey=<<APIKEY>>&includeDeprecated=true&modStartDate=2022-01-01T00:00:00:001%20Z&modEndDate=2022-01-02T00:00:00:000%20Z&startIndex=0&resultsPerPage=500

Usually all updated result will fit on a single page (with 500 results), but you should still expect to page through pages using startIndex.

The Vulmatch UI allows user to select products they are using vendor, product and version fields from these products (I do not expose all CPE fields to the user as it becomes much to complex for a user to manage – more on that in part 3).

Now that a user can select from a database of currently available products downloaded from the NVD API (and organise them into configurations), I now need CVE data to get the vulnerability details that affect these products.

CVEs

First, let me take a second to jump back and explain what a CVE is, for those unfamiliar.

In January 1999, the MITRE Corporation published “Towards a Common Enumeration of Vulnerabilities”.

Its aim; to identify, define, and catalog publicly disclosed cybersecurity vulnerabilities.

Very soon after this meeting, the original 321 Common Vulnerabilities and Exposures (CVE) Entries, including entries from previous years, was created and the CVE List was officially launched to the public in September 1999.

Each CVE has a unique ID. In its first iteration, 9,999 CVEs were allowed per year because CVE IDs were assigned using the format CVE-YYYY-NNNN (CVE-2001-1473). Currently, tens-of-thousands of CVEs are reported a year. To account for this explosion of CVEs, the CVE ID syntax was extended by adding one more digit to the N potion from four to five digits to CVE-YYYY-NNNNN in 2015.

Whilst CVEs are ultimately managed MITRE and a network of CNA (CVE numbering authorities), NIST (the same organisation managing CPEs) manages a more comprehensive analysis of CVEs in the US National Vulnerability Database (NVD).

NVD CVE lifecycle

The NVD is tasked with analysing each CVE once it has been published to the CVE List (MITREs list) using the reference information provided with the CVE and any publicly available information at the time of analysis to associate Reference Tags, Common Vulnerability Scoring System (CVSS) v2.0, CVSS v3.1, CWE, and CPE Applicability statements.

Once a CVE is published and NVD analysis is provided, there may also be additional maintenance or modifications made. References may be added, descriptions may be updated, or a request may be made to have a set of CVE IDs reorganised (such as one CVE ID being split into several). Furthermore, the validity of an individual CVE ID can be disputed by the vendor that can result in the CVE being revoked (e.g at the time of writing CVE-2022-27948 is disputed).

The full process is described here.

You can browse the NVD CVE database here to see examples of the data the NVD team include with each CVE.

For Vulmatch, the key takeaways for working with CVEs are that 1) they are published regularly, 2) the NVD is one of the most comprehensive public sources of CVE analysis and, 3) NVD can updated the data inside at CVE record once it is published.

The NVD CVE data can be also accessed via their APIs. It is the same API used for CPEs, so no new API key is needed.

IMPORTANT: In late 2022 the NVD will release the 2.0 version of its APIs. This post considers the currently available 1.x version.

There are two endpoints for the v1 CVE API;

  1. GET CVEs: Useful for getting new CVEs and those that have been recently updated using a range of filters.
GET https://services.nvd.nist.gov/rest/json/cves/1.0/?apiKey=<<APIKEY>>
  1. GET CVE: Provides details on a specific CVE
GET https://services.nvd.nist.gov/rest/json/cve/1.0/<<CVD_ID>>?apiKey=<<APIKEY>>

A simple request to the GET CVEs endpoint will search the entire NVD database of CVEs. When I created this post (2022-07-23T10:46Z) there were 180,783 CVE records in the database;

GET https://services.nvd.nist.gov/rest/json/cves/1.0/?apiKey=<<APIKEY>>
{
  "resultsPerPage": 20,
  "startIndex": 0,
  "totalResults": 180783,

Using the pubStartDate and pubEndDate parameters I can see the number of vulnerabilities (CVEs) identified in the first three months of this year (2022);

GET https://services.nvd.nist.gov/rest/json/cves/1.0/?apiKey=<<APIKEY>>}&pubStartDate=2022-01-01T00:00:00:001%20Z&pubEndDate=2022-04-01T00:00:00:000%20Z
{
    "resultsPerPage": 20,
    "startIndex": 0,
    "totalResults": 6019,

6019 CVEs!

You can identify each CVE and its detail inside the cve objects returned.

        "cve": {
          "data_type": "CVE",
          "data_format": "MITRE",
          "data_version": "4.0",
          "CVE_data_meta": {
            "ID": "CVE-2015-8965",
            "ASSIGNER": "[email protected]"
          },

The actual CVE ID is found in the cve.CVE_data_meta.ID value, shown above.

I can also query the GET CVE for this individual CVE;

GET https://services.nvd.nist.gov/rest/json/cve/1.0/CVE-2015-8965?apiKey=<<APIKEY>>

You will see the response payload for this request is exactly the same as using the GET CVEs response (the benefit of using it is that the data request is more effecient to perform).

If you look at the NVD public site for a CVE, you will be able to start to see how the page is formed using data in the response.

For example; https://nvd.nist.gov/vuln/detail/CVE-2015-8965

For each CVE_Items (a CVE)…

The description section is rendered from cve.description.

References from cve.references.

cve.problemtype contains Weakness Enumeration information.

The severity (CVSS scores) is covered in the impact section.

And configurations contains Known Affected Software Configurations (using combinations of CPE patterns).

You will also see publishedDate and lastModifiedDate values. These is useful for finding updates to CVEs.

For newly submitted CVEs cve.publishedDate = cve.lastModifiedDate. Here is an example;

https://nvd.nist.gov/vuln/detail/CVE-2022-1145

CVE-2022-1145

GET https://services.nvd.nist.gov/rest/json/cve/1.0/CVE-2022-1145?apiKey=<<APIKEY>>

You will see in the API response "cve"."publishedDate": "2022-07-23T00:15Z" and "cve"."lastModifiedDate": "2022-07-23T00:15Z".

Notice how the screenshot RECIEVED at the top.

Sometime you will also see AWAITING ANALYSIS where cve.publishedDate = cve.lastModifiedDate. Here is an example;

https://nvd.nist.gov/vuln/detail/CVE-2022-20141

CVE-2017-20141

GET https://services.nvd.nist.gov/rest/json/cve/1.0/CVE-2017-20141?apiKey=<<APIKEY>>

It is also possible to come across CVE that have been rejected. If a CVE is rejected, you will see it on the NVD website but not via the API. Here is an example;

https://nvd.nist.gov/vuln/detail/CVE-2022-33014

CVE-2022-33014

GET https://services.nvd.nist.gov/rest/json/cve/1.0/CVE-2022-33014?apiKey=<<APIKEY>>

Notice how it show REJECTED at the top of the page.

However, the API response shows;

{
    "message": "Unable to find vuln CVE-2022-33014"
}

Generally for CVEs older than 3 or 4 days, you will see cve.lastModifiedDate > cve.publishedDate when the NVD team have provided a more detailed review (e.g. assigning CVSS scores).

Here is an example of a CVE approved and reviewed by NVD;

https://nvd.nist.gov/vuln/detail/CVE-2022-21455

CVE-2022-21455

GET https://services.nvd.nist.gov/rest/json/cve/1.0/CVE-2022-21455?addOns=dictionaryCpes&apiKey=<<APIKEY>>

And one final example; this time a CVE already approved by NVD, but awaiting further review which may result in further changes to the information provided;

https://nvd.nist.gov/vuln/detail/CVE-2021-44228

CVE-2021-44228

Notice how it show MODIFIED at the top of the page.

In the case of modified CVEs, the API responses do not show the changes that were made, it only shows the latest record.

The CPE ("cve"."configurations"."nodes") contains cpe_matches that contain a lastModifiedDate. Though this date refers to the CPE creation and modification (also shown via responses in the CPE API), seperate to the CVE.

There is simply no way to track when updates happened and what was changed using a single API request.

To demonstrate, in the change history section of the web app you will see some recently updated references on 2022-07-22 for CVE-2021-44228;

CVE-2021-44228 change history

e.g. Added - Reference - ` http://packetstormsecurity.com/files/167794/Open-Xchange-App-Suite-7.10.x-Cross-Site-Scripting-Command-Injection.html`

Though looking at the API response;

GET https://services.nvd.nist.gov/rest/json/cve/1.0/CVE-2021-44228?apiKey=<<APIKEY>>

The "cve"."references"."reference_data" section of the response shows the newly added data, but not when it was added against the CVE record;

{
    "url": "http://packetstormsecurity.com/files/167794/Open-Xchange-App-Suite-7.10.x-Cross-Site-Scripting-Command-Injection.html",
    "name": "http://packetstormsecurity.com/files/167794/Open-Xchange-App-Suite-7.10.x-Cross-Site-Scripting-Command-Injection.html",
    "refsource": "MISC",
    "tags": []
}

Therefore for our use-case, I need to poll the NVD API regularly to pull in updates and manually identify the differences since last update for all the fields.

It is important to note this does mean for historic records (downloaded on backfill) there is no way for us to identify changes as I will only be able to view the most recent record at time of request.

To initially backfill data to put all current CVE records in the Vulmatch database I can page though results using a mix of pubStartDate and pubEndDate parameters in addition to paging parameters.

My recommendation is to ingest data on a month by month basis like so;

Request 1 (date of first CVE is October 1999)

GET https://services.nvd.nist.gov/rest/json/cves/1.0/?apiKey=<<APIKEY>>&pubStartDate=1999-10-01T00:00:00:001%20Z&pubEndDate=1999-11-01T00:00:00:000%20Z&addOns=dictionaryCpes&resultsPerPage=500&sortOrder=publishedDate&startIndex=0
{
    "resultsPerPage": 57,
    "startIndex": 0,
    "totalResults": 57,

Until resultsPerPage * (startIndex + 1) is >= (is greater than or equal to) totalResults.

Note. (startIndex + 1) is to account for the zero rated index.

In the above example I would not paginate any further (using startIndex=1)because 57 * (0 + 1) = 57 which is equal to total results.

At which point I would move to the next calendar month, November 1999 (pubStartDate=1999-11-01T00:00:00:001%20Z&pubEndDate=1999-12-01T00:00:00:000%20Z&), and so on.

Like with request to the CPE endpoint, there should be a few seconds between each request to ensure it is not rejected by the server.

Once the backfill is complete I run daily requests to check for CVE records that have been added or updated. I run the request daily as it appear NVD update records in batch once every working day (Mon - Fri).

For this daily check, I can use the modStartDate and modEndDate parameters for the previous 24 hours from execution time (modEndDate = time of execution, modStartDate = modEndDate - 24h) like so;

GET https://services.nvd.nist.gov/rest/json/cves/1.0/?apiKey=<<APIKEY>>&modStartDate=2022-04-01T00:00:00:001%20Z&modEndDate=2022-04-02T00:00:00:000%20Z&addOns=dictionaryCpes&resultsPerPage=500&startIndex=0

Usually all new and updated results will fit on a single page (with 500 results), but you should still expect to page through pages using startIndex.

This API query will also capture new CVEs as even new CVEs have a lastModifiedDate value, even if it equals publishedDate, as explained earlier.

Next time: Known Affected Software (CPE) Configurations

This post introduced the concepts of CPEs and CVEs, and how to download the data for these records.

However, in order to match CPEs a Vulmatch user has selected to a CVE, we need to understand the logic of the configurations section inside a CVE response.

All will be explained in the next post.




Our brand new Discord!

Like this blog?

Sign up to receive new posts in your inbox.


Stixify

Stixify. Extract machine readable intelligence from unstructured data.

Extract machine readable intelligence from unstructured data.

Obstracts

Obstracts

Turn any blog into structured threat intelligence.


Vulmatch

Vulmatch

Know when software you use is vulnerable, how it is being exploited, and how to detect an attack.

SIEM Rules

SIEM Rules. Your detection engineering database.

View, modify, and deploy SIEM rules for threat hunting.