Documentation Index

Fetch the complete documentation index at: https://help.nucleussec.com/llms.txt

Use this file to discover all available pages before exploring further.

AWS S3 Bucket Data Export

  • 10 minute(s) read
Prev Next

You can configure the AWS connector to upload all active asset and finding data from your Nucleus project to your own S3 buckets for your use outside of the Nucleus platform. Only active, manually mitigated, and custom findings are uploaded.

Deployment Restriction

At this time, this feature is only available for customers using Nucleus on AWS. GCP, Azure, and Self-Hosted customers may not use this feature.

Before you get started, please review the documentations for Creating a cross-account role and setting permissions and follow the steps to setup permissions for that role to allow S3 exports

Setting up S3 data uploads

  1. Log in to your Nucleus project.

  2. From the navigation bar on the left, under Project Administration, select Connector Setup.

  3. Select Amazon Web Services.
    aws-connector-icon.png

  4. In the General tab, click the green plus button to add a new Role. Alternatively, you can bulk import credentials using a CSV file as detailed below.

  5. In the Label field, enter a label for the AWS account.

  6. In the Role ARN field, enter the Amazon Resource Name (ARN) of the role used for the connection.

  7. Click Verify Credentials. If the credentials were entered correctly, a message confirming a successful connection will appear.

  8. Select the S3 Data tab at the top.

  9. Click the green plus button to add a new S3 bucket. Alternatively, you can bulk import credentials using a CSV file as detailed below.

  10. In the Label field, enter a label for the AWS account.

  11. In the S3 Bucket field, enter the address of the S3 bucket.

  12. From the Region menu, select the region the S3 bucket is located in.

  13. In the Role ARN field, enter the Amazon Resource Name (ARN) of the role used for the connection.

  14. Click Verify Credentials. If the credentials were entered correctly, a message confirming a successful connection will appear.

  15. Set the frequency to upload data.

    • To upload data on a set schedule:
      1. Select the Upload asset data (daily) checkbox.
      2. From the Start time menu, select the time asset data will upload every day. The time selected is in Coordinated Universal Time (UTC).
      3. From the Buckets menu, select which buckets the asset data will upload to.
      4. Select the Upload finding data (weekly) checkbox.
      5. From the Day menu, select which day of the week finding data will upload.
      6. From the Start time menu, select the time finding data will upload every day. The time selected is in Coordinated Universal Time (UTC).
      7. From the Buckets menu, select which buckets the finding data will upload to.
    • To upload data in realtime:
      1. Select the Upload finding deltas as new scans are ingested checkbox. When a new custom finding is created, or a custom finding is edited (any changed parameter, either at the unique level or the justification level), the finding will be uploaded to S3.
      2. From the Buckets menu, select which buckets data will upload to.
    • To upload all asset and finding data immediately:
      1. Select the Upload entire asset and finding data from Nucleus to S3 on save checkbox.
      2. From the Buckets menu, select which buckets data will upload to.

  16. Optionally select Zip files prior to uploading them to S3 to compress the file size of the data uploaded to the S3 bucket.

  17. Click Save & Finish.

Bulk import credentials template

If your organization has many S3 buckets, you can bulk import role ARNs by clicking Bulk Import Buckets and uploading a CSV structured in the following way:

label,s3bucket,region,crossaccountrole

Example template

The following is an example CloudFormation template that can be deployed to AWS accounts to provide access to S3 buckets.

Warning

When using the template below, replace ACCOUNT_ID_FROM_NUCLEUS, EXTERNAL_ID_FROM_NUCLEUS_PROJECT, and BUCKET_NAME_# with the relevant AWS Account, AWS External ID, and bucket name values.

AWSTemplateFormatVersion: "2010-09-09"
Resources:
  NucleusAWSIntegrationRole:
    Type: "AWS::IAM::Role"
    Properties:
      RoleName: "NucleusAWSS3BucketConnectorRole"
      Description: "Role used by a Nucleus Project AWS connector to upload asset and finding data to one or more S3 buckets."
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              AWS:
                - "arn:aws:iam::ACCOUNT_ID_FROM_NUCLEUS:root"
            Action:
              - "sts:AssumeRole"
            Condition:
              StringEquals:
                "sts:ExternalId": "EXTERNAL_ID_FROM_NUCLEUS_PROJECT"
      Policies:
        - PolicyName: "NucleusAWSConnectorPolicy"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action: "iam:ListAccountAliases"
                Resource: "*"
              - Effect: Allow
                Action: "s3:PutObject"
                Resource:
                  - "arn:aws:s3:::BUCKET_NAME_1"
                  - "arn:aws:s3:::BUCKET_NAME_1/*"
                  - "arn:aws:s3:::BUCKET_NAME_2"
                  - "arn:aws:s3:::BUCKET_NAME_2/*"

About uploaded data

Format

The data is uploaded in JSON format and separated into paginated files for each type of top-level object.

These files are uploaded into three directories; /assets, /findings and /externalIssues. The /assets directory contains data about all assets, including those without any findings associated with them, and includes metadata regarding the assets themselves (e.g., host information, Nucleus statistics, etc). The /findings directory contains data for all findings, including associated assets, groups, instances, justifications, and teams. The /externalIssues directory contains data about issues created by Nucleus in 3rd party Issue Trackers like Jira and ServiceNow.

Files with full in their name contain a complete upload of all data. Alternatively, files with delta in their name contain only data that has changed since the last scan (e.g., new findings, changed values).

Note: Delta uploads for external issues is not currently supported.

Object Description File Name
Assets Anything that can have a vulnerability correlated with it. This object contains data for all assets. /assets/asset-data-FULL/DELTA-TIMESTAMP
Asset Findings Anything that can have a vulnerability correlated with it. This object only contains data for assets with findings associated with it. /findings/assets-finding-data-FULL/DELTA-TIMESTAMP
Asset Groups Flexible collections of assets that allow you to organize your assets across your entire organization. /findings/assetGroups-finding-data-FULL/DELTA-TIMESTAMP
Findings Unique (top-level) and instance-level vulnerabilities that exist on assets.

When a finding is detected in an asset, that makes it an instance of that finding. If the same vulnerability is discovered in 5 assets, there will be 5 instances of that vulnerability.		 'Vuln' and 'Vuln-Compliance' type unique findings:
/findings/findings-finding-data-FULL/DELTA-TIMESTAMP

'Compliance' and 'Vuln-Compliance' type unique findings:
/findings/compliance-findings-finding-data-FULL/DELTA-TIMESTAMP

'Vuln' and 'Vuln-Compliance' type findings:
/instances/instances-finding-data-FULL/DELTA-TIMESTAMP

'Compliance' and 'Vuln-Compliance' type findings:
/instances/compliance-instances-finding-data-FULL/DELTA-TIMESTAMP
Justifications Justifications track the changes (e.g., a status change) to instances over time and subsequent scans. Any time there is a change to a finding (manually or via scan) there will be a justification record for that change. /findings/justifications-finding-data-FULL/DELTA-TIMESTAMP
Teams Groups of users that collectively own assets and vulnerabilities in Nucleus. These Teams can then be linked to assets to define asset ownership, and then be further linked to vulnerabilities to define vulnerability ownership. /teams-finding-data-FULL/DELTA-TIMESTAMP
External Issues External issues are created by Nucleus through integrations with 3rd party Issue Trackers like Jira and ServiceNow. This object can be linked to finding instances. /externalIssues/externalissues-instances-finding-data-FULL/DELTA-TIMESTAMP
About asset files

The /assets/asset-data-full-TIMESTAMP files contain data for all assets, including those without any findings associated with them. The /findings/assets-finding-data-FULL/DELTA-TIMESTAMP files contain data only for assets with findings associated with them.

Additionally, a "connectivity-verification" directory is created when when first verifying the S3 bucket connection. This directory contains empty, timestamped JSON files to confirm the AWS connector can successfully upload files.

These files are connected to each other via the following joining keys:

  • Findings and Instances are connected via finding_key.
  • Findings and Justifications are connected via finding_number.
  • Instances and Justifications are connected via instance_key.
  • Instances and External Issues are connected viainstance_key.
  • Assets and Instances are connected via host_id (where assets.asset_id = instances.host_id).

Schemas

Assets

Primary Key Type
host_id INT
Foreign Keys Type
ip_address VARCHAR(64)
host_name VARCHAR(2048
host_type VARCHAR(128)
host_criticality VARCHAR(16)
operating_system_name VARCHAR(256)


Asset Groups

Foreign Keys Type
host_id INT
asset_group_sources VARCHAR(16)
asset_group_name VARCHAR(256)


Findings

Foreign Keys Type
finding_number VARCHAR(251)
scan_type VARCHAR(24)
finding_type VARCHAR(24)
finding_severity VARCHAR(32)
finding_severity_adjusted VARCHAR(32) NULL
finding_exploitable TINYTIN(1)
finding_name VARCHAR(128)
finding_cve VARCHAR(8192)
finding_iava VARCHAR(170)
finding_type VARCHAR(32)
finding_description TEXT
finding_recommendation TEXT
finding_references JSON
finding_key CONCAT(scan_type, '-', finding_number)
epss_score FLOAT NULL
cisa_vulnerability_name VARCHAR


Instances

Foreign Keys Type
project_id INT
assessment_id INT
finding_number VARCHAR(251)
scan_type VARCHAR(24)
finding_type VARCHAR(24)
finding_justification_key VARCHAR(256)
finding_discovered DATETIME
finding_output TEXT
finding_port VARCHAR(12)
finding_path VARCHAR(4096)
finding_result VARCHAR(16)
host_id INT
scan_date DATETIME
instance_key CONCAT(scan_type, '-', finding_number, '-', host_id, '-', finding_justification_key)
finding_key CONCAT(scan_type, '-', finding_number)
justification_assignee VARCHAR(512)
justification_text TEXT
justification_status_name VARCHAR(64)
justification_verification_date DATETIME
justification_datetime DATETIME
due_date DATE
justification_status_mitigating TINYINT(1)
finding_score INT
nucleus_threat_rating VARCHAR
nucleus_exploited ENUM('Yes', 'No', '')
nucleus_exploited_by_malware ENUM('Yes', 'No', '')
nucleus_exploited_by_ransomware ENUM('Yes', 'No', '')
nucleus_likely_to_be_exploited ENUM('Yes', 'No', '')
nucleus_public_exploit_available ENUM('Yes', 'No', '')
nucleus_private_exploit_available ENUM('Yes', 'No', '')
nucleus_exploit_weaponized ENUM('Yes', 'No', '')
nucleus_impacts_ot ENUM('Yes', 'No', '')
nucleus_fix_available ENUM('Yes', 'No', '')
nucleus_patch_available ENUM('Yes', 'No', '')
nucleus_ease_of_exploitation VARCHAR
nucleus_exploitation_consequence VARCHAR
nucleus_remote_exploitation ENUM('Yes', 'No', '')
nucleus_zero_day ENUM('Yes', 'No', '')
nucleus_zero_day_previously ENUM('Yes', 'No', '')
nucleus_mitigation_available ENUM('Yes', 'No', '')
nucleus_widely_exploited ENUM('Yes', 'No', '')
nucleus_media_mentions_alltime INT NULL
nucleus_media_mentions_30day INT NULL
nucleus_media_mentions_90day INT NULL
nucleus_media_mentions_180day INT NULL
threat_group_insights JSON
malware_insights JSON
mitigations_insights JSON
patch_insights JSON
exploits_insights JSON
nucleus_actors JSON
nucleus_malware JSON

NOTE:
The finding_score field (Nucleus Risk Score) is only included in full exports, not delta exports. If a risk score cannot be calculated, the value will be 0

*Justification fields are not included in instances files by default. If you would like this option enabled for your organization, please contact Nucleus support.


Justifications

Foreign Keys Type
host_id INT
finding_number VARCHAR(251)
scan_type VARCHAR(24)
finding_justification_key VARCHAR(256)
instance_key CONCAT(scan_type, '-', finding_number, '-', host_id, '-', finding_justification_key)
justification_assignee VARCHAR(512)
justification_text TEXT
justification_status_id INT
justification_verification_date DATETIME
has_file TINYINT(1)
security_comment_text VARCHAR(8224)
justification_datetime DATETIME
due_date DATE


Justification Status

Foreign Keys Type
justification_status_id INT
justification_status_name VARCHAR(64)
justification_status_mitigating BOOL


Teams

Primary Key Type
team_id INT
Foreign Keys Type
host_id INT
finding_number VARCHAR(251)
scan_type VARCHAR(24)
finding_justification_key VARCHAR(256)
instance_key CONCAT(scan_type, '-', finding_number, '-', host_id, '-', finding_justification_key)


External Issues

Foreign Keys Type
host_id INT
finding_number VARCHAR(251)
scan_type VARCHAR(24)
finding_justification_key VARCHAR(256)
instance_key CONCAT(scan_type, '-', finding_number, '-', host_id, '-', finding_justification_key)

Examples

Below are example files for each object type:

Assets

{
   "page":1,
   "pageTotal":1,
   "assets":[
      {
         "asset_id":"1234567",
         "asset_name":"abcdefg",
         "asset_name_secondary":[
            "ec2-233-252-0-211.us-east-2.compute.amazonaws.com",
            "ip-198-51-100-120.us-east-2.compute.internal"
         ],
         "ip_address":"233.252.0.211",
         "ip_address_secondary":[
            "198.51.100.120"
         ],
         "asset_groups":[
            "\/Hosts\/ec2",
            "\/Hosts"
         ],
         "asset_type":"Host",
         "scan_date":"2022-06-14 21:32:59",
         "asset_info":{
            "aws.account-id":"112233445566",
            "aws.region":"us-east-2",
            "aws.service":"ec2",
            "aws.partition":"aws",
            "aws.arn":"arn:aws:ec2:us-east-2:111111111:instance\/i-111111111111",
            "aws.ec2.instance-id":"i-111111111111",
            "aws.ec2.image-id":"ami-123abc456def",
            "aws.ec2.instance-state.code":"16",
            "aws.ec2.instance-state.name":"running",
            "aws.ec2.private-dns-name":"ip-198-51-100-120.us-east-2.compute.internal",
            "aws.ec2.dns-name":"ec2-233-252-0-211.us-east-2.compute.amazonaws.com",
            "aws.ec2.key-name":"key",
            "aws.ec2.ami-launch-index":"0",
            "aws.ec2.instance-type":"t2.micro",
            "aws.ec2.launch-time":"2021-02-10T23:55:55.000Z",
            "aws.ec2.placement.availability-zone":"us-east-2a",
            "aws.ec2.placement.tenancy":"default",
            "aws.ec2.monitoring.state":"disabled",
            "aws.ec2.subnet-id":"subnet-123456abc",
            "aws.ec2.vpc-id":"vpc-12345678",
            "aws.ec2.private-ip-address":"198.51.100.120",
            "aws.ec2.ip-address":"233.252.0.211",
            "aws.ec2.source-dest-check":"true",
            "aws.ec2.group-set-names":[
               "launch-wizard-1"
            ],
            "aws.ec2.group-set-ids":[
               "sg-987654321"
            ],
            "aws.ec2.architecture":"x86_64",
            "aws.ec2.root-device-type":"ebs",
            "aws.ec2.root-device-name":"name",
            "aws.ec2.virtualization-type":"hvm",
            "aws.tags.keyname":"value",
            "aws.tags.owner":"Owner",
            "aws.tags.name":"Name",
            "aws.ec2.hypervisor":"xen",
            "aws.ec2.iam-instance-profile.arn":"arn:aws:iam::123456789:instance-profile\/ec2role",
            "aws.ec2.iam-instance-profile.id":"ABCDEFGHIJKLMNOP",
            "aws.ec2.ebs-optimized":"false",
            "aws.ec2.ena-support":"true"
         },
         "scan_date_timestamp":1655242379,
         "operating_system_name":"",
         "mac_address":"10:00:00:00:00:10",
         "finding_count_critical":"23",
         "finding_count_high":"321",
         "finding_count_medium":"285",
         "finding_count_low":"65",
         "finding_count_informational":"4",
         "finding_count_pass":"0",
         "finding_count_fail":"0",
         "finding_vulnerability_score":"900",
         "asset_public":"1",
         "asset_criticality":"Critical",
         "asset_data_sensitivity_score":"10",
         "asset_complianced_score":"10",
         "asset_criticality_score":"10",
         "asset_inactive_date":"",
         "image_config":"",
         "image_config_digest":"",
         "image_manifest":"",
         "image_manifest_digest":"",
         "image_secondary_registries":[
            
         ],
         "image_tags":[
            
         ],
         "image_registry":"",
         "image_repo":"",
         "image_distro":"",
         "operating_system_features":"",
         "operating_system_version":"",
         "image_platform_os":"",
         "image_platform_os_features":[
            
         ],
         "image_platform_os_version":"",
         "image_platform_arch":"",
         "image_platform_arch_features":[
            
         ],
         "image_platform_arch_variant":"",
         "active":true,         
         "business_owners"" [
            "John Doe"
         ],
         "owner_team": {
            "team_id": "4000040",
            "team_name": {
               team_id": "400040",
               "team_name": "Incident Response"
            }
         },
         "support_team": {
            "team_id": "4000032",
            "team_name": {
               team_id": "400032",
               "team_name": "IT Support",
            }
         },
         "branch":"",
         "repo_url":""
      },
      ...
   ]
}


Asset Findings

{
   "page":1,
   "pageTotal":1,
   "assets":[
      {
         "host_id":"123456789",
         "ip_address":"192.0.2.134",
         "host_name":"192.0.2.134",
         "host_type":"Host",
         "host_criticality":"Critical",
         "operating_system_name":""
      },
      ...
   ]
}


Asset Groups

{
   "page":1,
   "pageTotal":1,
   "assetGroups":[
      {
         "host_id":"123456789",
         "asset_group_name":"Example Asset Group",
         "asset_group_source":"manual"
      },
      ...
   ]
}


Findings and Compliance Findings

{
   "page":1,
   "pageTotal":1,
   "findings":[
      {
        "finding_number":"12345",
        "scan_type":"QUALYS",
        "finding_type":"Vuln",
        "finding_severity":"Low",
        "finding_severity_adjusted":null,
        "finding_exploitable":"0",
        "finding_name":"Web Server HTTP Trace\/Track Method Support Cross-Site Tracing Vulnerability",
        "finding_cve":"CVE-2004-2320,CVE-2007-3008",
        "finding_iava":"",
        "finding_type":"Vuln",
        "finding_description":"A Web server was detected that supports the HTTP TRACE method. A vulnerability related to this method was discovered. A malicious, active component in a Web page can send Trace requests to a Web server that supports this Trace method. Although unlikely and difficult to achieve, it's possible, in the presence of other browser vulnerabilities, for the active HTML content to make external requests to arbitrary Web servers beyond the hosting Web server.",
        "finding_recommendation":"Solutions for some of the common Web servers are supplied below. For other Web servers, please check your vendor's documentation...",
        "finding_references":{
           "ID":"86473",
           "CVE":"CVE-2004-2320,CVE-2007-3008",
           "Category":"Web server",
           "PCI Flagged":"1",
           "Bugtraq IDs":"<a href=\"http:\/\/www.securityfocus.com\/bid\/24456\" target=\"_blank\">24456<\/a>,<a href=\"http:\/\/www.securityfocus.com\/bid\/9506\" target=\"_blank\">9506<\/a>"
        },
        "finding_key":"QUALYS-12345",
        "finding_description_adjusted":null,
        "finding_recommendation_adjusted":null,
        "epss_score":null,
        "cisa_vulnerability_name":"",
        "nucleus_threat_rating":"",
        "nucleus_exploited":"",
        "nucleus_exploited_by_malware":"",
        "nucleus_exploited_by_ransomware":"",
        "nucleus_likely_to_be_exploited":"",
        "nucleus_public_exploit_available":"",
        "nucleus_private_exploit_available":"",
        "nucleus_exploit_weaponized":"",
        "nucleus_impacts_ot":"",
        "nucleus_fix_available":"",
        "nucleus_patch_available":"",
        "nucleus_ease_of_exploitation":"",
        "nucleus_exploitation_consequence":"",
        "nucleus_remote_exploitation":"",
        "nucleus_zero_day":"",
        "nucleus_zero_day_previously":"",
        "nucleus_mitigation_available":"",
        "nucleus_widely_exploited":"",
        "nucleus_media_mentions_alltime":null,
        "nucleus_media_mentions_30day":null,
        "nucleus_media_mentions_90day":null,
        "nucleus_media_mentions_180day":null,
        "threat_group_insights":[],
        "malware_insights":[],
        "mitigations_insights":[],
        "patch_insights":[],
        "exploits_insights":[],
        "nucleus_actors":[],
        "nucleus_malware":[]
      },
      ...
   ]
}


Instances and Compliance Instances

{
   "page":1,
   "pageTotal":6,
   "instances":[
      {
         "project_id":"123456789",
         "assessment_id":"111111111",
         "finding_number":"112345-abcdefg-67890",
         "scan_type":"NESSUS",
         "finding_type":"Vuln"
         "finding_justification_key":"abcdefghijklmnopqrstuvwxyz",
         "finding_discovered":"2022-05-31 21:00:39",
         "finding_output":"The remote host (203.0.113.2) is considered as dead - not scanning\nThe remote host (203.0.113.2) did not respond to the following ping methods :\n- TCP ping\n- UDP ping\n- ICMP ping\n",
         "finding_port":"0\/tcp",
         "finding_path":"",
         "finding_result":"FAILED",
         "finding_severity": "Medium",
         "finding_severity_adjusted": null,
         "host_id":"222222222",
         "scan_date":"2022-05-31 21:00:39",
         "finding_score": 700,
         "instance_key":"NESSUS-112345-abcdefg-67890-abcdefghijklmnopqrstuvwxyz",
         "finding_key":"NESSUS-112345-abcdefg-67890"
      },
      ...
   ]
}


Justifications

{
   "page":1,
   "pageTotal":1,
   "justifications":[
      {
         "host_id":"111111111",
         "finding_number":"abcdefg",
         "scan_type":"ACUNETIX",
         "finding_justification_key":"abcdefg-static\/111213\/js\/-",
         "instance_key":"ACUNETIX-abcdefg-111111111-abcdefg-static\/111213\/js\/-",
         "justification_assignee":null,
         "justification_text":"",
         "justification_status_id":"10",
         "justification_verification_date":null,
         "has_file":"0",
         "security_comment_text":"Not yet provided.",
         "justification_datetime":"2022-06-09 20:29:04",
         "due_date":null,
         "justification_status_name":"Potential",
         "justification_status_mitigating":0
      },
      ...
   ]
}


Teams

{
   "page":1,
   "pageTotal":1,
   "teams":[
      {
         "team_id":"1111111",
         "host_id":"2222222",
         "finding_number":"FINDING-5",
         "scan_type":"SUPERGREP",
         "finding_justification_key":"FINDING-5-SUPERGREP-1111111-path\/to\/vulnerable\/file5.txt-",
         "instance_key":"SUPERGREP-FINDING-5-1111111-FINDING-5-SUPERGREP-1111111-path\/to\/vulnerable\/file5.txt-",
         "team_name":"RemediationTeam"
      },
      ...
   ]
}


External Issues

{
    "page": 1,
    "pageTotal": 1,
    "externalIssues": [
        {
            "host_id": "31000018",
            "finding_number": "FINDING-1",
            "scan_type": "CUSTOMSCANNER",
            "finding_justification_key": "NETWORK-1-CUSTOMSCANNER-31000018--0/tcp",
            "instance_key": "CUSTOMSCANNER-NETWORK-1-31000018-NETWORK-1-CUSTOMSCANNER-31000018--0/tcp",
            "issue_type": "SERVICENOWAPP",
            "issue_key": "INC0047849",
            "issue_title": "Nucleus - [High] - FINDING-1",
            "issue_url": "https://exampleco.service-now.com/task.do?sys_id=32474ab787db4e1007a6a60d3fbb35a8",
            "issue_parent": "INC0047844",
            "issue_priority": "",
            "issue_status": "New",
            "issue_assignee": "",
            "issue_created": "2024-07-16 19:20:03",
            "issue_updated": "0000-00-00 00:00:00",
            "issue_synced": "0000-00-00 00:00:00",
            "issue_closed": "0",
            "finding_mitigated": "0"
        },
        ...
    ]
}