Extract asset and identity data in Splunk Enterprise Security
Collect, extract, and add asset and identity data in Splunk Enterprise Security to reduce manual updates and improve data integrity. In a Splunk Cloud Platform deployment, work with Splunk Professional Services to design and implement an asset and identity collection solution.
Perform these procedures to extract asset and identity data in Splunk Enterprise Security:
Collect asset and identity data
Follow these steps to collect asset and identity data:
- Determine where the asset and identity data in your environment is stored.
- Collect and update your asset and identity data automatically to reduce the overhead and maintenance that manual updating requires and improve data integrity.
- Use Splunk DB Connect or another Splunk platform add-on to connect to an external database or repository.
- Use scripted inputs to import and format the lists.
- Use events indexed in the Splunk platform with a search to collect, sort, and export the data to a list.
Following are some suggested collection methods for assets and identities:
| Technology | Asset or identity data | Collection methods |
|---|---|---|
| Active Directory | Both | AD LDAP and a custom search. |
| Both | Splunk Supporting Add-on for Active Directory | |
| Both | SecKit Windows Assets Technology Add-on for Splunk Enterprise Security * | |
| LDAP | Both | AD LDAP and a custom search. |
| CMDB | Asset | Splunk DB Connect for integrating with 3rd Party structured data sources, and a custom search. |
| ServiceNow | Both | Splunk Add-on for ServiceNow |
| Bit9 | Asset | Splunk Add-on for Bit9 and a custom search. |
| Cisco ISE | Both | Splunk Add-on for Cisco ISE and a custom search. |
| Microsoft SCOM | Asset | Splunk Add-on for Microsoft SCOM and a custom search. |
| Sophos | Asset | Splunk Add-on for Sophos and a custom search. |
| Symantec Endpoint Protection | Asset | Splunk Add-on for Symantec Endpoint Protection and a custom search. |
| Amazon Web Services (AWS) | Both | Create Cloud Asset Lookup and Create Cloud Identity Lookup |
| Azure | Both | Create Cloud Asset Lookup and Create Cloud Identity Lookup |
| Google Cloud Platform | Both | Create Cloud Asset Lookup and Create Cloud Identity Lookup |
| Configuration Management Database (CMDB) | Asset | SecKit SA Common tools for populating assets and identities in Enterprise Security and PCI apps * |
For more information on custom search commands, see Create custom search commands for apps in Splunk Cloud Platform or Splunk Enterprise.
Format an asset or identity list as a lookup
Format your collected asset or identity data into a lookup file so that it can be processed by Splunk Enterprise Security.
Prerequisite Collect asset and identity data for Splunk Enterprise Security
Follow these steps to format your asset and identity data as a lookup:
- Create a plain text, CSV-formatted file with Unix line endings and a
.csvfile extension. - Use the correct headers for the CSV file. See Asset and identity lookup configurations for the headers expected by Splunk Enterprise Security.
- Populate the rows of the CSV with the asset or identity fields. The maximum number of characters per value in a field is 975. For a multivalue field, each value in the list can be 975 characters. See Asset and identity lookup configurations for reference.
For an example asset list, review the Demonstration Assets lookup. Locate the list in Splunk Web:
- In the Splunk Enterprise Security app, select Security content then Content management.
- Locate the list in the file system. The
demo_assets.csvfile is located in theSA-IdentityManagement/lookups/directory.
If you use a custom search to generate a lookup, make sure that the lookup produced by the search results contains fields that match the headers.
Asset and identity lookup configurations
Enterprise Security manages specific props.conf settings as part of the asset and identity framework. In order for these files to be configured properly, all configurations need to be populated in the SPLUNK_HOME/etc/apps/SA-IdentityManagement/local/props.conf file. If there are existing identity correlation lookup definitions in the SPLUNK_HOME/etc/apps/SA-IdentityManagement/default/props.conf file, remove them so they can be managed by the asset and identity framework.
Asset lookup header
ip,mac,nt_host,dns,owner,priority,lat,long,city,country,bunit,category,pci_domain,is_expected,should_timesync,should_update,requires_av,cim_entity_zoneAsset lookup fields Populate the following fields in an asset lookup.
Note: To add multi-homed hosts or devices to the asset list, add each IP address to the
ip field for the host, pipe-delimited. Multi-homed support is limited, and having multiple hosts with the same IP address on different network segments can cause conflicts in the merge process.| Field | Data type | Description | Example values |
|---|---|---|---|
| ip | pipe-delimited numbers | A pipe-delimited list of single IP address or IP ranges. An asset is required to have an entry in at least one of the key fields such as: ip, mac, nt_host, or dns fields. All of the key fields are multi-value fields. | 2.0.0.0/8|1.2.3.4À.168.15.9-192.168.15.27|5.6.7.8|10.11.12.13 |
| mac | pipe-delimited strings | A pipe-delimited list of MAC address. An asset is required to have an entry in at least one of the key fields such as: ip, mac, nt_host, or dns fields. All of the key fields are multi-value fields. | 00:25:bc:42:f4:60|00:50:ef:84:f1:21|00:50:ef:84:f1:20 |
| nt_host | pipe-delimited strings | A pipe-delimited list of Windows machine names. An asset is required to have an entry in at least one of the key fields such as: ip, mac, nt_host, or dns fields. All of the key fields are multi-value fields. | ACME-0005|SSPROCKETS-0102|COSWCOGS-013 |
| dns | pipe-delimited strings | A pipe-delimited list of DNS names. An asset is required to have an entry in at least one of the key fields such as: ip, mac, nt_host, or dns fields. All of the key fields are multi-value fields. | acme-0005.corp1.acmetech.org|SSPROCKETS-0102.spsp.com|COSWCOGS-013.cwcogs.com |
| owner | string | The user or department associated with the device | f.prefect@acmetech.org, DevOps, Bill |
| priority | string | Recommended. The priority assigned to the device for calculating the Urgency field for findings on the analyst queue. An "unknown" priority reduces the assigned Urgency by default. | unknown, low, medium, high or critical. |
| lat | string | The latitude of the asset in decimal degrees, using +/- to indicate direction. | 37.780080 |
| long | string | The longitude of the asset in decimal degrees, using +/- to indicate direction. | -122.420170 |
| city | string | The city in which the asset is located | Chicago |
| country | string | The country in which the asset is located | USA |
| bunit | string | Recommended. The business unit of the asset. Used for filtering by dashboards in Splunk Enterprise Security. | EMEA, NorCal |
| category | pipe-delimited strings | Recommended. A pipe-delimited list of logical classifications for assets. Used for asset and identity correlation and categorization. See Asset/Identity Categories. | server|web_farm|cloud |
| pci_domain | pipe-delimited strings | A pipe-delimited list of PCI domains. See Configure assets in the Splunk App for PCI Compliance Installation and Configuration Manual. | cardholder, trust|dmz, untrust If left blank, defaults to untrust. |
| is_expected | boolean | Indicates whether events from this asset should always be expected. If set to true, the Expected Host Not Reporting detection performs an adaptive response action when this asset stops reporting events. | "true", or blank to indicate "false" |
| should_timesync | boolean | Indicates whether this asset must be monitored for time-sync events. It set to true, the Should Timesync Host Not Syncing detection performs an adaptive response action if this asset does not report any time-sync events from the past 24 hours. | "true", or blank to indicate "false" |
| should_update | boolean | Indicates whether this asset must be monitored for system update events. | "true", or blank to indicate "false" |
| requires_av | boolean | Indicates whether this asset must have anti-virus software installed. | "true", or blank to indicate "false" |
| cim_entity_zone | string | Required when entity zones are turned on. Lowercase word to use as a default zone name. For use in situations when you have multiple private IP spaces. This word auto-populates in the cim_entity_zone fields if you do not specify your own values when formatting an asset or identity list as a lookup. | my_zone |
Identity lookup header
identity,prefix,nick,first,last,suffix,email,phone,managedBy,priority,bunit,category,watchlist,startDate,endDate,work_city,work_country,work_lat,work_long,cim_entity_zoneIdentity lookup fields
| Field | Data type | Description | Example |
|---|---|---|---|
| identity | pipe-delimited strings | Required. A pipe-delimited list of username strings representing the identity. After the merge process completes, this field includes generated values based on the identity lookup configuration settings. |
a.vanhelsing|abraham.vanhelsing|a.vanhelsing@acmetech.org |
| prefix | string | Prefix of the identity. | Ms., Mr. |
| nick | string | Nickname of an identity. | Van Helsing |
| first | string | First name of an identity. | Abraham |
| last | string | Last name of an identity. | Van Helsing |
| suffix | string | Suffix of the identity. | M.D., Ph.D |
| string | Email address of an identity. | a.vanhelsing@acmetech.org | |
| phone | pipe-delimited strings | A pipe delimited field for telephone number of an identity. | 123-456-7890 |
| managedBy | string | A username representing the manager of an identity. | phb@acmetech.org |
| priority | string | Recommended. The priority assigned to the identity for calculating the Urgency field for findings on the analyst queue. An "unknown" priority reduces the assigned Urgency by default. | unknown, low, medium, high or critical. |
| bunit | string | Recommended. A group or department classification for identities. Used for filtering by dashboards in Splunk Enterprise Security. | Field Reps, ITS, Products, HR |
| category | pipe-delimited strings | Recommended. A pipe-delimited list of logical classifications for identities. Used for asset and identity correlation and categorization. See Asset/Identity Categories. | Privileged|Officer|CISO |
| watchlist | boolean | Marks the identity for activity monitoring. | Accepted values: "true" or empty. See User Activity Monitoring in the Use Splunk Enterprise Security manual. |
| startDate | string | The start or hire date of an identity. | Formats: %m/%d/%Y %H:%M, %m/%d/%y %H:%M, %s |
| endDate | string | The end or termination date of an identity. | Formats: %m/%d/%Y %H:%M, %m/%d/%y %H:%M, %s |
| work_city | string | The primary work site City for an identity. | |
| work_country | string | The primary work site Country for an identity. | |
| work_lat | string | The latitude of primary work site City in decimal degrees, using +/- to indicate direction. | 37.780080 |
| work_long | string | The longitude of primary work site City in decimal degrees using +/- to indicate direction. | -122.420170 |
| cim_entity_zone | string | Required when entity zones are turned on. Lowercase word to use as a default zone name. For use in situations when you have multiple private IP spaces. This word auto-populates in the cim_entity_zone fields if you do not specify your own values when formatting an asset or identity list as a lookup. | my_zone |
Configure a new asset or identity list
Configure a new asset or identity lookup in Splunk Enterprise Security. This multistep process adds the lookup in Splunk Enterprise Security and defines the lookup for the merge process.
Prerequisites Collect and extract asset and identity data in Splunk Enterprise Security Format the asset or identity list as a lookup in Splunk Enterprise Security. Assets and identities framework supports only exact-matching of IPv6 addresses.
Steps
Add the new lookup table file
These lookup table files are consumed by the asset and identity framework and merged together. The product of the merge is called an "expanded lookup."
- From the Splunk menu bar, select Settings > Lookups > Lookup table files.
- Select New.
- Select a Destination App of SA-IdentityManagement.
- Select the lookup file to upload.
- Type the Destination filename that the lookup table file should have on the search head. The name should include the filename extension. For example,
network_assets_from_CMDB.csv - Select Save to save the lookup table file and return to the list of lookup table files.
Note: In a distributed environment, these lookup table files are not replicated from the search heads to the indexers. Only the expanded lookup is replicated to the indexers. However, these lookup files are still replicated between search heads. If an asset or identity lookup table file grows in excess of 1GB+, it should be broken down into smaller files (for example, by location or by type or by easily identifiable category). When making changes to lookup files, only the updated files are replicated across search heads, reducing bundle sizes.
Set permissions on the lookup table file to share it with Splunk Enterprise Security
Add a new lookup definition
- From the Splunk menu bar, select Settings > Lookups > Lookup definitions.
- Select New.
- Select a Destination App of SA-IdentityManagement.
- Type a name for the lookup source. This name must match the name defined later in the input stanza definition on the Identity Management dashboard. For example,
network_assets_from_CMDB. - Select a Type of File based.
- Select the lookup table file created. For example, select
network_assets_from_CMDB.csv. - Select Save.