Troubleshoot Splunk UBA event processing

This section contains information to help you analyze activity and diagnose problems with event processing in your Splunk UBA deployment.

Identify all sourcetypes in your data

Run the following search to identify the sourcetypes in the data being ingested by the Splunk platform. Identifying sourcetypes is useful when you want to verify that you have the necessary data for Splunk UBA to function or to unlock desired use cases. 

| metasearch index=*
| stats count by index, sourcetype
| table sourcetype, index, count

Identify all available indexes, sourcetypes, and EPS

Identify all available indexes, sourcetypes, and average events per second (EPS). The EPS value is important to make sure you are sizing your Splunk UBA cluster correctly. See Scaling your Splunk UBA deployment in the Plan and Scale your Splunk UBA Deployment manual. 

| tstats count as eps where index=* earliest=-30d@d groupby index, sourcetype _time span=1s
| stats count as NumSeconds max(eps) perc99(eps) perc90(eps) avg(eps) as avg_eps by index, sourcetype
| addinfo
| eval PercentageOfTimeWithData = NumSeconds / (info_max_time -info_min_time)
| fields - NumSeconds info*
| eval EffectiveAverage = avg_eps * PercentageOfTimeWithData
| fieldformat PercentageOfTimeWithData = round(PercentageOfTimeWithData*100,2) . "%"

Events from a data source do not appear in Splunk UBA Web

Events from a data source are being processed but do not appear in Splunk UBA Web.

Cause Solution
There might be a delay of up to 5 minutes before any information about processed events appears in Splunk UBA Web. To view event processing details, add ?system into the URL.
  1. In Splunk UBA Web, select Manage > Data Sources.
  2. Select a data source. The URL in Splunk UBA Web might be something like: https://ubaserver1/#Y2FzcGlk==
  3. Add ?system into the URL. For example: https://ubaserver1/?system#Y2FzcGlk==
  4. Reload the page with the updated URL.

Additional information is displayed for that data source, such as EPS trend, events categorized by view or model, and connector statistics.

Active Directory events are not being parsed

You notice that some Active Directory (AD) events are not being parsed.

Cause Solution
Invalid values are present in the EntityValidations.json file. Invalid values cause the AD token resultCode to not be populated. This value is important for categorizing AD events. Open the /etc/caspida/local/conf/etl/configuration/EntityValidations.json file and see if 0x0 is present in the generic section. If so, remove it. If you do not have any customized values for invalidValues, remove the entire section or keep it empty, as shown below: 
"invalidValues" : { }

If you edit the file, use proper JSON syntax with your edits.

Error messages when viewing contributing events

When viewing the contributing events for an anomaly, you receive an error message like the following: 

Cannot find search head definition for endpoint https://<host>:<port> in splunk_search_head.json

To resolve this, check the following:

  1. Make sure DNS is configured correctly on your system. All nodes in your Splunk UBA deployment must point to the same DNS server. If DNS is not configured correctly, you may see this error when you are trying to view contributing events over a VPN connection.
  2. Verify that the following host names are an exact match. Use a fully qualified domain name (FQDN) in both of the following places: