Audit and health dashboards

Use the Audit and health dashboards to validate the security and integrity of the data in Splunk Enterprise Security. Ensure that forwarders are functioning, that data has not been tampered with and is secured in transmission, and that analysts are reviewing the findings generated by detections.

The Audit and health dashboards provide insight into background processes and tasks performed by Splunk Enterprise Security. Some audit dashboards allow you to review actions taken by users in Splunk Enterprise Security, while others provide insight into your deployment and the status of your data models and content use.

Audit the analyst queue

The Analyst queue dashboard provides an overview of activity on findings. The panels display how many findings are being reviewed and by which user, along with a list of the most recently reviewed events. The metrics on this dashboard allow security managers to review the activities of analysts.

Panel Description
Review activity by reviewer Displays the numbers of findings and investigations reviewed by each user. This panel is useful for determining which user is performing the security incident reviews and if the total number of reviews is changing over time. The drilldown opens a search with all activity by the selected reviewer.
Top reviewers Displays the top users that review security incidents in the analyst queue. The panel includes details for each user, including the date they first reviewed a finding or investigation, the date they last performed a review, and the total number of findings and investigations reviewed. The drilldown opens a search with all activity by the selected reviewer.
Findings by status: Last 48 hours Displays the status, count, and urgency for all findings in the last 48 hours. This panel is useful for determining if the analyst queue users are keeping up with security incidents, or whether a backlog of unreviewed incidents is forming. The drilldown opens the analyst queue on the Mission Control page and searches on the selected urgency and status over the lat 48 hours.
Findings by owner: Last 48 hours Displays the owner, count, and urgency for all findings in the last 48 hours. This panel is useful for determining how many events are assigned to a user and the urgency of the events. The drilldown opens the analyst queue on the Mission Control page and searches on the selected urgency over the lat 48 hours.
Mean time to triage: Last 14 days Displays the average time it took for a finding to be triaged after it was created over the last 14 days, split by the name of the finding. This panel is useful for determining how quickly analysts are triaging findings, or whether certain types of events take longer to triage than others. The drilldown opens the analyst queue on the Mission Control page and searches on the matching finding names over the last 14 days.
Mean time to closure: Last 60 days Displays the average time it took for a finding to be closed after it was created over the last 60 days, split by the name of the finding. This panel is useful for determining how long it takes to close certain types of findings and investigations. The drilldown opens the analyst queue on the Mission Control page and searches on the matching finding names that have a status of closed from the last 60 days.
Recent review activity Displays the 10 most recent changes on the analyst queue, such as triage actions. The drilldown opens a search with the selected rule ID.

To audit data from the analyst queue in Splunk Enterprise Security prior to version 3.2, you must perform an ad hoc search like the following example:

CODE

Data sources

The reports in the Analyst queue audit dashboard reference fields in the notable index and the incident review objects in a KVStore collection.

Audit the suppressions

The Suppression dashboard provides an overview of finding suppression activity. This dashboard shows how many events are being suppressed, and by whom, so that finding suppression can be audited and reported on.

The metrics on this dashboard allow security managers to review the activities of analysts, which is useful for tuning detections. You can identify detection rules that are generating more events than your analysts are capable of looking at, and tune them accordingly.

Panel Description
Suppressed events over time: Last 24 hours Displays findings suppressed in the last 24 hours.
Suppression history over time: Last 30 days Displays the history of suppressed findings.
Suppression management activity Displays suppression management activity for the time period.
Expired suppressions Displays expired suppressions.

Data sources

The reports in the Suppression audit dashboard reference events in the Notable index.

Audit the modular actions

The Modular actions dashboard provides an overview of the response actions initiated by adaptive response actions, including finding creation and risk scoring.

Panel Description
Action invocations over time by name Displays a time chart of the adaptive response actions triggered by name.
Top actions by name Displays the top adaptive response actions by name.
Top actions by search Displays the top adaptive response actions by search.
Recent response actions Displays the most recent adaptive response actions.

Data sources

The reports in the Adaptive response action center dashboard reference fields in the Audit data model. For a list of data model objects and constraints, see Splunk Audit Logs in the Common Information Model Add-on manual.

Audit the filters in your deployment

The Per-panel filter dashboard provides information about the filters currently in use in your deployment.

The following table describes the panels for this dashboard.

Panel Description
Per-panel by reviewer Displays the count of updates to per-panel filters by user.
Top users Shows users, sparkline for trends, number of views, and first and last time viewed.
Recent filter activity Activity by time, user, action, and filename.

Audit the threat intelligence

The Threat intelligence dashboard tracks and displays the current status of all threat and generic intelligence sources. As an analyst, you can review this dashboard to determine if threat and generic intelligence sources are current, and troubleshoot issues connecting to threat and generic intelligence sources.

Panel Description
Intelligence downloads Displays the status of all intelligence sources defined on the Intelligence downloads configuration page. Use the filters to sort by status or download location.
Intelligence audit events Displays log events related to intelligence downloads configured on the Intelligence downloads configuration page and modular inputs configured on the Threat intelligence manager configuration page. Use the filters to sort and filter the events displayed.

If an intelligence download fails, a search automatically creates a system message. See Troubleshoot intelligence downloads in Splunk Enterprise Security.

Data sources

The reports in the Threat intelligence audit dashboard reference events in the _internal index and state information from the /services/data/inputs/threatlist REST endpoint.

Audit the usage of the Artificial Intelligence Toolkit (AITK)

The Machine learning dashboard displays information related to usage of the Artificial Intelligence Toolkit (AITK).

Panel Description
Machine learning toolkit errors and failed fit and apply searches: Last 7 days The mlspl.log log file itself doesn't contain a lot of details about specific models and when they ran as part of a search or a rule. As an analyst, you can review this chart to help determine where MLTK errors are happening. It shows all the MLTK errors over the last 7 days. If you click on the chart to drill-down into the details, you can see the audits of failed searches that contain the fit or apply commands, which can help you correlate errors with the actual searches that produced the issues.
Machine learning models The list shows the names of the MLTK models. If you click on a model name to drill-down into the details, it opens a custom search that helps audit your model generating searches and the corresponding rules that apply them. See Audit searches using an MLTK Model.
List of model generating searches The button shows all the MLTK model generating searches and their statuses.

Audit the health of your Splunk Enterprise Security configuration

Use the ES configuration dashboard to compare the latest installed version of Splunk Enterprise Security to prior releases and identify configuration anomalies. The dashboard does not report changes to add-ons (TA.) Select the previous version of Splunk Enterprise Security installed in your environment using the Previous ES Version filter.

Mode Description
Unshipped The unshipped setting compares the latest installed version of Splunk Enterprise Security with the content in the ES installation package. Any item that was not provided as part of the Splunk Enterprise Security installation, such as files or scripts used for customization, is labeled as an Unshipped item. Review unshipped items to evaluate their use, determine if they are still needed, and reconcile if necessary. The unshipped setting ignores the Previous ES version filter.
Removed stanzas The removed stanzas setting compares the latest installed version of Splunk Enterprise Security with the version that you select in the filter. Removed stanzas are configuration stanzas that changed between versions, such as a deprecated threat list or input. Review removed stanzas to evaluate their use, determine if they are still needed, and reconcile if necessary.
Local overrides The local overrides setting compares the installed version of Splunk Enterprise Security with the version that you select in the filter. A setting that conflicts with or overrides the installed version of Splunk Enterprise Security is labeled as a Local override. Review any local override settings to evaluate their use, determine if they are still needed, and reconcile if necessary.

Audit the forwarders

The Forwarder dashboard reports on hosts forwarding data to Splunk Enterprise.

Use the search filters and time range selector to focus on groups of forwarders or an individual forwarder.

Filter by Description Action
Show only expected hosts An expected host is a host defined in ES by the expected host field is_expected in the Asset table. Drop-down, select to filter by
Host Filter by the host field in the Asset table. Text field. Wildcard with an asterisk (*)
Business Unit Filter by the business unit bunit field in the Asset table. Text field. Wildcard with an asterisk (*)
Category Filter by the category field in the Asset table. Drop-down, select to filter by
Panel Description
Event count over time by host Displays the number of events reported over the time period selected in the filter. The events are split by host.
Hosts by last report time Displays a list of hosts, ordered by the last time they reported an event.
Splunkd process utilization Displays the resource utilization of the forwarder's Splunk daemon splunkd.
Splunk service start mode Displays the host names that are forwarding events, but are not configured to have splunkd start on boot.

Data sources

Relevant data sources for the Forwarder Audit dashboard include data from all forwarders in your Splunk environment and the Application_State data model. See the Common Information Model Add-on Manual for more information. The Common Information Model fields bunit and category are derived by automatic identity lookup, and do not need to be mapped directly.

Audit the indexing

The Indexing dashboard is designed to help administrators estimate the volume of event data being indexed by Splunk Enterprise. The dashboard displays use EPD (events per day) as a metric to track the event volume per index, and the rate of change in the total event counts per index over time. The EPD applies only to event counts, and is unrelated to the volume per day metric used for licensing.

Panel Description
Key indicators The key indicators on this dashboard are scoped to "All time," not the "Last 24 hours".
Events per day over time Displays a column chart representing the event counts per day.
Events per day Displays a table representing event counts per day and the average eps.
Events per index (last day) Displays a table of event counts per index for the last day.

Data sources

The reports in the Indexing audit dashboard reference data generated by the Audit - Events Per Day - Lookup Gen saved search and are stored within a KVStore collection.

Audit the searches run on Splunk Enterprise

The Search audit dashboard provides information about the searches run in Splunk Enterprise. This dashboard is useful for identifying long running searches and tracking search activity by user.

Panel Description
Searches over time by type Shows the number of searches run over time by type, such as ad-hoc, scheduled, or real-time. Helps determine whether Splunk's performance is being affected by excessive numbers of searches.
Searches over time by user Shows the number of searches executed by each user. Helps determine when a particular user is executing an excessive number of searches. The splunk-system-user is the name of the account used to execute scheduled searches in Splunk Enterprise.
Top searches by run time Lists the most expensive searches in terms of duration. Helps to identify specific searches that might be adversely affecting Splunk performance.

Data sources

The reports in the Search audit dashboard reference scheduled search auditing events from the audit index.

Audit the views in Splunk Enterprise Security

The View audit dashboard reports on the most active views in Splunk Enterprise Security. The View audit dashboard allows tracking of views that are being accessed on a daily basis and helps to identify any errors triggered when users review dashboard panels.

Panel Description
View activity over time Displays the Splunk Enterprise Security views that have the greatest access counts over time. The drilldown opens a search view of all page activity for the time selected.
Expected view activity Lists the views set up in the Expected view lookup. You want to review these views on a daily basis for your deployment. Select a dashboard to see details in the Expected view scorecard panel below. See Modify asset and identity lookups in Splunk Enterprise Security.
Web service errors Displays errors that occurred while loading the web interface. Helps identify custom views that contain errors or an underlying issue that need to be escalated to Splunk.

Data sources

The reports in the View audit dashboard reference fields in the Splunk Audit data model. For a list of data model objects and constraints, see Splunk Audit Logs in the Common Information Model Add-on Manual.

Audit the managed lookups and collections

The Managed lookups dashboard reports on managed lookups and collections such as services, data, transforms, KV Store lookups, and CSV lookups in Splunk Enterprise Security. The Managed lookups audit dashboard shows the growth of lookups over time and the markers for anomalous growth. You can use this to help determine if any managed lookups are growing too large for your particular environment's performance and need to be pruned.

Field Description
Name Displays the name of the Splunk Enterprise Security lookup. The drill-down takes you to all the contributing events for this particular lookup name from the audit_summary index.
Growth Lists the lookup size over time as measured by a saved search that writes to the audit_summary index, running every 24 hours, displayed as a sparkline.
Count Displays the estimated number of rows in the lookup file.
Size Displays the size of the file in megabytes, sorted by the largest first.