Analyze search usage statistics

The CMC Search Usage Statistics dashboard provides information about your searches on a per-instance basis. You can split the data by user, host, and source type to better review the results. Analyzing this data helps you answer questions like these:

  • Which users run a large number of searches, or long-running searches, or both?
  • Which hosts handle the greatest number of searches, and what is the median runtime?
  • What are the most heavily used search types in the deployment?

You can then examine searches in more detail to determine if they can be optimized.

See also:

Review the Search Usage Statistics dashboard

This dashboard includes panels of summary, graphical, and tabular data about your search usage statistics. The filter lets you specify a time range and instance values and opt to include or exclude ad hoc searches.

This dashboard contains one panel with a variable in the title: Search Activity by <variable>.

To investigate your panels, go to Cloud Monitoring Console > Search > Search Usage Statistics. Use the following table to understand the dashboard interface.

Panel or Filter Description
Instances Choose to view all instances or specify a particular instance.
Time Range Set the time range for the data display.
Only Ad Hoc Searches Choose Yes to limit the displayed data to only ad hoc searches. Choose No to view results for both ad hoc and scheduled searches.
Overview Shows the total number of searches finished, successfully and unsuccessfully completed, and the total search runtime of all searches.
Searches Provides a graphical representation of the information in the Overview panel. Specify a Split by option to view the related Searches graph by user, host, or search type. Search types displayed grouped in the following general and granular categories:
  • acceleration and summarization: See Overview of summary-based search acceleration.
  • ad hoc: See Ad hoc search.
  • dashboard: Searches that populate a dashboard. Search types in this granular category are ad hoc, acceleration, other, or scheduled.
  • other: Searches that don't fall into the other categories. These searches are generally performed by a Splunk Cloud Platform administrator.
  • realtime: This is a granular category for scheduled searches that are continually running in the background.
  • scheduled: See Scheduled search. This category excludes realtime searches.
  • subsearch: See Subsearch. Search types in this granular category are ad hoc and other.
Search Activity by <variable> The <variable> is determined by Split by choice. The table lists the following information:
  • Search type and count
  • Median and cumulative runtimes
  • Timestamp of the last search
Cache Activity Shows your deployment's cache (local storage) activity, aggregated by bucket count or GB.

The chart shows the following bucket rates tracked over time:

  • bucket_download: Buckets that have automatically transferred from AWS S3 storage to the cache. Spikes occur when searches need to localize data into the cache and are a part of your deployment's normal execution. Spikes over a longer duration may indicate indexer overload, which affects optimal search and ingest performance.
  • bucket_eviction: Buckets that have been automatically cleared from the cache.
  • bucket_upload: Spikes occur when more data is being transferred to AWS S3, though this does not impact system performance. This metric tracks the following:
    • Buckets that have automatically been added to AWS S3 storage. These are generally hot buckets that are converting to warm buckets.
    • Buckets that already exist in AWS S3 storage but their internal details have changed; this results in an automatic resubmit and re-upload. For example, when a Splunk Cloud Platform administrator uses the delete command, this action affects the bucket's internal details and results in a re-upload of the modified bucket.
Search Details The table lists the following information:
  • Report name/search string: Shows the report name for saved searches or the search string for ad hoc searches.
  • Search runtime: Enter a value in the Search runtime >= (seconds) field to restrict the results to searches that meet or exceed this runtime.
  • Search start: The initial start time of the search job.
  • Earliest and latest time: The earliest and latest times of the search's time range.
  • Search type
  • User and host names
  • SID: Search identifier

Interpret search usage statistics results

To review searches by user, follow these steps:

  1. Change the time frame to widen the scope. For example, set it to the week prior to the current date.
  2. Split the search by users so that you can see if there are a few users who are typically running longer searches.
  3. Sort by Cumulative Runtime to see which users have the most cumulative search time.
  4. Sort by Median Runtime to see which users are running the median longest searches.
  5. If the user running the most or longest searches is the system user, you may want to review your applications to make sure that you have optimized them, and that they are providing the expected value. You may discover that some applications are not needed or are not used.

To review long-running searches, follow these steps:

  1. Expand the time range to at least 24 hours. Searches are automatically sorted by long-running searches.
  2. Set Only Ad Hoc Searches to No if it isn't already. This setting ensures that you see only scheduled searches, which are more likely to be long-running searches than ad hoc searches.
  3. Scroll to the Search Details panel where the searches are sorted by search runtime.
  4. Select the search name to view more details, and scroll to the bottom of the screen. Two events are displayed. In the second event, you can see the search query.

Note: If you discover a long-running query that runs frequently, you may want to expand the time range to a week or longer to see how commonly this search is run. If it is running frequently, consider optimizing the search.

Be sure to monitor the Cache Activity panel for any spikes in bucket download size and count that have a long duration. Constant and excessive downloading of data into the cache churns the local file system on your indexers, and sustained bucket download spikes may cause performance impacts in searching and ingestion. To remediate these issues, you may need to modify search patterns, alter retention, or upgrade your workload-based or ingestion-based subscription entitlements so more of your searchable data is able to reside in the cache.