Identify and triage indexing performance problems

Use this topic to begin to troubleshoot indexing performance problems in Splunk software.

Problems

The following are some symptoms of indexing performance issues:

Messages in Splunk Web indicate data stalls on indexers or on instances sending data to indexers.
Forwarders are unable to send data to indexers.
Receiving ports on indexers are being closed. You might have learned of this by looking at the Splunk TCP Input: Instance Monitoring Console dashboard.
Event-processing queues are saturated. You might have learned of this from the Monitoring Console Health Check or from a Monitoring Console platform alert.
Indexing rate is unusually low. You might have learned of this from the Monitoring Console Health Check.
Data is arriving late. See Event indexing delay.

Gather information

You need three pieces of information to begin to diagnose indexing problems: indexing status, indexing rate, and queue fill pattern.

Before you continue, consider reading How indexing works in Managing Indexers and Clusters of Indexers.

Determine indexing status

The indexer processor can be in one of several states: normal, saturated, throttled, or blocked.

View the current state using any of these methods:

Monitoring Console health check
Monitoring Console indexing performance views
"Saturated event-processing queues" platform alert, included with the Monitoring Console
server/introspection/indexer endpoint

See About the Monitoring Console in Monitoring Splunk Enterprise.

Determine and categorize the indexing rate

Distinguish between an indexing rate that is nonexistent (0 MB/s), low (1 MB/s), or high (at least several MB/s).

Use the Monitoring Console to determine indexing rate. This information is available in several locations within the Monitoring Console:

Overview > Topology with the Indexing rate overlay.
Indexing > Indexing Performance: Deployment.
Indexing > Indexing Performance: Instance.

Determine queue fill pattern

Indexing queue fill profiles can be grouped into three basic shapes. For this diagnosis, differentiate between flat and low, spiky, and saturated.

Indexing is not necessarily "blocked" even if the event-processing queues are saturated.

Use the Monitoring Console to determine the queue fill pattern. See Monitoring Console > Indexing > Indexing Performance: Deployment and also Monitoring Console > Indexing > Indexing Performance: Instance.

Here is an example of a flat and low queue fill pattern.

Here is an example of a spiky but healthy queue fill pattern. Although some of the queues saturate briefly, they recover quickly and completely.

Here is an example of a saturated queue fill pattern.

Causes and solutions

Once you have the three pieces of information from Gather information, use the following table to diagnose your system.


Indexing status	Indexing rate	Queue fill pattern	Diagnosis	Remedy
Normal	Either low or high	Flat or spiky	Your indexer is running normally.	None required.
Normal	High	Saturated	Indexers are at or near capacity.	If this behavior persists through all cycles of system usage, you can configure more indexers, parallel pipelines, or high performance storage. See Dimensions of a Splunk Enterprise deployment in the Capacity Planning Manual.
Normal	Low	Saturated	Possibilities include: Insufficient provisioning. A configuration error (very large events or a bad transform). Search competing with your system I/O or CPU. A Splunk software defect.	Respectively: Store hot buckets on better hardware and/or get more CPU power. To find a bad transform, run the Monitoring Console Health Check, click Event processing issues, and follow the drilldown. Investigate search processes in Monitoring Console > Resource Usage: Machine. For a suspected Splunk software defect, file a Support case. See Known issues in the Release Notes.
Blocked or throttled	Nonexistent (zero)	Saturated	Possibilities include: Disk full. Indexer is forwarding to another indexer or a third-party system that is not accepting data. Hardware failure. Out of memory.	Respectively: Manage disk space. Check forwarding configuration. Ensure the indexer can write to disk. Ensure the indexer has enough memory. Contact Support.

Documentation