Monitor current SVC usage of your workload-based subscription

If your Splunk Cloud Platform subscription plan measures your deployment's ingestion and search workload consumption by Splunk Virtual Compute (SVC) units, Splunk Cloud Platform administrators use the Workload dashboard on the CMC to monitor usage. For more information about the SVC entitlement for your workload-based subscription, see Performance considerations in the Splunk Cloud Platform Service Description. Be sure to view the correct service description version for your Splunk Cloud Platform deployment version.

Review the Workload dashboard

The Workload dashboard contains panels visible to Splunk Cloud Platform administrators that show SVC entitlement and usage for either ingest-based or workload-based subscriptions over a specific time range.

This dashboard shows your deployment's overall SVC usage and can help locate where you can optimize your organization's SVC consumption. Hover your mouse pointer over a vertical bar or a point on a line to view data for a specific hour.

The SVC usage per hour by search type and SVC usage per hour by top <variable> panels represent less accurate data due to sampling rates. These panels use the search_launcher process, which represents searches that take less than 10 seconds to complete. This process might hide a lot of data. For more accurate data, view the Search time by search type and Search time by top 10 apps, users, and searches panels.

To investigate your panels, go to Cloud Monitoring Console > License Usage >  Workload. Use the following table to understand the dashboard interface.

Panel Description
Total number of licensed SVCs Shows the number of SVCs assigned to your organization's subscription per your license entitlement.

This panel displays an N/A for the following scenarios:

  • Subscription status: Your organization has a new workload-based subscription and Splunk is still processing your SVC entitlement. Once this process is complete, your entitlement will appear.
  • Subscription type: Your organization uses ingest-based licensing. Contact your Splunk account representative to convert your subscription type from ingest-based to workload-based.
Peak SVC usage Shows your organization's SVC usage against the license limit.

This chart shows hourly usage calculated in standard 1 hour time blocks, meaning 9:00-9:59 AM or 11:00-11:59 PM. Use the time picker to adjust the granularity by 1 hour, 15 minutes, or 5 minutes. Finer time granularity selection offers increased visibility into when SVC usage peaks or dips within a given timeframe, so you can understand whether usage is consistently high or if there might be specific workloads causing spikes in usage.

The displayed data excludes data gathered during both the current hour and one previous hour. This means that if you are viewing this chart at 2:58 PM, data from 1:00-1:59 PM (the previous hour) and 2:00-2:59 PM (the current hour) is excluded from calculation. At 3:00 PM, data from 1:00-1:59 PM will be included, and at 4:00 PM, the data from 2:00-2:59 PM will be included. This exclusion is to ensure the correct calculation of your organization's SVC utilization.

For workload-based subscriptions:

  • Color-coded vertical bars show the following about SVC usage:
    • Blue bars indicate usage that is below the optimal threshold.
    • Yellow bars indicate usage that is at or above the optimal threshold of 80% of the licensed amount. Splunk Cloud Platform administrators might see issues with their deployment when the usage remains elevated for extended periods of time.
    • Red bars indicate usage that is above 90% of the licensed amount. This indicates a degraded state. Splunk Cloud Platform administrators will likely see issues with their deployment when the usage remains degraded for extended periods of time.
  • Color-coded horizontal reference lines show the following:
    • Green: Your organization's average SVC utilization.
    • Yellow: The optimal utilization threshold, which is calculated as 80% of the license limit.
    • Red: Your organization's SVC entitlement or license limit.

Generally, SVC usage should be less than 80% to maintain performance. 80% to 90% is considered elevated usage. Greater than 90% usage might cause degraded performance. If utilization exceeds 80%, look at the detail panels and consider optimizing processes that are high SVC consumers. Or, you can contact your Splunk account representative to discuss increasing your license entitlement.

For ingest-based subscriptions, the following elements don't appear:

  • Reference lines for SVC entitlement and 80% utilization threshold.
  • The yellow elevated and red degraded usage bars.

Note: The displayed SVC values for ingest-based subscriptions are only a projected estimate. The actual appropriate SVC entitlement for your organization might be affected by various usage factors. To determine the appropriate SVC entitlement for your deployment and to convert your ingest-based subscription to a workload-based subscription, contact your Splunk account representative.
Peak SVC usage as a percentage of allocated SVCs per tier Shows SVC peak usage as a percentage of SVCs provisioned by the search head and indexer tier. Use the time picker to adjust the granularity by 1 hour, 15 minutes, or 5 minutes.

Provisioned SVCs are allocated to the search head and indexer tiers after initial sizing conversations about intended workloads and requirements, with intention to minimize the footprint for both tiers. Viewing the usage as a percentage of provisioned SVCs provides insight on a tier level and helps you understand what utilization looks like if one tier is over extended. Review the percentage usage on each tier to identify which tier is close to exceeding the optimal range of greater than 80%.

This panel has the following limitations:

  • This panel uses a new calculation as of CMC version 3.12.0 and does not display historical data. The data requires history before it's visible in the CMC. On day of release, this panel will contain approximately a week's worth of data.
  • This panel does not break down usage percentage by individual search heads.

The displayed data excludes data gathered during both the current hour and one previous hour. This means that if you are viewing this chart at 2:58 PM, data from 1:00-1:59 PM (the previous hour) and 2:00-2:59 PM (the current hour) is excluded from calculation. At 3:00 PM, data from 1:00-1:59 PM will be included, and at 4:00 PM, the data from 2:00-2:59 PM will be included.

Peak SVC usage per hour split by process Shows SVC consumption per hour by system processes and resources.
  • Ingestion: Encompasses both ingestion and indexing processes. This includes any index or scripted_input process and also processes on indexers that are not counted in the search or shared services categories. See the SVC Usage by Ingestion panel for a breakdown of the ingested data by either index or source type.
  • Search: Encompasses any running search process where the process_type starts with search.
  • Shared services: Encompasses internal system processes necessary to maintain service to your deployment. This includes any other non-search process on the search head, such as kvstore and splunk_web processes.
<variable> (search seconds, SVC usage) per hour by search type Search seconds per hour by search type shows search seconds per hour by search type. This is the default view for this panel.
  • REST_API: Searches that use the Splunk REST API. See Basic concepts about the Splunk platform REST API.
  • ad-hoc: Searches that are unscheduled and manually run. See ad hoc search.
  • dashboard: Searches run by your dashboards
  • scheduled: Searches that are saved and scheduled so they automatically run. See scheduled search.
  • summary director: Maintenance tasks that run in the background involving caching and summarization to ensure searches are processed.

Select estimated SVC to view SVC usage per hour by search type. This shows SVC consumption per hour as categorized by one of the following assigned search types. If the consumption can't be categorized in an assigned search type, it is grouped in the general other category.

  • ad-hoc: Searches that are unscheduled and manually run. See ad hoc search.
  • report acceleration: Searches that are related to accelerated data models or reports. See data model acceleration, report acceleration, and How data model acceleration differs from report acceleration and summary indexing.
  • scheduled: Searches that are saved and scheduled so they automatically run. See scheduled search.
  • scheduled realtime: Searches where the search_mode field value is realtime indexes (RT Indexes) and the search_type field value is scheduled.
  • search launcher: Ephemeral searches that are managed by the search launcher, which is a splunkd helper process that is responsible for forking new search processes and managing a high number of fast-running searches on deployments. Because the individual ephemeral searches are being quickly processed, your deployment's SVC usage for these searches is based on the search launcher process to ensure an accurate SVC calculation.
<variable> (Search seconds, SVC usage) by top 10 <process type> (apps, searches, users) Search seconds by top 10 <process type> shows search seconds per hour grouped by consumer type and search head. You can identify which apps, users, and searches per search head have relatively high search times. This is the default view for this panel.

Select estimated SVC to view SVC usage by top 10 <process type>. This shows high consumers of SVC per hour grouped by consumer type and search head so you can take steps to optimize their consumption. For example, by analyzing the users and searches data, you can contact high consumers of SVC and discuss ways to optimize their consumption, such as improving their search queries.

Select one of the following options from the Process type drop-down menu:

  • Apps: Lists a maximum of the top 10 apps and their respective SVC consumption.
  • Users: Lists a maximum of the top 10 users and their respective SVC consumption. These users may be human or virtual administrators.
  • Searches: Shows which searches utilize the greatest SVC as a percentage of the total consumption.

Select one of the following options from the Search head drop-down menu:

  • All: Shows all search heads in your Splunk Cloud Platform deployment. This category includes all the data ingested and processed in the deployment.
  • Historical: Shows a different view of All. This category includes all the data ingested, processed, and summarized in the deployment prior to the CMC 2.9.0 release.
  • Specific search head name: Shows data for a specific search head that has been ingested, processed, and summarized in the deployment as of and after the CMC 2.9.0 release.

Note: One virtual administrator is the internal splunk-system-user, which runs jobs and processes like summary refreshes, report accelerations, and data model accelerations for a deployment on behalf of a Splunk Cloud Platform customer. Running these processes consumes SVCs. If the SVC usage of splunk-system-user seems abnormal, Splunk Cloud Platform administrators should contact the deployment's administrator to investigate the increased consumption.
Dispatched and skipped search count per hour Shows the number of searches per hour that are dispatched or skipped.

The yellow vertical lines indicating elevated SVC usage and the red vertical lines indicating degraded SVC usage correlate to the same lines in the SVC Usage panel.

Peak SVC usage per hour by indexing source Shows SVC consumption per hour by ingestion source. Select either Index or Sourcetype from the drop-down menu.
Hourly rate of ingestion Shows the hourly rate of ingestion in GB. When data ingestion rates are high, the indexer consumes more resources to process and ingest data. High ingestion rates can increase SVC usage.

Interpret SVC usage results

See the table in Review the Workload dashboard in this topic for information on keeping your SVC usage within license limits.

In the Events tab for a search, the search_label field includes the _ACCELERATE_{SID_NUMBER} value so you can search for an event using its SID value.

You can also set up an alert action (for example, send an email) to be performed when a platform alert is triggered. Go to Settings > Searches, Reports, and Alerts and select New Alert to define a new alert action.