Share performance and usage data in Splunk Enterprise
Changes in version 9.1.0
There are minor changes to Splunk data collection practices in version 9.1.0, mainly for the Splunk Assist service. Data collection remains on by default. For more information on why Splunk changed its policy to enable the collection of usage data, see the 8.0 version of this topic.
The support usage data that Splunk collects for Splunk Assist and for telemetry are the same. The targets for these data sources, however, are different. You might need to update any firewall settings that you have before you can use Splunk Assist, even though the Splunk platform can send support usage data back to Splunk.
You can still opt out of data sharing at any time, but if you do, you cannot use the Splunk Assist service, which requires that data sharing is active. See How to opt out.
To learn more about Splunk Assist, see About Splunk Assist in the Monitoring Splunk Enterprise Manual.
Benefits of sharing data with Splunk
When you share data with Splunk, you receive the following benefits:
- Improved product quality. By collecting accurate information about the topology decisions and deployment scale used by our customers, we can replicate those topology configurations and scale in our internal testing, helping us improve your product experience.
- Timely notification of known bugs, version incompatibilities, and configuration issues. When you share data about the product versions you have deployed, we can provide accurate messages and support to help you with bugs, upgrade tasks, version compatibility problems, and other configuration issues you might experience.
- Relevant feature enhancements. We prioritize what features to develop and enhance first based on the features customers use the most. By sharing your data, you influence these data-driven decisions in favor of the features you use at your organization.
- You can use the Splunk Assist service to monitor your deployment in accordance with Splunk best practices for security, performance, and configuration.
For more information, see How Splunk uses the data it collects.
What data Splunk collects
The following table summarizes the data that your Splunk platform deployment sends to Splunk when you enable data collection. Follow the links to see examples of this data.
Type of data | Description | Examples |
---|---|---|
Aggregated usage data | Includes features used, deployment topology, and performance metrics in both the platform and apps. This data is not associated with your license ID. You must enable Aggreated usage data to use the Splunk Assist service. | Aggregated usage data examples App usage data examples |
Support usage data | Support usage data is the same as the aggregated usage data, but the license ID remains associated with your data when it reaches Splunk. You must enable support usage data to use the Splunk Assist service. | Aggregated usage data examples App usage data examples |
License usage data | Includes your license ID, active license group and subgroup, total license stack quota, total license pool consumption, license stack type, license pool quota, license pool consumption. | License usage data examples |
Software version data | Includes the version of Splunk Enterprise and of each installed app, along with relevant metadata about deployment architecture. | Software version data examples |
Some cloud and hybrid products modify the kinds of data that Splunk collects. When that happens, a separate agreement or notification states how the data collection differs for that product.
For instructions on how to view the data that your deployment collects and sends to Splunk, see View what data is sent from your deployment.
Examples of data sent to Splunk
Aggregated usage, support usage, and license usage data is sent to Splunk as a JSON packet that includes information like the component name and deployment ID, in addition to the data for the specific data collection component. The deploymentID is unique to a deployment and does not change on upgrade or even after uninstall and reinstall of Splunk Enterprise on the same machine.
Here is an example of a complete JSON packet:
{
component: deployment.app
data: { [-]
enabled: true
host: 878e7b21bf98580dbdb4ed3baf6c35d78aa5bc3d3c824eb8714a313c
name: search
version: 8.0.0
}
date: 2019-09-23
deploymentID: d6d8e776-a8d3-5467-a03b-375577646cbb
executionID: 2FC293C59049AC0D44B677D3A9D786
timestamp: 1569294102
transactionID: 4E1CFC7E-BE9F-355D-7DDE-D4F8D5E4852D
version: 3
splunkVersion: 8.1.2
visibility: anonymous,support
}
The following tables list the component names, descriptions, and an example of what data is collected for that component. For ease of use, the examples for aggregated usage and license data show examples of only the data
field from the JSON object.
Aggregated usage data examples
The following example demonstrates the data sent to Splunk when sharing of aggregated usage data is enabled.
Component | Description | Example |
---|---|---|
app.RapidDiag.cliAccessMetrics
|
RapidDiag CLI interface usage statistics. |
|
app.RapidDiag.uiAccessMetrics
|
RapidDiag UI interface usage statistics. |
|
app.RapidDiag.executionMetrics
|
RapidDiag task execution statistics. |
|
app.session.coreLibrarySettings.save
|
Tracks if certain core library settings are toggled on or off. |
|
app.session.createNewDashboardDialog.interact
|
General telemetry collected when a new dashboard is created. |
|
app.session.dashboard.load
|
Dashboard characteristics, generated as session data when a dashboard loads. |
|
app.session.dashboard.interact
|
Whether a user pressed Cancel or Continue for the URL warning modal. |
|
app.session.dashboard.error
|
If an asynchronous error occurred in a CustomJS script used by a dashboard. |
|
app.session.dashboard.telemetry
|
General telemetry collected when adding and configuring dashboard elements. |
|
app.session.dataactions.interact
|
User interactions in the dataactions UI. |
|
app.session.dataactions.load
|
Number of rulesets and type of deployment. |
|
app.session.datainteractions.load
|
Apps installed per Splunk instance. |
|
app.session.globalBanner.error
|
Unexpected error responses from GET/POST requests to the global banner endpoint, and the status code. |
|
app.session.globalBanner.interact
|
Tracks when a user clicks a banner link. |
|
app.session.html_dashboard
|
Count the number of HTML dashboards in the Splunk Enterprise instance. |
|
app.session.html_dashboard.load
|
Track the number of times an HTML dashboard is loaded. |
|
app.session.metrics.interact
|
Track the type of filter the user set on a chart. |
|
app.session.metrics.process
|
De-identified chart configuration data related to the queries sent by workspace charts. |
|
app.session.page.interact
|
Tracks user interactions with search, reports, alerts, data models, tags, lookups, and search macros. |
|
app.session.page.load
|
Tracks loads and whether web services are supported, generated as session data when a page loads. |
|
app.session.pageview
|
Page view session data, generated whenever a user visits a new page. |
|
app.session.pivot.interact
|
Changes to pivots, generated as session data when a user makes a change to a pivot. |
|
app.session.pivot.load
|
Pivot characteristics, generated as session data when a pivot loads. |
|
app.session.roles.srchFilter
|
Event actions on the authoritzation/roles page of Splunk Web |
|
app.session.rum.mark
|
Track performance of the first meaningful paint for the global banner settings page and the view itself, when enabled. |
|
app.session.rum.measure
|
Track performance of the first meaningful paint for the global banner settings page and the view itself, when enabled. |
|
app.session.search.interact
|
Search page interactions, session data generated by each user interaction with the search page. |
|
app.session.session_start
|
Session data generated when a user is first authenticated. Contains the deploymentID (identifier for deployment), eventID (identifier for this specific event), experienceID (identifier for this session), userID (hashed username), data.guid (GUID for instance serving the page). |
|
app.session.spotlightSearch.redirect
|
Tracks selections and redirects from settings menu interactive search bar to settings menu pages. |
|
app.session.tableUI.interact
|
Tracks interactions on the Table UI page. |
|
app.session.template.load
|
Tracks the number of times users access HTML template files that Splunk Enterprise no longer uses. |
|
app.session.udf.telemetry
|
General telemetry collected on visualization usage and settings. |
|
app.splunk_monitoring_console
|
Determines whether splunk_monitoring_console is enabled. If enabled, determines whether the mode is standalone or distributed. |
|
assist-app.appVersion.<appId>
|
Splunk Assist - App Assist |
|
assist-certificate.expiry
|
Splunk Assist - Certificate Assist |
|
assist-app.appVersion.<appId>
|
Splunk Assist - Config Assist |
|
scripted_inputc.telemetry
|
Describes how much data is ingested through scripted input. |
|
cherrypy.load
|
How frequently CherryPy routes are used. |
|
deployment.app
|
Apps installed on search head and peers. |
|
deployment.clustering.indexer
|
Host name of an indexer, replication factor, and search factor for indexer cluster. |
|
deployment.clustering.member
|
Indexer cluster member status. |
|
deployment.clustering.searchhead
|
Indexer cluster and search head connection status. |
|
deployment.distsearch.peer
|
Distributed search peer status. |
|
deployment.forwarders
|
Forwarder architecture: Number of hosts, number of forwarder instances, OS/version, CPU architecture, Splunk Enterprise version, distribution of forwarding volume |
|
deployment.httpEventCollector
|
Describes how much data is ingested through HEC for Splunk apps, add-ons, and connectors. |
|
deployment.index
|
Index type and configuration. Includes indicator of whether a metrics index has subsecond search capability. |
|
deployment.licensing.slave
|
License slaves. |
|
deployment.node
|
GUID, host, number of virtual and physical cores, CPU architecture, memory size, storage (partition) capacity, OS/version, Splunk Enterprise version |
|
deployment.remoteupgrade
|
Information about remote upgrade of universal forwarder. |
|
deployment.shclustering.member
|
Search cluster member status. |
|
htmlcleaner.dashboard
|
General telemetry collected on CSS tag usage. |
|
instrumentation.performance
|
Performance of instrumentation queries. |
|
licensing.stack
|
Licensing quota and consumption. |
|
modinputc.telemetry
|
Describes how much data is ingested through Splunk apps, add-ons, and connectors. |
|
performance.bundleReplicationCycle
|
Metrics for the bundle replication cycle. |
|
performance.indexing
|
Indexing performance: Core utilization, storage utilization, memory usage, indexing throughput, search latency. |
|
performance.search
|
Search performance: Core utilization, storage utilization, memory usage, indexing throughput, search latency. |
|
preactivation.activate_button.click
|
Splunk Assist: Click of the 'Turn on Splunk Assist' button on the preactivation page |
|
preactivation.support_button.click
|
Splunk Assist: Click of the 'Contact Splunk support' button on the preactivation page |
|
onboarding.activate_button.click
|
Splunk Assist: Click of the 'Turn on Splunk Assist' button on the onboarding page (landing page of Assist) |
|
overview.category_card.click
|
Splunk Assist: The mouse click event of the category cards at the top of the overview page |
|
overview.topology.node.click
|
Splunk Assist: The mouse click event of the topology cards in the Indicators breakdown panel on the overview page |
|
overview.overview_list.open_assist.click
|
Splunk Assist: The mouse click event of the action button on Overview table. Clicking on this button will open up the assist page for the indicator indicated by indicatorName |
|
usage.admissionRules.report
|
Admission rules: Status, list of rules enabled and rules triggered for filtered searches. |
|
usage.app.page
|
App name, page name, locale, number of users, number of page loads, generated as session data. |
|
usage.authMethod.config
|
Authentication method: Hashed host and GUID, authentication method (Splunk, LDAP, or SAML), MFA type (none, Duo, or RSA). |
|
usage.bucketmerge.clustered
|
Usage of cluster bucket merge command, cluster bucket list command, and cluster bucket merge command with -dryrun option. |
|
usage.bucketmerge.standalone
|
Usage of bucket merge command, bucket list command, and bucket merge command with --dryrun option. |
|
usage.configtracker.config
|
Whether or not the feature is enabled or disabled. What "mode" the feature is in (e.g. - diff, track_only, auto.) And what kinds of file paths, and/or fields are added to the denylist. |
|
usage.configtracker.introspection
|
Configuration file change logs made on a Splunk instance. |
|
usage.configtracker.searches
|
Configuration file change SPL queries that were run on an environment, and their corresponding results. |
|
usage.durableSearch
|
Number of users of the durable search feature, how durable search is being used (for scheduled searches? for summary indexing?), and commonly-used durable search setting values. |
|
usage.healthMonitor.currentState
|
Distributed health report: Enabled status, number of clicks, node status (node path, current color, worst color in last 24 hours), Splunk version. |
|
usage.healthMonitor.report
|
Health report manager: Alert actions and enabled status, feature thresholds and enabled status. |
|
usage.indexing.sourcetype
|
Indexing volume, number of events, number of hosts, source type name. |
|
usage.ingestactions.deletions
|
Count of destination and ruleset deletions |
|
usage.ingestactions.destinations
|
Characteristics of routing destinations |
|
usage.ingestactions.rulesets
|
Count of routing destinations, ruletset types, and ruleset conditions |
|
usage.kvstore
|
Metrics and performance data about KV store. |
|
usage.lookups.lookupDefinitions
|
Lookup definition metadata with hashed lookup names. |
|
usage.passwordPolicy.config
|
Password policy management: hashed host and GUID, attribute configurations. |
|
usage.python
|
Default setting for Python version in the app, path of the script with its name hashed, version of Python used in the script. |
|
usage.rest
|
Usage of an endpoint, HTTP method, status code, and user agent in a REST request made from a Splunk Enterprise SDK. The data that is collected includes the partial endpoint URL of the target feature. Any user-identifiable data or resource names in the URL are discarded. |
|
usage.savedSearches.alert
|
Usage of the saved search alerting functionality: triggering conditions and modes, alert actions, alert suppression, schedules, and so on. |
|
usage.search.concurrent
|
Distribution of concurrent searches. |
|
usage.search.report_acceleration
|
Report acceleration metrics. |
|
usage.search.searchTelemetry
|
List of commands and corresponding counts for all searches run on the system in the span of one day. |
|
usage.search.searchtelemetry.type
|
Search type, count, average bytes read, max bytes read, duration. |
|
usage.search.searchtelemetry.sourcetypeUsage
|
Sourcetype usage. |
|
usage.search.type
|
Number of searches of each type. |
|
usage.smartStore.Config
|
SmartStore global configuration, per index configuration, hashed internal and external index names. |
|
usage.streamingMetricAlerts
|
Usage of the streaming metric alerting functionality: group by alerts, triggering evaluations and thresholds, alert suppression, result enrichment or filtering, and alert actions. |
|
usage.users.active
|
The number of active users per day. |
|
usage.workloadManagement.report
|
Workload management: Hashed host and GUID, OS/version, server roles, WLM support and enable status, pool configurations, rule configurations. |
|
Support usage data examples
Support usage data is the same data as the aggregated usage data, but if you opt to send support usage data, Splunk can use the license GUID to identify usage data from a specific customer account to help troubleshoot support cases.
See Aggregated usage data examples.
Support usage data is distinct from diagnostic file data. Diagnostic files are never automatically generated and can only be sent to Splunk Support manually by a user with the appropriate permissions. For more about diagnostic files, see Generate a diag in the Troubleshooting Manual.
License usage data examples
The following example demonstrates the type of data sent to Splunk when sharing of license usage data is enabled.
Component | Description | Example |
---|---|---|
licensing.stack
|
Licensing quota and consumption |
|
Software version data examples
The following example demonstrates the software version data sent to Splunk for Splunk Enterprise when sharing of software version data is enabled.
Description | Example |
---|---|
CPU architecture | x86_64 |
Operating system | Linux |
Product | enterprise |
Splunk roles | admin |
License group, subgroup, and hashed GUID | Enterprise, Production, <GUID> |
Splunk software version | 7.0.0 |
The following example demonstrates the software version data sent to Splunk for each app when sharing of software version data is enabled for that app.
Description | Example |
---|---|
App ID, name, and version | gettingstarted, Getting Started, 1.0 |
Splunk version | 7.0 |
Platform, architecture | Darwin, x86_64 |
App usage data examples
In addition to the data enumerated in this topic, certain apps collect usage data. See the documentation for each app for details and examples.
- Splunk Add-on Builder: Share data in Splunk Add-on Builder
- Splunk App for AWS: Share data in the Splunk App for AWS
- Splunk Business Flow: Share data in Splunk Business Flow
- Splunk DB Connect: Share data in Splunk DB Connect
- Splunk Enterprise Security: Share data in Splunk Enterprise Security
- Splunk Industrial Asset Intelligence: Share data in Splunk Industrial Asset Intelligence
- Splunk IT Service Intelligence: Share data in Splunk IT Service Intelligence
- Splunk Machine Learning Toolkit: Share data in the Splunk Machine Learning Toolkit
- Splunk Security Essentials: Splunk Security Essentials Telemetry
How Splunk collects the data
If aggregated, support, or license usage data collection is enabled, a few instances in your Splunk Enterprise deployment collect data through scheduled searches. Most of the searches run in sequence, starting at 3:05 AM on the node that runs the searches, unless you change the schedule. All searches are triggered with a scripted input.
In addition, when aggregated or support data collection is enabled, session data about user activity transmits from the browser directly to the Splunk telemetry API.
Which instance runs the searches and sends data to Splunk
One primary instance in your deployment runs distributed searches that collect most of the usage data. This primary instance is also responsible for sending the data to Splunk. The instance that acts as the primary instance depends on the details of your deployment:
- If indexer clustering is enabled, the cluster manager is the primary instance. If you have more than one indexer cluster, each cluster manager is a primary instance.
- If search head clustering is enabled but not indexer clustering, each search head captain is a primary instance.
- If your deployment does not use clustering, the searches run on a search head.
If you opt out of instrumentation, the searches from the primary instance do not run.
Additional instances in your deployment run a smaller number of searches, depending on colocation details. If data collection is enabled, the data from these searches is collected by the primary node and sent to Splunk. If you opt out, these searches still run, but no data is sent.
https://quickdraw.splunk.com/telemetry/destination
or https://*.api.splkmobile.com
. If necessary, add these URLs for outbound traffic to your firewall allow list.Instrumentation in the Splunk Enterprise file system
After the searches run, the primary instance packages the searched data and sends it to Splunk. It also indexes the data to the _telemetry
index. Session data is transmitted directly to the telemetry API from the browser. It does not go to the _telemetry
index. The _telemetry
index retains the data for two years by default and is limited in size to 256 MB.
The instrumentation app resides in the file system at $SPLUNK_HOME/etc/apps/splunk_instrumentation
.
How Splunk uses the data it collects
If you share aggregated usage data, Splunk collects data about your Splunk software usage and aggregates it together with similar data from other deployments so Splunk can understand what features and workflows are most important to users and improve its products and services over time. Collected license IDs are used only to verify that data is received from a valid Splunk product and persisted only for deployments opting into license or support usage reporting. These license IDs help Splunk analyze how different Splunk products are being deployed across the population of customers and are not attached to any aggregated usage data.
If you share support usage data, Splunk links the data about your software usage to your installed license ID so that Splunk can provide improved support and services for your deployment. The Splunk Assist service uses support usage data to identify and provide insights to let you align your Splunk Enterprise deployment with Splunk best practices for security, performance, and configuration. The Support and Customer Success teams use this data to identify and troubleshoot support issues that you file and improve your Splunk software implementation.
If you share license usage data, Splunk uses the data to ensure compliance with your purchased offering.
If you share Splunk product version data, Splunk uses the data to track how many deployments use particular versions of Splunk software offerings and to provide in-product notifications when updates are available. For apps, version data is correlated with information about app downloads to populate app analytics views on Splunkbase provided to the app's developer, and to compute the number of installs on the app details page.
How Splunk transmits and stores the data it collects
When you enable aggregated, support, and license usage data sharing, Splunk Enterprise runs searches to collect this data and sends the search summaries to a collection endpoint. Session data and Splunk software version data is not included in the searches. Session data is sent from your browser as the events are generated. Version data about Splunk Enterprise is sent to Splunk by your browser after you log into Splunk Web. Version data about your Splunk apps is sent to Splunk daily through a REST call from splunkd to splunkbase.splunk.com. Data is transmitted to Splunk from a single primary instance in your deployment. See Which instance runs the searches and sends data to Splunk.
The Splunk platform encrypts telemetry data with transport layer security (TLS) before it leaves your deployment, and verifies authentication before it stores the data securely on Splunk cloud infrastructure. The infrastructure that customer telemetry uses has strict access controls that are subject to regular audit. For more information about how Splunk collects, uses, and discloses information about the data collected, see the Splunk Privacy Policy. For more information about Splunk's data privacy, security, and compliance practices, see Splunk Protects.
View the data your Splunk Enterprise deployment sends to Splunk
You can view aggregated usage, support usage, and license usage data that your deployment has recently sent in Splunk Web.
- Navigate to Settings > Instrumentation.
- Click the category of data you wish to view in Search.
This log is available only after the first run of the collection. To inspect the type of data that gets sent before you opt in on your production environment, you can opt in on your sandbox environment.
To view the browser session data, use JavaScript logging in your browser. Look for network events sent to a URL containing splkmobile
. Events are triggered by user actions such as navigating to a new page in Splunk Web.
To view version data that is sent for Splunk Enterprise, watch JavaScript network traffic as you log into Splunk Web. The data is sent inside a call to quickdraw.splunk.com.
How to opt out
Splunk collects support usage, aggregated usage, license data, and software version data by default. You can opt in or out at any time.
Prerequisite
To enable or disable collection of usage data, the user that you use to log into Splunk Enterprise must hold a role that includes theedit_telemetry_settings
capability.
Opt out of sharing aggregated or support usage data
To change your aggregated or support usage data sharing settings, follow these steps:
- Click Settings > Instrumentation in Splunk Web.
- Click the gear icon next to Usage Data.
- Adjust the sliders to enable or disable sharing aggregated or support usage data.
Opt out of sharing license data automatically
By default, Splunk collects license usage data based on your installed license to ensure compliance with your purchased offering. To disable sharing license data automatically, edit your local copy of the telemetry.conf
configuration file and set sendLicenseUsage = false
.
Certain license programs require that you report your license usage. The easiest way to do this is to automatically send this information to Splunk. If you disable automatic license data sharing, you can send license data manually. Follow these steps each time you want to send data manually:
- On a search head, log into Splunk Web.
- Select Settings > Instrumentation.
- Click Export.
- Select a date range and data type.
- Click Send to send data to Splunk directly or click Export to export the data to your local machine and send the data to Splunk using another mechanism.
Opt out of sharing software version data
To stop sending Splunk data about the version of Splunk Enterprise you have installed, edit the web.conf configuration file and set the value of the updateCheckerBaseURL
setting to 0
.
In addition, you can turn off version data sharing for each Splunk app. To disable notifications of new versions and stop sending Splunk data about the app version, set check_for_updates
to false
in the local copy of the app.conf
file for each app.
Opt out of sharing data and prevent future admins from opting in
To opt out from all collection of usage, support, and license data and prevent other admins from enabling it in the future, do the following on one search head in each cluster and on each non-clustered search head:
- Click Settings > Instrumentation in Splunk Web.
- Click the gear icon next to Usage Data.
- Disable all options.
- Click Settings > Roles.
- Remove the
edit_telemetry_settings
capability from theadmin
role. Users with this role no longer receive notifications about data collection, nor can they access Settings > Instrumentation in Splunk Web.
If you want to disable collection of usage information across multiple deployments of the Splunk platform that are not centrally managed, block DNS resolution of e1345286.api.splkmobile.com
.
How to adjust your data collection schedule
If you share data, the collection process begins daily at 3:00 AM by default. You can change the frequency and timing of this collection.
If all instances in your deployment are running Splunk Enterprise version 7.1.0 or later, you can schedule instrumentation to run starting at any hour of the day on a daily or a weekly schedule. The collection process runs a few searches in sequence on several instances in your deployment. Depending on the size of your deployment and whether you run instrumentation daily or weekly, it can take a few minutes before the final searches run on the primary instance to package and send the data to Splunk. See Which instance runs the searches.
Changing the instrumentation collection schedule has trade-offs. Scheduling the collection to run weekly instead of daily might decrease the total search load for the week. A weekly collection takes longer than a daily collection, because it gathers data from all seven days. If you choose weekly collection, set it for a day and time when you expect the search load to be low.
Change the collection schedule using Splunk Web
- On a search head, in Splunk Web, navigate to Settings > Instrumentation.
- Next to Usage Data, click the gear icon.
- Click Edit usage data schedule.
- Select a frequency, day, and time.
- Click Save.
You do not need to restart the search head.
Change the collection schedule using configuration files
You can change the collection schedule by editing the telemetry.conf
file. For guidelines on editing this file, see telemetry.conf.
- At the command line on any search head, navigate to
$SPLUNK_HOME/etc/apps/splunk_instrumentation/local/
. - Create or edit
telemetry.conf
. - Edit the values for any of
scheduledHour
,scheduledDay
, andreportStartDate
according to the guidelines intelemetry.conf.spec
.
Impacts on performance during collection of shared data
How to enable data sharing for Splunk Assist
If you want to use the Splunk Assist service to monitor your Splunk Enterprise deployment according to Splunk best practices, or need to turn data sharing back on after you have opted out, use this procedure to confirm that data sharing is active.
- Log into your Splunk Enterprise instance.
- From the system bar, click Settings > Instrumentation.
- On the "Instrumentation" page, click the gear icon next to Usage Data.
- In the pop-up window that appears, review the Aggregated Usage Data and Support Usage Data toggle switches. Ensure that both toggle switches are set to "Enabled".
- Click the gear icon again to close the Usage Data settings popup.
Data sharing is now on.