Manage data retention settings

Each index uses two settings on the New Index page to determine when to delete data:

  • The maximum size of the raw index data (MB, GB, or TB, specified in the Max raw data size field)
  • The maximum age of events in the index (specified in the Searchable time (days) field)

When the index reaches the specified maximum size or events reach the specified maximum age, the oldest data is deleted or is moved to your self-storage location (depending on your configuration).

For example, the system ingests data from a particular datasource at an approximate rate of 10 GB per day, and you want to retain and search against the last 90 days worth of data. This would cause the system to normally ingest approximately 900 GB over the configured 90 day searchable time period. 

Based on your search and data retention requirements, set these values so that the Searchable time (days) value is reached before the Max raw data size threshold is reached. A best practice is to set the maximum raw data size to a significantly larger value than the normal total ingestion amount. Doing this allows for unanticipated bursts of data that might otherwise cause the system to start deleting data before reaching the desired retention limit.

Given the above parameters, you might configure the retention settings as follows:

  • Max raw data size set to 1800 GB (double the example 900 GB normal total ingestion amount)
  • Searchable time (days) set to 90

The graphic shows data retention settings for an index. It is intended to orient the user.

These values together account for both your ingestion rate and the time you want to retain the data. You need to consider these factors for each index that you create.

The new data retention settings appear after you click Save and refresh the page.

Check your data retention in the Cloud Monitoring Console to ensure you estimated your ingestion rate correctly and your storage consumption is within your entitlement. If you did not correctly estimate your ingestion rate, you might have a shorter retention period than expected.

Splunk Cloud Platform administrators can specify the settings that determine when data is removed from a specific index. For more information, see the following:

Splunk Cloud Platform includes several internal indexes that are named starting with an underscore (_). The data retention period for these internal indexes cannot be modified.