Retention and deletion lifecycle

Understand how Machine Data Lake retention, expiration, deletion, promotion jobs, sharing, and downstream consumers relate before you remove data or change lifecycle settings.

Retention settings

Machine Data Lake uses separate lifecycle settings for raw tables and promoted datasets. Changing one setting does not automatically change the retention period for every related dataset or route.

Setting Where you set it What it controls What to verify
Raw table retention The raw table creation workflow and the raw table Manage tab. How long landed data remains available in the raw table for preview, Catalog discovery, raw search, and promotion. Verify the available values and default value shown in your environment before you create or update the dataset.
Static promotion retention default The raw table promotion-limits settings. The default retention period for promotion target created by static promotions that are created from the raw table. Verify whether the workflow lets you override the default for the promotion you create.
Streaming promotion rolling window The raw table promotion-limits settings and the streaming promotion workflow. How long matching events remain available in the promoted streaming destination. Confirm that the rolling window does not exceed the maximum retention configured for the raw table.
Open Sharing expiration The sharing controls shown for the dataset type where Open Sharing is supported. How long the sharing profile or token can be used by authorized external consumers. Verify the expiration period and revocation behavior before you share production data.

When retention begins

Treat raw table retention as applying after events land in the raw table. Treat promoted dataset retention as applying after the promotion creates or updates the promoted destination. Raw table retention and promoted dataset retention are independent lifecycle controls.

When a retention period expires, the expired data is no longer available from the affected dataset. Expiration of data in one route does not guarantee that related copies, preserved Splunk index routes, shared profiles, or downstream consumer copies are also removed.

Lifecycle impact

Review the impact before you delete a dataset, delete a job record, revoke sharing, or reduce retention. Some actions can interrupt searches, dashboards, alerts, downstream analysis, promotion jobs, or external consumers.

Lifecycle action Raw data and Catalog metadata Promoted data and jobs Sharing and downstream consumers
Raw data expires Expired raw data is no longer available for raw search, preview, or future promotion. Catalog metadata might not update immediately. Existing promoted datasets keep their own lifecycle unless the confirmation message or product behavior states otherwise. Consumers that depend on raw data must use a promoted, shared, or preserved route that still retains the data.
Raw table is deleted The raw table is removed from supported Machine Data Lake workflows after the deletion completes. Review the confirmation message for effects on active promotions, completed promotion jobs, promoted datasets, and future promotion from that source. Review any Open Sharing profiles, preserved index routes, dashboards, alerts, notebooks, reports, or machine learning workflows that depend on the raw table.
Promoted dataset is deleted The source raw table remains subject to raw table retention unless the source is separately deleted or expired. Deleting a promoted dataset can cancel related promotion jobs. Review the confirmation message before you confirm deletion. Searches, dashboards, alerts, Open Sharing profiles, and downstream consumers that use the promoted dataset can stop working.
Promotion job record is deleted The source raw table is not deleted by deleting only the job record. Deleting a job record removes the record from the job list. It does not delete the promoted dataset. Consumers of the promoted dataset can continue to use the dataset if their access remains valid.
Open Sharing profile or token expires or is revoked The dataset remains governed by its own retention and access settings. Promotion jobs and promoted datasets are not deleted only because a sharing profile expires or is revoked. External consumers lose access through that sharing profile or token. Confirm whether they copied data into another system.

Before you confirm deletion

  • Confirm the dataset type: raw table, promotion target created by static promotion, Splunk index target created by streaming promotion, local Splunk index, or non-MDL federated dataset.
  • Review the confirmation message in the workflow and stop if it does not match the expected lifecycle impact.
  • Check active and completed promotion jobs, shared profiles or tokens, dashboards, alerts, reports, notebooks, and machine learning workflows that reference the dataset.
  • Confirm whether a preserved Splunk index route or another downstream system still contains a copy of the data.
  • Verify that you have the required dataset management permission or administrator role. For permission details, see Access control, roles, and capabilities.
Attention:

Before you reduce retention or delete a dataset, review the confirmation message and verify the effect on raw data, promoted datasets, active jobs, sharing profiles, and downstream consumers. If the lifecycle impact is unclear, contact Splunk Support before you continue.