Promote data to an analytics table

Create a static or streaming analytics table promotion from selected data in a Machine Data Lake raw table.

  • You have static promotion permission for static promotions or streaming promotion permission for streaming promotions on the selected raw table.

  • You have view or query access to the raw table so that you can review dataset details and run promotion preview before you submit the promotion.

  • You know the fields, field names, and field types that the analytics workflow requires. Define the exact output schema before you start the promotion.

  • You have sample events that contain the fields you need to extract from the raw data.

  • To use props and transforms field suggestions, verify that you have access to supported extraction content for the source type. An add-on supplies this content. For information about props and transforms field extractions, see Configure custom fields at search time.

Promote data to an analytics table when you need selected fields for analytics search, reporting, notebooks, business intelligence, machine learning, or large-scale analysis. Use a static promotion for a fixed historical time range. Use a streaming promotion to keep the analytics table updated as matching events arrive.

Note: A streaming promotion includes matching events that arrive after the job starts. To promote events already stored in the raw table, create a separate static promotion.

An analytics table promotion requires field extraction and schema definition. When a technical add-on provides supported P&T extractions for the source type, the workflow automatically suggests fields. Review the suggestions, select the fields you need, and define the output schema. If no applicable content is available, use regular expressions to extract the required fields. Use supported Search Processing Language version 2 (SPL2) to extract additional fields or shape the schema. Promotion does not remove the source data from the raw table, which keeps its own retention settings.

  1. From the global navigation bar in Splunk Cloud Platform, select the Catalog (Image of the "Catalog" icon) icon.
    On the Catalog page, the Datasets tab is selected by default.
  2. In the Datasets tab, find the raw table that contains the data you want to promote.

    You can search, filter by time range, sort table columns, or move through pages of datasets.

  3. Select the raw table and then select Promote.

    The promotion workflow opens for the selected dataset.

  4. On the Configure step, select the analytics table promotion type.

    For Splunk search, dashboards, alerts, scheduled searches, monitoring, or correlation, see Promote data to a Splunk index.

    Promotion type When to use
    Static analytics table Create an analytics table from a fixed historical time range.
    Streaming analytics table Keep an analytics table updated as matching events arrive.
  5. Set the time range or rolling window for the selected promotion type.

    For a static promotion, select the historical time range that contains the data you need. Keep the range focused on the events required for your analysis.

    For a streaming promotion, set the rolling window to specify how long data remains available in the analytics table.

  6. (Optional) To narrow the promotion to specific events, select Add filter.

    Choose a field such as host, sourcetype, or source, and enter the value that identifies the events to include. Add more filters to narrow the data further, or remove a filter that you no longer need.

  7. Select Next.
  8. On the Preview page, review the suggested fields.

    Field suggestions appear automatically when an add-on provides supported props and transforms extractions for the source type.

    If no applicable props and transforms extraction content is available, select Extract fields from raw and use regular expressions to extract the required fields.

  9. Select the fields to include in the analytics table.

    Keep only the fields that the analytics table must contain. Remove fields that are not needed for the analytics workflow, and confirm that the retained fields use clear names and expected values.

  10. Define the exact output schema for the analytics table.

    Confirm each field name, field type, and output shape before you create the table. The schema determines the fields and types available when you query the analytics table.

    Use representative events to check fields that are missing from some events or contain values of different types.

  11. When required fields are missing or the output schema needs additional shaping, turn on SPL2 editor and add custom SPL2.

    Use supported custom SPL2 to extract additional fields from _raw, rename fields, cast values, or shape the table schema when the field extraction output is not sufficient.

  12. Select the Run (The Run icon) icon or press Control+Enter to preview the analytics table output.

    Validate that the preview includes the expected rows, excludes unwanted events, and shows the exact fields and types that the analytics workflow requires.

  13. Revise the time range, filters, extracted fields, schema, or custom SPL2 until the preview matches the analytics table you intend to create.

    If preview reports an extraction error, invalid field, or type conflict, review the error details. Revise the extraction, selected fields, or supported custom SPL2, and run preview again.

    Run preview again after each change. Do not continue until the preview output matches the required schema and representative rows.

  14. When the preview matches the analytics table you want to promote, select Next.
  15. On the Review and finalize page, enter a unique promoted dataset name and optional description.

    Use a name and description that help you and your team understand what the analytics table contains, which raw table it came from, and which schema it uses.

  16. Review the Promotion job summary and Dataset summary sections.
  17. Select Start promotion.
    Machine Data Lake creates a promotion job and opens the promotion job list in the Catalog Promotion jobs tab. Machine Data Lake submits the promotion request. With review enabled, static promotion requests above the raw table's positive volume limit wait for administrator approval. Approval starts the promotion job. Rejected requests do not run, and the rejection remains visible to the requester. Requests that do not require review proceed to execution.

When execution starts, Machine Data Lake creates an analytics table using the selected filters, fields, and field types.A static promotion includes data from the selected historical time range. A streaming promotion continues to add matching events while the job is active. The source data remains in the raw table.

Note: You can find the promoted dataset in the Catalog only on the search head where you created the promotion.

In Catalog, select the Promotion jobs tab and refresh the job list. Open the job details and follow the promoted dataset link when the static promotion completes or the streaming promotion becomes active. See Monitor and manage promotion jobs.

Search the resulting analytics table and compare representative rows with the preview. Confirm that field names, values, and types match the output you validated before promotion. Include events where an extracted field is missing or contains values of different types.

If the output differs from the preview, review the promotion job details and available error information before using the table for analysis.

To search the analytics table, see Search Machine Data Lake data from the Catalog.