Troubleshoot Machine Data Lake

Resolve common setup, preview, promotion, field-summary, pipeline-limit, and stack-health issues in Machine Data Lake.

Use this topic when a Machine Data Lake setup, landing, preview, promotion, search, or management workflow fails or returns incomplete data. Start with the stage where the problem appears: sender connectivity, landing preview, raw event validation, a preserved index route, Catalog metadata, promotion preview or job state, promoted search, or lifecycle management. If a production route changes unexpectedly, restore the previous known-good input, routing, or landing configuration before you retry. Before contacting Splunk Support, collect the dataset name, unique test event identifier and time range, input and pipeline details, promotion job identifier, error message, and last-known-good configuration.

Some remedies require administrator access to stack services, service accounts, content sync, or Ingest Processor capacity. You can troubleshoot only the datasets and promotion jobs that you can view or manage.

Search Processing Language version 2 (SPL2) modules or data orchestration services are missing or unhealthy.

Verify SPL 2 and search-path services

  1. Verify that SPL 2 modules are installed and turned on.
  2. Verify that data orchestration services are healthy.
  3. Retry raw search, Static promotion to an analytics table, or Splunk index promotion workflows after the services are healthy.

The promotion profile uses unsupported SPL2 commands for Splunk index promotion processing.

Revise unsupported Splunk index promotion commands

  1. Review the job error and the selected promotion type.
  2. Remove or replace processing that is not offered by the promotion preview for that promotion type.

Field-summary or promotion-preview field data is delayed, incomplete, unavailable, or filtered incorrectly.

Investigate field-summary and preview-field issues

  1. Check whether field values or promoted-field selections are still updating.
  2. Review ingest-time filtering and confirm that inBytes values are not unexpectedly zero.
  3. Account for delayed or missing field values when validating preview and discovery results.
  4. If the promotion preview shows no available fields, invalid fields, or an update error, revise the filters or selected fields and run preview again. Contact an administrator if the fields remain unavailable.

Catalog metadata is delayed, stale, incomplete, or unavailable because background metadata collection has not refreshed yet or is not running as expected.

Check the Catalog metadata refresh

  1. Check the dataset last updated time in the Catalog before treating missing metadata as a data issue.
  2. Allow for the documented refresh interval and check the dataset last-updated time. Catalog field values and raw table statistics are not real-time landing signals.
  3. If field-value filters do not apply to a Splunk local index or non-MDL federated dataset, use keyword search instead.
  4. If metadata remains stale or the Catalog listing is unexpectedly empty, contact an administrator to verify that Catalog background operations are running for the stack.

The Ingest Processor pipeline limit has been reached.

Reduce pipeline usage

  1. Review raw landing pipelines and streaming promotion pipelines.
  2. Remove unused pipelines or pause work that creates additional streaming promotion pipelines.
  3. Retry the dataset or promotion operation after capacity is available.

The service role for Machine Data Lake is missing or does not include the capabilities required for the operation.

Verify the Machine Data Lake service role

  1. If raw table creation, promotion, or Splunk index target creation fails with an authorization or 403 error, ask an administrator to verify that the Data Management service account role exists on the same search head unit or search head cluster that the Data Management connection uses.
  2. For Splunk index promotion failures during target index creation, verify that the service role includes indexes_edit.
  3. After the service role is provisioned or corrected, retry the failed operation.

The promotion uses unsupported lookup-based processing.

Remove lookup-based promotion processing

  1. Remove the Lookup action or SPL2 lookup reference from the promotion configuration.
  2. Run promotion preview again to confirm that the promotion returns the expected data.
  3. Create a new promotion after the preview succeeds.
  4. If the promotion still fails after you remove lookup-based processing and preview succeeds, contact Splunk Support and provide the promotion job identifier and error message.