Machine Data Lake limits and caveats

Review capacity, search, discovery, promotion, service-account, and error-message caveats for Machine Data Lake.

Capacity and scale

Area	Caveat	Guidance
Ingest Processor pipelines	Landing and dual-routing workflows that use an Ingest Processor pipeline, and streaming promotion, count toward applicable Ingest Processor service limits. Direct sender landing does not by itself require a customer-configured Ingest Processor pipeline.	Review current Ingest Processor service limits before activating additional Ingest Processor landing, dual-routing, or streaming-promotion pipelines.
Static promotions	Static or historical promotion counting and blocking behavior can differ from streaming pipeline behavior.	Review the promotion estimate and configured promotion volume limit before you start the promotion.
Landing scale	Supported landing scale depends on the provisioned environment, source mix, event rate, dataset count, and current service limits.	Review current service limits and confirm the planned production workload with Splunk Support before onboarding at high scale.
Streaming promotion scale	Streaming promotion fan-out can multiply processing load.	Treat streaming promotion capacity separately from raw landing capacity.

Promotion availability

Area	Caveat	Guidance
Promotion initiation	Create promotions from the Catalog promotion workflow. Starting a promotion by API is not supported in this release.	Use the Catalog to configure, preview, and start static promotion to a Splunk index, static promotion to an analytics table, and streaming promotion to a Splunk index.
Existing promotion targets	Promotion creates a new promoted dataset. You cannot promote data into an existing Splunk index or existing analytics table.	Use a unique promoted dataset name and promote only the time range, filters, and fields that you need for the workflow.
Streaming promotion target	Streaming promotion is supported for Splunk index promotion. Streaming promotion to an analytics table is not available.	Use streaming promotion to a Splunk index for matching events that arrive after the promotion job becomes Active. Streaming promotion does not backfill data already stored in the raw table; use a separate static promotion for history. Use static promotion to an analytics table for structured historical analysis.
Lookup-based promotion processing	Lookup actions and SPL2 lookup processing are not supported in Machine Data Lake promotion workflows.	Remove lookup-based processing before you preview or start the promotion. Use supported filters, field selection, masking, deduplication, summarization, or custom command functions only where the promotion workflow supports them.
Existing Splunk index workflows	If existing searches, dashboards, alerts, or other knowledge objects depend on an existing Splunk index, promotion does not add data to that existing index.	Review downstream dependencies before changing the data path. Continue using the existing Splunk index path when those dependencies must keep using the existing index.

Discovery and search

Area	Caveat	Guidance
Catalog refresh	Catalog metadata does not refresh in real time. Dataset statistics, field schema, and field values refresh on background schedules that vary by dataset type.	Use the last updated time and allow for refresh latency before treating missing metadata as a data issue.
Catalog field values	Field values for Machine Data Lake raw tables and promoted datasets can lag behind landed data or promotion completion because metadata in the Catalog refreshes on a background schedule. For large promotion jobs, default field-value filters such as source, sourcetype, host, and time might not include the promoted dataset immediately after the promotion job completes.	Allow time for field summaries to refresh before relying on field-value filters for recently landed data.
Machine Data Lake dataset statistics	Machine Data Lake raw table statistics, such as size and event time range, refresh on a background schedule and are not a real-time landing signal.	Use dataset statistics as recent metadata, not as real-time confirmation of landing.
Splunk index schema	Splunk index field schema might refresh less frequently than Machine Data Lake field values.	Allow additional time before troubleshooting missing Splunk index field schema.
Federated dataset schema	Federated dataset field schema is available only for catalogs crawled by Splunk. Field schema might not be available for external catalogs.	Do not assume field schema is available for every federated dataset in the Catalog.
Field-value filtering	In Splunk Cloud Platform 10.5.x, field-value filtering is available for Machine Data Lake datasets only. It is not available for Splunk local indexes or federated datasets.	Use keyword search for non-Machine Data Lake datasets.
Stale metadata indicators	The Catalog might not show a warning banner when background metadata is paused, delayed, or stale.	Check the last updated time and retry after the expected refresh interval. Contact an administrator if metadata remains stale.
Machine Data Lake raw table collection	Catalog metadata collection is batched and can require additional time in environments with many raw tables.	Use the dataset last-updated time and event validation before treating missing Catalog metadata as a landing failure. Contact Splunk Support if metadata remains stale.
Raw table search fields	Raw table search supports source, sourcetype, host, and time.	Use these fields and time filters to narrow validation searches before you promote data for richer field behavior.
Raw search cost	Raw tables are storage-optimized, not search-optimized, and the search experience can warn you before opening raw search.	Use raw search to validate and narrow the data slice, then promote data when you need higher-performance search, dashboards, alerting, or downstream analytics.
Search Processing Language version 2 and data orchestration	Raw table and analytics search paths depend on Search Processing Language version 2 (SPL2) modules and data orchestration services.	Verify those services when raw table search or analytics search fails.

Scope boundaries

Area	Caveat	Guidance
Open Sharing	Open Sharing depends on dataset type, stack support, and sharing permissions.	Use the sharing controls shown in the Catalog. Revoke a sharing profile or token when the authorized access window ends.
Multiple search head units	If your deployment has multiple search head units, use the search head unit linked to the Machine Data Lake Data Management connection for raw table creation, data landing, discovery, raw search, preview, promotion, management, and deletion. Cross-search-head-unit workflows for Machine Data Lake raw tables are not supported in this release. Contact Splunk Support before changing the linked search head unit.
Raw storage writes	Data written directly to object storage outside the supported landing workflow is not registered as a Machine Data Lake raw table.	Land data by using supported input and Data Management workflows so that the Catalog can show dataset metadata, lineage, search, and promotion actions.
Federated data	Federated datasets are queried in place and are not Machine Data Lake raw tables.	Use federation when data must remain in another supported system and you only need an in-place query path.
Automation boundaries	Use the Catalog for promotion, sharing, and dataset management. Use only documented APIs for supported automation. The Catalog remains the supported workflow for promotion and dataset management unless an API is explicitly documented.