Create a Machine Data Lake raw table

Create a Machine Data Lake (MDL) raw table to land matching events and control who can view, query, or promote the data.

1. Open Data Management app and select Datasets.
2. Select Create dataset.
3. On the Select data store page, select Machine Data Lake (MDL), and then select Next.
4. On the Define dataset page, enter the dataset settings.
   1. In Dataset name, enter the dataset name.

      The name can include alphanumeric characters, underscores, and hyphens.

   2. (Optional) In Dataset description, enter a concise description that helps you recognize the dataset later.
   3. In Retention, select how long to keep data in the dataset.
   4. In the Promotion limits section, enter the maximum data retention period for static promotions from this dataset, and then select the time unit.

      You can select years, months, weeks, or days.

   5. Enter the maximum rolling window for streaming promotions from this raw table, and then select the time unit.

      Data outside the window is removed from the promoted destination, but remains subject to the source raw table retention period.

   6. In Volume limit, enter the maximum amount of data in gigabytes (GB) that can be promoted from this dataset.
5. Select Next.
6. On the Configure permissions page, add roles and accounts that need dataset-specific access.

   The dataset inherits default access from capabilities and policies.

   1. Select one or more roles or accounts from Add roles and users.
   2. Select the add icon.
   3. For each role or account, select an access level.

      • View

      • Query and view

      • Static promote, query, and view

      • Streaming promote, static promote, query, and view

      To remove a role or account from this list, select Remove.

7. Select Next.
8. On the Define data landing page, define how the process_data pipeline selects matching events and transforms them before they are written to the raw table.

   Use the source, sourcetype, and host values you recorded when preparing the data source. Partitions determine which incoming events the pipeline processes. Actions, such as filtering and masking, affect the preview results and the data written to the raw table after the configuration is saved.

   Data landing is one supported path for getting data into Machine Data Lake. Use it to select events that are already flowing through a supported HEC, Universal Forwarder, or Heavy Forwarder path to a Splunk index. Define partition conditions by source, sourcetype, or host to route matching events to this Machine Data Lake raw table. With MDL-only routing, matching events are redirected to the raw table instead of the existing index. With dual routing, matching events land in the raw table and continue to the existing index. Events that do not match the partition conditions continue to their configured destination. Before you save the landing definition, preview representative matching and non-matching events. Confirm that matching events land in Machine Data Lake and that non-matching events follow the intended destination. If the preview is unexpected, revise the partitions, filters, masking, actions, or routing mode before you create the dataset.

   For more information about Ingest Processor pipeline flow, see How data moves through the Ingest Processor solution.

   1. Select the process_data pipeline.
   2. (Optional) Add a partition to define the subset of data that the pipeline processes.

      You can add a partition by sourcetype, source, or host.

      For each partition, select source, sourcetype, or host as the field, use the IN operator, enter or select at least one non-empty value, and then select Apply. To add multiple partition conditions, select the plus icon again. Data that does not match the partition conditions follows the configured non-matching destination for the pipeline.

      For more information about configuring partitions, see Create pipelines for Ingest Processor. For more information about predicate syntax, see Predicate expressions.

   3. (Optional) Add sample data to preview the landing definition.

      Use representative sample data in the same format as the events that you want to land. Include events with the source, sourcetype, and host values that you use in the landing definition.

      Select Add sample data, enter or upload sample data, select Raw text or CSV as the sample data format, and then select Apply. You can enter a maximum of 30,000 characters. After you add sample data, the Sample data section shows Inserted Sample.

   4. Select the Preview Pipeline icon () to preview the landing definition.

      The preview results appear in the results table. The table shows the number of results, the preview time range, and the fields returned by the current configuration. Use the View: Table menu to review the results in table view.

   5. (Optional) Add an action to filter, mask, or process matching events.

      Actions affect preview results and the data written to the raw table after the landing definition is saved. In the Actions section, select the plus icon (), and then select one of the following actions:

      • Filter values: Filter events by matching values. Events excluded by the filter do not land in the raw table.

      • Mask values in _raw: Use a regular expression to replace matching values in the _raw field before the events land in the raw table. Enter the matching regular expression, enter the replacement value, optionally select Match case, and then select Apply.

      • Apply custom command function: Apply a published custom command function to transform matching events before they land in the raw table. Search for and select a custom command function, enter any required parameters, and then select Apply.

      For more information, see Filter and mask data using Ingest Processor and Create and apply a custom command function for the Ingest Processor solution.

   6. Review the fields that the pipeline detects from the sample data or preview output.

      Use Filter fields to find a field. Use the sort menu to sort fields by name from A to Z or Z to A, move selected fields to the top, or show hidden fields. Select or clear field check boxes to show or hide fields in the preview table.

   7. (Optional) Turn on SPL editor, the Search Processing Language (SPL) editor, to view or edit the landing definition as Search Processing Language version 2 (SPL2).

      After you edit the Search Processing Language version 2 definition, run preview again to confirm the results before you continue.

   8. Confirm that the preview results match the events that you want to land.

      Select Preview, or press Control+Enter, to run the preview again. If preview results are empty or unexpected, revise the sample data, partitions, filters, masking, or actions, and run the preview again.

9. Select Next.
10. On the Review page, review the dataset configuration.

    Verify the dataset name, data source, dataset description, retention period, promotion limits, and roles with query access.

    • If the configuration is correct, select Create Dataset.
    • If you need to change the configuration, select Back and update the settings.
    The raw table appears on the Datasets page. Data that matches the landing definition can now land in Machine Data Lake.