Define a DDSS dataset

Define a Dynamic Data Self Storage (DDSS) dataset in the Data Management app to facilitate federated search of data stored in a specific DDSS location in AWS.

Define a Dynamic Data Self Storage (DDSS) dataset in the Data Management app for use in federated searches. Each DDSS dataset you define lets you run federated searches over data stored in a specific DDSS location in an Amazon S3 bucket without having to reindex it first.

Note: Federated Search for DDSS currently does not support creation of datasets for DDSS locations in Azure or GCP environments.

  • You must have a Splunk Cloud Platform (SCP) deployment with dynamic data self storage (DDSS) locations configured in Amazon S3 buckets. See Store expired Splunk Cloud Platform data in your private archive in the Splunk Cloud Platform Admin Manual.

  • Your user account on the SCP deployment must have a role with the edit_connections and edit_datasets capabilities. See Define roles on the Splunk platform with capabilities in the Splunk Cloud Platform Manage Users and Security manual.

  • You must have an AWS account with sufficient permissions to manage the Amazon S3 buckets that serve as locations for your DDSS datasets and apply policies or permissions to them. You also must have permissions that allow you to create and manage SQS queues for those Amazon S3 buckets.

Note: Federated Search for DDSS does not support search of DDSS data that is archived in the S3 Glacier Deep Archive or S3 Glacier Flexible Retrieval storage classes. If the DDSS location you want to search contains data that is stored in either of these storage classes you must restore it to a searchable storage class before you can search it. For more information, see Restoring an archived object in the Amazon Simple Storage Service User Guide.
  1. In Splunk Cloud Platform, select Data Management from the Apps panel.
  2. Navigate to the Datasets page, and then select Create dataset.
  3. On the Select data store page, select Dynamic Data Self Storage(DDSS), then select Next.
  4. On the Define dataset page, provide values for the following fields and select Next:
    Setting Description
    S3 bucket name Select the name of the Amazon S3 bucket that contains the DDSS dataset.
    DDSS index Select the index in your Splunk Cloud Platform deployment that your DDSS dataset is associated with. This will be an index that has been configured for Dynamic Data Self-Storage and is set up to move expired data to the location at the S3 bucket path you have provided.
    Note: This field is unavailable until you enter an S3 bucket path that is associated with one or more DDSS indexes in your Splunk Cloud Platform deployment.

    If no indexes appear in the DDSS index dropdown, select Refresh indexes to update the index list.

    Each DDSS index can be associated with only one DDSS dataset. DDSS indexes that already have a DDSS dataset association won't appear in the DDSS index dropdown.

    After you supply the S3 bucket name and DDSS index, Splunk software displays an Index path for your DDSS dataset. Save this path. You will need it when you set up event notifications for the Amazon S3 bucket in Step 6.

    For more information about configuring DDSS indexes, see Store expired Splunk Cloud Platform data in your private archive in the Splunk Cloud Platform Admin Manual.
    Dataset name Supply a unique name for your dataset. The dataset name can contain only alphanumeric characters, underscores, and hyphens.
    Dataset description (Optional) Provide a description for your dataset.
  5. On the Configure dataset page, select the copy icon to copy the provided DDSS resource prefix to your clipboard. You will need this DDSS resource prefix in the following step.

    Here is an example of a DDSS resource prefix: fa2-nnn-aname-ddss-2-abcdefghijkl-.

  6. To arrange for the Splunk-native data catalog that backs your DDSS dataset to be updated automatically so it is consistent with the dataset it represents, set up an SQS queue and event notification for the Amazon S3 bucket that contains your DDSS dataset. Do this by following the instructions in Set up automated updates for a Splunk-native DDSS data catalog in AWS.
    Note: When you create the SQS queue, you must give it a Name that begins with the DDSS resource prefix you copied in Step 5.

    After you set up the SQS queue, you'll obtain an Amazon Resource Name (ARN) that you can paste into the SQS queue ARN field.

  7. (Optional) Activate Knowledge object discovery at search time to run federated searches over your DDSS dataset that include knowledge objects such as extracted fields, field aliases, calculated fields, event types, and tags.

    Knowledge object discovery is based on the inferred source types in your DDSS dataset. When your dataset is created, you can see the source types that the crawler service inferred for your dataset, and then use those source types in your searches. Go to the Datasets listing page, select the dataset, and note the source types listed under Sourcetype in the dataset detail sidebar.

    For more information, see Activate knowledge object discovery for federated searches of DDSS datasets.

    Note: When Knowledge object discovery at search time is activated for a DDSS dataset, federated searches of that dataset might take longer to complete.
  8. Select Next.
  9. On the Update policies page, you'll apply two generated policies to your AWS account.
    1. AWS S3 Bucket Policy

      • Copy the AWS S3 Bucket policy from your dataset definition.

      • In a new browser tab, open the AWS Management Console and navigate to the Amazon S3 console.

      • Select your bucket, go to the Permissions tab, and and append the copied bucket policy to the existing Bucket policy.

        Note: Do not overwrite existing policies. Resolve security warnings, errors, and suggestions before saving your changes.

        For additional guidance, see Adding a bucket policy by using the Amazon S3 console in the Amazon Simple Storage Service User Guide.

    2. AWS SQS Queue Policy

      • Copy the AWS SQS Queue Policy from your dataset definition.

      • In the AWS console, open the Simple Queue Service console.

      • Select the SQS queue you defined for the Amazon S3 bucket that holds your DDSS data. Select Edit and append the copied SQS queue policy to the existing Access policy.

        Note: Do not overwrite existing policies. Resolve security warnings, errors, and suggestions before saving your changes.

        For additional guidance, see Configuring an access policy in Amazon SQS in the Amazon Simple Queue Service Developer Guide.

  10. When both policies are pasted and saved to their appropriate locations in your AWS account, select Next.
  11. On the Review page, review your dataset definition. If the details appear correct, select Create Dataset to create your dataset.

Your DDSS dataset is in the process of being created. This process might take a few minutes to complete.

On the Datasets listing page you can see the Status of your DDSS dataset, and you can use that status value to guide your next actions regarding it.

Status Description Action
Ready The dataset is available for use in federated searches.
Processing The crawler process is running over the dataset.

Selection of Create dataset on the Review step causes the crawler process to initiate schema discovery for the dataset.

The crawler process might take a few minutes to complete.

Note: If more than 10 minutes pass and the crawler process is still in Processing status, a dataset setup error might be causing it to fail to complete. Review the current dataset configuration for errors such as an incorrect location path. Then delete the dataset that is stuck in Processing status and try to recreate it without errors.
Needs action The schema discovered by the crawler requires review and confirmation.

Go to the Edit page for your DDSS dataset. Review the schema that the crawler process has discovered, make edits to it as necessary, and confirm that you have inspected it. See Review the crawler-discovered schema for a DDSS dataset.
Error An error occurred during dataset creation or processing. Review the configuration for your DDSS dataset, correct issues, and recreate the dataset if necessary.