About Federated Search for DDSS

Federated Search for DDSS lets you run federated searches from your Splunk Cloud Platform deployment over Dynamic Data Self Storage (DDSS) datasets that you are storing in Amazon S3 locations.

Federated Search for DDSS lets you run federated searches from your Splunk Cloud Platform deployment over Dynamic Data Self Storage (DDSS) datasets that you are storing in Amazon S3 locations. When you run these federated searches, you use familiar SPL2 search commands and syntax.

Note: Federated Search for DDSS currently does not support federated search of DDSS datasets that are located in Azure or GCP environments.

DDSS datasets

Federated Search for DDSS is part of the Data Management app, where you'll set up your federated search experience by defining DDSS datasets that represent your configured DDSS locations in AWS.

A DDSS dataset is a searchable data object. In your definition of the DDSS dataset, the DDSS location is identified by the combination of the S3 bucket path for the location and the name of a DDSS index that is associated with that bucket path.

Splunk-native data catalog generation

Federated Search for DDSS searches apply filtering and statistical functions to data catalogs that contain column, schema, and partition definitions for DDSS datasets. This means that a data catalog must be associated with each DDSS dataset you intend to search.

Federated Search for DDSS builds a Splunk-native data catalog for each dataset you define. Splunk software automatically infers the dataset schema and partitions with a crawler.

You must set up an SQS queue and event notification in your AWS account to keep this catalog in sync with your dataset as your dataset changes over time. For more information, see Define a DDSS dataset.

What you need to get started

To get started with federated search of data you store in dynamic data self storage (DDSS) locations, you must have the following things:

  • You must have an Splunk Cloud Platform (SCP) deployment with DDSS locations configured in Amazon S3 buckets. See Store expired Splunk Cloud Platform data in your private archive in the Splunk Cloud Platform Admin Manual.

  • Your user account on the SCP deployment must have a role with the edit_connections and edit_datasets capabilities. See Define roles on the Splunk platform with capabilities in the Splunk Cloud Platform Manage Users and Security manual.

  • You must have an AWS account with sufficient permissions to manage the Amazon S3 buckets that serve as locations for your DDSS datasets and apply policies or permissions to them. You also must have permissions that allow you to create and manage SQS queues for those Amazon S3 buckets.

Checklist of tasks to set up Federated Search for DDSS

The following checklist guides you through the cross-account setup of Federated Search for DDSS.

Step Task Description
1 Define a DDSS dataset

  • Provide baseline information for your dataset, including the Amazon S3 bucket name and DDSS index.

  • Ensure the Splunk-native data catalog that backs up your dataset is updated as data is added, removed, or restored to the dataset. Set up an SQS queue and event notification for the Amazon S3 bucket that contains the DDSS dataset.

  • To facilitate dataset access, copy generated S3 bucket and SQS queue policies and append them to corresponding policies in your AWS account.
2 Give your users role-based access control of federated datasets After you have successfully created a DDSS dataset, give your users role-based access to it.
3 Write and run federated searches over federated datasets with SPL2 Run federated searches over your new DDSS dataset with SPL2.