Migrate legacy Amazon S3 destinations to datasets for Ingest Processor pipelines

Migrate legacy destinations in the Ingest Processor service to datasets managed in the Data Management app so you can run federated searches on the data and continue updating configurations.

Starting with Splunk Cloud Platform version 10.4.2604, you can send data from Ingest Processor to Amazon S3 using the connections and datasets that are managed in Splunk Cloud Platform instead of the legacy destinations that are managed in the Ingest Processor service. You can use these datasets as both pipeline destinations and data sources for federated searches.

Additionally, support for creating legacy Amazon S3 destinations and editing their configuration settings is being gradually removed in Splunk Cloud Platform version 10.4.2604 and higher. The following table summarizes the changes to support for legacy Amazon S3 pipeline destinations:

Splunk Cloud Platform version Changes to Amazon S3 destination support
10.4.2604 and higher

  • To differentiate between the two kinds of Amazon S3 pipeline destinations, the Ingest Processor service associates legacy destinations with the Amazon S3 (routing only) kind and associates updated datasets with the Amazon S3 kind.

  • You can no longer create Amazon S3 (routing only) destinations. To define a new Amazon S3 location to send data to, you must create an Amazon S3 dataset in Splunk Cloud Platform. For more information, see Send data from Ingest Processor to Amazon S3.
10.5.2605 and higher You can no longer edit the configuration settings of Amazon S3 (routing only) destinations.
Note: Existing Amazon S3 destinations will continue to work with incoming data as expected. These product updates do not disrupt ongoing data ingestion or processing.

To make use of the expanded support for federated searches and ensure that you can continue to configure your Amazon S3 destinations, upgrade to Splunk Cloud Platform version 10.6 or higher and then replace your Amazon S3 (routing only) destinations with Amazon S3 datasets.

However, this migration is subject to the following limitations:

  • Existing data stored in Amazon S3 remains unchanged, but new incoming data sent to the Amazon S3 dataset uses a different schema.

  • Storing data with inconsistent schemas in the same Amazon S3 bucket can prevent data retrieval workflows such as federated searches from working as expected, so you must configure your Amazon S3 dataset to send data to a different bucket than your existing Amazon S3 (routing only) destination.

  • Consequently, because your data is divided between different buckets, the scope of your federated searches is limited to the data in the bucket that's associated with the Amazon S3 dataset.

To replace Amazon S3 (routing only) destinations with Amazon S3 datasets in your pipelines, complete the steps that follow.

Each Amazon S3 dataset uses a connection to access Amazon S3 and work with the data in the specified bucket:

  • The connection contains the configuration settings for authenticating to your Amazon Web Services (AWS) account.

  • The dataset contains the configuration settings for specifying the path to the S3 bucket and the format of the output data.

The same connection can be used by multiple datasets.

Your Splunk Cloud Platform deployment must be on version 10.6 or higher.

  1. In the Ingest Processor service, navigate to the Destinations page.
  2. Select an Amazon S3 (routing only) destination to open the details panel. From this panel, select Migrate.
  3. On the Migrate <destination name> page, select Data Management app.
  4. In the Data Management app, do the following:
    1. Create an Amazon S3 connection that accesses the same Amazon Web Services (AWS) account and region as your Amazon S3 (routing only) destination.

      For more information, see the following pages:

    2. Create an Amazon S3 dataset that specifies a different S3 bucket than your Amazon S3 (routing only) destination.

      For more information, see Create an Amazon S3 dataset for Ingest Processor pipelines.

      The following table describes how to set the specific configuration options in order to ensure that the data is supported by federated searches, while also minimizing the differences between the data that was previously ingested by your Amazon S3 (routing only) destination and the incoming data that will be ingested by this Amazon S3 dataset.

      Option in the Amazon S3 dataset Set to the following value
      Amazon S3 location A value using the format s3://<bucket>/<path>, where <bucket> is the name of a different bucket than the one specified in the Bucket name setting in the legacy destination, and <path> is the name or path of a folder in the bucket where you want to store your data.
      Output schema Select Pipeline output.
      Output format The value that is outside the parentheses in the Output data format setting from the legacy destination.
      Compression type The Compression type setting from the legacy destination.
      File name prefix The File prefix setting from the legacy destination.
      Partition schema

      This option is available in Splunk Cloud Platform version 10.6 only.

      Select Time partition path.
      Partition schema

      Select Time partition path.
  5. Return to the Migrate <destination name> page in the Ingest Processor service and do the following:
    1. Review the list of pipelines that are using the legacy destination.
    2. For each pipeline that's listed, open it for editing, change the destination to the new Amazon S3 dataset, and then save your changes.
  6. Confirm that no applied pipelines are using the legacy destination, and then select Delete to delete the Amazon S3 (routing only) destination.

Your legacy Amazon S3 destination is replaced with an Amazon S3 dataset that additionally supports federated searches and will remain configurable in upcoming Splunk Cloud Platform releases.