Migrate legacy Amazon S3 destinations to datasets for Ingest Processor pipelines

Migrate legacy destinations in the Ingest Processor service to datasets managed in the Data Management app so you can run federated searches on the data and continue updating configurations.

Starting with Splunk Cloud Platform version 10.4.2604, you can send data from Ingest Processor to Amazon S3 using the connections and datasets that are managed in the Data Management app instead of the legacy destinations that are managed in the Ingest Processor service.

With the Data Management app, each Amazon S3 pipeline destination is now configured through a connection and a dataset:

  • The connection contains the configuration settings for authenticating to your Amazon Web Services (AWS) account.

  • The dataset contains the configuration settings for specifying the path to the S3 bucket and the format of the output data.

Each dataset uses a connection to access Amazon S3 and work with the data in the specified bucket. The same connection can be used by multiple datasets.

You can use these datasets as both pipeline destinations and data sources for federated searches.

To make full use of this expanded functionality and ensure that you can continue to configure your Amazon S3 destinations, replace your legacy destinations with datasets at your earliest convenience.

Note: Existing Amazon S3 destinations will continue to work with incoming data as expected, so these updates will not disrupt data ingestion or processing. However, support for creating and editing the configuration settings of legacy Amazon S3 destinations is being deprecated.

Be aware of the following changes to support for legacy Amazon S3 pipeline destinations:

Splunk Cloud Platform version Changes to Amazon S3 destination support
10.4.2604

  • To differentiate between the two kinds of Amazon S3 pipeline destinations, the Ingest Processor service associates legacy destinations with the Amazon S3 (routing only) kind and associates updated datasets with the Amazon S3 kind.

  • You can no longer create Amazon S3 (routing only) destinations. To define a new Amazon S3 location to send data to, you must create an Amazon S3 dataset using the Data Management app. For more information, see Send data from Ingest Processor to Amazon S3.
10.5.x You can no longer edit the configuration settings of Amazon S3 (routing only) destinations.

To migrate Amazon S3 (routing only) destinations to Amazon S3 datasets, complete the following steps.

Your Splunk Cloud Platform deployment must be on version 10.4.2604 or higher.

  1. In the Ingest Processor service, navigate to the Destinations page.
  2. Select an Amazon S3 (routing only) destination to open the details panel. From this panel, select Migrate.
  3. On the Migrate <destination name> page, select Data Management app.
  4. In the Data Management app, do the following:
    1. Create an Amazon S3 connection that accesses the same Amazon Web Services (AWS) account and region as your Amazon S3 (routing only) destination.

      For more information, see the following pages:

    2. Create an Amazon S3 dataset that specifies the same S3 bucket and output event schema as your Amazon S3 (routing only) destination.

      For more information, see Create an Amazon S3 dataset for Ingest Processor pipelines.

      The following table describes how to set the relevant configuration options in order to keep the target S3 bucket and output event schema consistent.

      Option in the Amazon S3 dataset Set to the following value
      Amazon S3 location A value using the format s3://<bucket>/<path>, where <bucket> is the Bucket name setting from the legacy destination and <path> is the Folder name setting from the legacy destination.
      Output schema The value that is enclosed in parentheses in the Output data format setting from the legacy destination.
      Output format The value that is outside the parentheses in the Output data format setting from the legacy destination.
      Compression type The Compression type setting from the legacy destination.
      File name prefix The File prefix setting from the legacy destination.
  5. Return to the Migrate <destination name> page in the Ingest Processor service and do the following:
    1. Review the list of pipelines that are using the legacy destination.
    2. For each pipeline that's listed, open it for editing, change the destination to the new Amazon S3 dataset, and then save your changes.
  6. Confirm that no applied pipelines are using the legacy destination, and then select Delete to delete the Amazon S3 (routing only) destination.

Your legacy Amazon S3 destination is replaced with an Amazon S3 dataset that additionally supports federated searches and will remain configurable in upcoming Splunk Cloud Platform releases.