Create an Amazon S3 connection for Ingest Processor pipelines

Create an Amazon S3 connection in the Data Management app to allow your pipelines to authenticate and send data to Amazon S3 buckets.

Before you can send data from Ingest Processor to an Amazon S3 bucket, you must first create an Amazon S3 connection in the Data Management app on Splunk Cloud Platform. This connection allows datasets and pipelines to access S3 buckets by authenticating through an IAM role or an access key pair.

  • Your Splunk Cloud Platform deployment must be on version 10.4.2604 or higher.

    Note: If your Splunk Cloud Platform deployment does not meet this requirement, see Create a legacy Amazon S3 destination for Ingest Processor.
  • Your user account on the Splunk Cloud Platform deployment must have the edit_datasets and admin_all_objects capabilities. For more information, see the following pages:
  • You have an Amazon Web Services (AWS) account and an AWS IAM role with permissions that let you attach and modify custom trust policies and permissions policies for IAM roles. Contact your AWS administrator for assistance with AWS permissions.

  1. In Splunk Cloud Platform, select Data Management from the Apps panel.
  2. Navigate to the Connections page, and then select Create connection.
  3. On the Select data store page, select Amazon S3, then select Next.
  4. On the General page, configure the following options, and then select Next:
    Option name Configuration instructions
    Connection name Enter a unique name for your connection.
    Connection description (Optional) Enter a description for your connection.
    AWS account ID

    Enter the 12-digit ID of the AWS account that has the S3 bucket you want to send data to.

    AWS region Select the region of your AWS account.
    Note: Avoid egress costs by using an AWS account that is in the same region as your Splunk Cloud Platform deployment. You are responsible for any egress charges incurred when sending data from a pipeline to Amazon S3.
  5. On the Select abilities page, do the following:
    1. Select Send data from Ingest Processor.
      You can optionally select additional abilities:
      • If you also want to use this connection to send data from Edge Processor pipelines to S3 buckets, then select Send data from Edge Processor.

      • If you also want to use this connection to run federated searches over S3 buckets, then select Run federated search.

    2. Select the authentication method you want to use for each ability that you selected during the previous step.
      As a best practice, for the Send data from Ingest Processor ability, select IAM role. IAM role authentication is more convenient and secure because it does not require you to store and rotate sensitive credentials such as a secret access key.
    3. Select Next to continue to the Storage authentication page.

The remaining connection creation steps vary depending on which authentication method you selected for the Send data from Ingest Processor ability during step 5. To finish creating this Amazon S3 connection, complete the instructions on the following pages as necessary: