Create input for custom source types for Amazon S3

Create input for custom source types in Amazon S3, set up necessary AWS resources and configuring data ingestion.

You can ingest data from custom source types for Amazon s3. This feature is available for single AWS accounts only.

  • In order to setup S3 input, you need to have SQS and S3 resources with logs data.

  • Ask your AWS admin to create the SplunkDMReadOnly IAM role in your AWS account.

    This role lets Splunk Cloud read the configurations from the various AWS services that data is collected from. Configure the SplunkDMReadOnly IAM role with the trust relationship and policy that you can copy from Data Manager. Make sure that the AWS administrator replaces the account identifiers in the policy. If you have already created this role, verify that its role policy matches the role policy that you can copy from Data Manager interface.

  • (Optional) Create an onboarding user.

    If you do not have the AWS ability to take action on AWS resources, ask your AWS admin to create the onboarding user in the AWS account. This user allows you to take actions on resources, such as creating CloudFormation stacks and listing S3 buckets. Configure the onboarding user with the IAM User policy that you can copy from Data Manager. Make sure that the AWS administrator replaces the account identifiers in the policy.

  • Prepare data sources for ingestion.

    Set up the data source that you have selected to send its logs to S3 buckets. See the AWS documentation for more information on how CloudTrail, S3 buckets, load balancers or CloudFront can be configured to send their logs to S3 buckets.

  • You have edited the props.conf and transforms.con configuration files. For more information, see Edit the configuration files.

  1. In Data Manager, select New Data Input.
  2. Select Amazon Web Services as the source you want to onboard.
  3. Select Next.
  4. On the AWS Data Onboarding page, select Amazon S3 and then select Next.
  5. On the Prerequisites for Onboarding AWS S3 Data page, check if all prerequisites are met. If not, ask you AWS admin to prepare the prerequisites.
  6. Select Next.
  7. On the Input AWS S3 Data Information page, enter the following information:
    1. The name of your data input. The name must be unique.
    2. From the drop-down list, select an IAM Roles Region.
    3. In the Source Type field, enter your custom source type.
    4. Enter a list of SQS queue URLs across your AWS regions.
    5. Enter a prefix for your S3 buckets or enter a list of S3 bucket ARNs to limit Splunk Cloud read access.
    6. Enter an Amazon Resource Name (ARN) for each AWS KMS key. AWS KMS keys are required for users with an encryption key setup.
    7. Select the destination repository from the Destination list.
      If you entered all required data correctly, the Review Data Input button is available.
  8. Select Review Data Input.
  9. Review information you entered. If everything is correct, select Next. If you want to make changed, select Close and make necessary changes.
    The Setup Data Ingestion page opens. The data source is added to the list in Data Manager.
  10. Decide if you want to establish resources on your AWS account now or later:
    • If you want to establish resources on your AWS account now, follow the steps in Establish resources on your AWS account
    • If you want to finish the setup later, go to the Data Management tab where you can see the ingest inputs with their statuses and details.
The data input is added to the table on the Data Manager page. You can check its status. To finish the setup, select the input name and select the AWS Setup Details tab. If you did this in the Setup Data Ingestion page, no action is needed.
Establish resources on your AWS account.