Set up automated updates for a Splunk-native DDSS data catalog in AWS

Create an SQS queue and set up event notification for the Amazon S3 bucket that contains your DDSS dataset, to keep your Splunk-managed data catalog in sync with dataset changes.

1. In a new browser tab, log in to your AWS account and navigate to the Simple Queue Service (SQS) console.
2. On the SQS dashboard, select Create queue.
   1. In the Details section, select the Standard queue type.
      The standard queue type provides high throughput for catalog updates.
   2. In the Name field, provide a name for your SQS queue. This name must begin with the DDSS resource prefix that you can copy from the Configure dataset page in the Splunk Data Management app. Paste in that prefix, and then add to it an SQS queue name of your choice.
      For example, if the DDSS resource prefix provided on the Configure dataset page is "fa2-nnn-aname-ddss-2-abcdefghijkl-", you might enter fa2-nnn-aname-ddss-2-abcdefghijkl-my-SQL-queue into the Name field.
   3. Scroll to the bottom of the page and select Create queue.
3. In the Details section, find the ARN field. This is the SQS queue ARN. Select the copy icon to copy it.
4. Go back to the browser tab that has your federated search dataset definition and paste the ARN value into the SQS queue ARN field.
5. In the browser tab that has your SQS queue information, scroll down to the Access policy section and select Edit.
6. In the Access policy editor, replace the default access policy with the following template:
   JSONCopy

   { "Version": "2012-10-17", "Id": "example-ID", "Statement": [ { "Sid": "example-statement-ID", "Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": "sqs:SendMessage", "Resource": "YOUR_SQS_QUEUE_ARN", "Condition": { "ArnLike": { "aws:SourceArn": "arn:aws:s3:::YOUR_S3_BUCKET_NAME" } } } ] }

   {
     "Version": "2012-10-17",
     "Id": "example-ID",
     "Statement": [
       {
         "Sid": "example-statement-ID",
         "Effect": "Allow",
         "Principal": {
           "Service": "s3.amazonaws.com"
         },
         "Action": "sqs:SendMessage",
         "Resource": "YOUR_SQS_QUEUE_ARN",
         "Condition": {
           "ArnLike": {
             "aws:SourceArn": "arn:aws:s3:::YOUR_S3_BUCKET_NAME"
           }
         }
       }
     ]
   }

7. Replace YOUR_SQS_QUEUE_ARN with the ARN for the SQS queue.
8. Replace YOUR_S3_BUCKET_NAME with the name of the Amazon S3 bucket that contains the DDSS location for your dataset.
9. Select Save to save your SQS queue access policy changes.

10. In the browser tab you opened for your AWS account, navigate to the Amazon S3 console.
11. From the General purpose buckets list, select the name of the S3 bucket that contains the Amazon S3 location for your dataset.
12. Select the Properties tab.
13. Scroll down to the Event notifications section and select Create event notification.
14. In the General configuration section, supply an Event name, such as SplunkS3DiscoveryEvent.
15. Ensure that the Prefix corresponds to the Index path that appears on the Define dataset page. See Define a DDSS dataset.
    For the Prefix field, enter everything in the Index path after the bucket name. For example, if the Index path for your dataset is s3://bucket1/path1/archived_data/ddss_index1, set the Prefix field to path1/archived_data/ddss_index1.
16. In the Event types section, select the following checkboxes:
    1. Select All object create events to notify your Splunk-managed data catalog when new data is uploaded to your dataset.
    2. Select All object removal events to notify your Splunk-managed data catalog when data is deleted or a delete marker is created.
    3. Select All restore object events to notify your Splunk-managed data catalog when objects are restored from the Glacier Deep Archive or Glacier Flexible Retrieval storage classes.
    4. Select Lifecycle transition events and All lifecycle expiration events to capture automatic promotion and expiration of dataset objects based on lifecycle rules.
17. In the Destination section, select SQS queue as the destination type.
18. Under Specify SQS queue, choose Enter SQS queue ARN.
19. Paste the SQS queue ARN you copied in the previous section into the SQS queue text field.
20. Select Save changes.