Ingest historical data from AWS S3

Bring historical data from object storage, such as Amazon S3, directly into Splunk Cloud Platform for investigation, compliance, or long-term analysis.

When you need to investigate security incidents, conduct compliance reviews, or analyze long-term trends, you often require access to large amounts of historical data. Instead of running repeated remote searches, which can be slow and use up a lot of Data Storage Units, you can now bring historical data directly into Splunk Cloud Platform from object storage like Amazon S3.

With this feature, you can do the following:
  • Select exactly which data you want to ingest from any Amazon S3 bucket.
  • Choose specific time ranges and set predefined source types.
  • Use flexible S3 path formats to target only the data that matters for your task.

Once you've set up your promote job, the data is ingested into a dedicated Splunk index created just for historical data. This index has infinite retention, so your data will stay available for as long as you need it.

You also have control over the ingestion process, you can pause, resume, cancel, or delete jobs as needed. Each job runs once and can't be rerun or edited after it finishes, which helps keep your data management simple and reliable.

Practical applications for ingesting historical data

Sometimes, you need to work with data that isn't currently available in your Splunk instance, whether for security, investigation, or compliance reasons. The historical data ingestion feature lets you bring in older logs from sources like AWS S3 buckets, making it easier to get the information you need. Here are some common situations where this can be useful:

  • Threat detection

    If a new security vulnerability or threat is discovered, you might want to check if your organization was impacted in the past. But what if Splunk only has recent data? With this feature, a security analyst can work with a Splunk admin to pull in older logs (such as AWS CloudTrail or web server logs) from S3 and run detection rules to look for signs of exposure. This also applies if you want to check historical data for indicators of compromise that weren't known at the time.

  • Threat investigation and hunting

    Security investigations sometimes require looking further back in time than your current Splunk data allows. For example, if you're investigating a security alert but need logs from a month that's not in Splunk, you can ingest just the logs you need, like firewall or VPN logs from specific dates, so you can complete your analysis and uncover important details or patterns.

  • Audit and compliance

    Audits and compliance checks sometimes call for historical log data that's missing from your Splunk instance. With this feature, you can bring in the necessary logs from earlier time periods, helping you meet audit requirements and generate accurate reports.

What you can expect from ingesting historical data

  • Each promote job runs once, ingesting all specified historical data. After completion, the job cannot be rerun or edited, ensuring data integrity and operational simplicity.

  • Choosing any past time range for your promote data, allows you to tailor data ingestion to the needs of your investigation or review.

  • Ingesting historical data from Amazon S3 buckets provides direct access to your cloud storage data.

  • Customizable paths lets you define exactly what data you need by specifying predefined source types and flexible S3 path formats.

  • Monitoring the progress of your promote jobs.

Limitations and considerations

Before using the promote feature, keep the following considerations in mind:

  • Each promote input can only be used once. If you need to ingest more data or make changes, you must create a new promote.

  • Only one promote job can run at a time. Attempting to run multiple concurrent promote jobs may result in duplicated events.

  • Once a promote job has started (including scheduled promote jobs), it cannot be modified. Only before submission can you edit the setup.

  • A promote currently supports data ingestion only from Amazon S3 buckets.

  • Promote is currently limited to the pre-populated AWS source types available in the user interface. Custom source types are not supported at this time.

  • By default, promote inputs uses a promote index with infinite retention in the Splunk Cloud platform.

  • There is no explicit maximum data size, but ingested data consumes Splunk SVCs and Ingest Volume License. To learn more about SVCs, see Monitor current SVC usage of your workload-based subscription.

  • This feature is available for all Splunk Cloud AWS customers, you don't have to purchase additional Extended Cloud Storage SKUs. If you don't have an access to this feature, contact your account team.