Edit the configuration files

Edit configuration files to define index-time operations for custom source types in Splunk, ensuring correct data indexing and parsing.

Before creating an input with a custom source type, you must edit the configuration files to define any required index-time operations, such as extractions or transformations. This is necessary because once data begins to flow into Splunk, index-time properties cannot be modified for already-ingested events, and changes made later will not apply retroactively. By configuring index-time extractions in advance using the appropriate settings for source type, source, or host, you ensure that your data is indexed and parsed correctly from the start. You can edit the configuration files in your local folder before creating the input to ensure these operations are applied properly.

Edit the following configuration files as needed:

  • props.conf: Set indexing property configurations, including timezone offset, custom source type rules, and pattern collision priorities. Also, map transforms to event properties.

    For more information about this file, see props.conf .

  • transforms.conf: configure regex transformations to perform on data inputs. Use this file in tandem with props.conf.

    For more information about this file, see transforms.conf.

  • tags.conf: configure tags for fields.

    For more information about this file, see tags.conf.

  • eventtypes.conf: create event type definitions.

    For more information about this file, see eventtypes.conf.

This example demonstrates how to dynamically reassign a sourcetype and set a new search-time field based on the content of an event.

Let's assume that you want to ingest data from your custom source type named aws:s3:web:logs.

During event processing, you want to achieve the following:
  • Dynamically transform the sourcetype based on the value of the loglevel field.

    • Events with the error log level are assigned the sourcetype aws:s3:web:error:logs.

    • Events with the info log level are assigned the sourcetype aws:s3:web:info:logs.

  • Set a search-time field log_severity according to the transformed sourcetype:

    • log_severity should be "high" for error logs

    • log_severity should be "normal" for info logs

To achieve this, you need to edit both the props.conf and transforms.conf files .
Add the following stanzas in the props.conf file:
[aws:s3:web:logs]
TRANSFORMS-set_sourcetype = set_error_sourcetype, set_info_sourcetype

[aws:s3:web:error:logs]
EVAL-log_severity = "high"

[aws:s3:web:info:logs]
EVAL-log_severity = "normal"
Add the following stanzas in the transforms.conf file:
[set_error_sourcetype]
REGEX = loglevel=(error|ERROR)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::aws:s3:web:error:logs

[set_info_sourcetype]
REGEX = loglevel=(info|INFO)
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::aws:s3:web:info:logs
Create input for custom source types for Amazon S3