Using source types to break and merge data in Edge Processors

The source type is one of the default fields that Splunk software assigns to events. It identifies the kind of data that you are working with and indicates the original source of the data.

You can use source type configurations to specify how your Edge Processors break and merge the inbound stream of data into distinct events. The event breaking and merging operations defined in your source type configurations are applied to inbound data if it meets the following criteria:

  • The sourcetype value of an event matches the name of a source type configuration.
  • The inbound data isn't already event-broken through other means, such as by the EVENT_BREAKER configuration in a universal forwarder.

By default, the Edge Processor solution includes event breaking and merging configurations for a variety of common source types. See Automatically recognized source types in the Splunk Cloud Platform Getting Data In manual for a list of default source types.

If the source type that you want to work with is not listed, you must configure that source type in the Splunk Cloud Platform that's connected to your Edge Processor tenant and then sync the source type to bring it into the Edge Processor service. You can also edit the default source types to meet your needs.

Note: This is step 3 of 6 for using an Edge Processor to process data and route it to a destination. To see an overview of all of the steps, see

Quick start: Process and route data using Edge Processors

.

This diagram shows an overview of the steps required to set up and use an Edge Processor.

Manage source types that were originally created in the Edge Processor service

Before the ability to sync source types from Splunk Cloud Platform became available, source types for Edge Processors had to be created and managed manually in the Edge Processor service. To update the source types that were created using this manual approach, you can do either of the following:

  • Edit the source type configuration from the Source types page in the Edge Processor service. For more information, see Edit or delete source types for Edge Processors.

  • Overwrite the source type with one that is synced from Splunk Cloud Platform. For more information, see the instructions in the next section.

Sync source types from Splunk Cloud Platform

Create and manage source types in the Splunk Cloud Platform deployment that's connected to the Edge Processor tenant, and then sync the source types so that the same configurations are available for Edge Processors.

Before you can sync the source types, you must create them in the Splunk Cloud Platform deployment that’s connected with your Edge Processor tenant. For information on creating and configuring source types in Splunk Cloud Platform, see Manage source types in the Splunk Cloud Platform Getting Data In manual.

Depending on whether you are adding a new source type to the Edge Processor tenant or overwriting a manually managed source type, you’ll need to configure the source type in Splunk Cloud Platform to meet specific requirements:

ScenarioConfiguration requirements

Adding a new source type to the Edge Processor tenant

  • The source type in Splunk Cloud Platform does not have the same name as a manually managed source type in the Edge Processor service.

  • The source type only has these line breaking options specified:

    • LINE_BREAKER

    • SHOULD_LINE_MERGE

    • BREAK_ONLY_BEFORE

    • MAX_EVENTS

    • TRUNCATE

    All other line breaking options are unsupported, and must be unspecified or set to their default values. The default values will not affect the line breaking behavior in the Edge Processor. For more information about these line breaking options and their default values, see props.conf in the Splunk Enterprise Admin Manual.

Overwriting a manually managed source type in the Edge Processor tenant

  • The source type in Splunk Cloud Platform has the same name as the source type in the Edge Processor tenant.

  • The source type in Splunk Cloud Platform has the exact same line breaking settings as the source type in the Edge Processor tenant, and does not include any other additional line breaking settings.

If a source type in Splunk Cloud Platform has the same name but a different configuration than a manually managed source type in the Edge Processor service, then the source type configuration in the Edge Processor service is retained and the one in Splunk Cloud Platform is not synced over.

If a source type in Splunk Cloud Platform specifies non-default values for any line breaking options other than LINE_BREAKER, SHOULD_LINE_MERGE, BREAK_ONLY_BEFORE, MAX_EVENTS, or TRUNCATE, then the source type is not synced over.

  1. In the Edge Processor service, navigate to the Source types page.
  2. Select Sync source types.
    Note: The maximum number of source types that can be synced from Splunk Cloud Platform is 1000. If this limit is exceeded, you will receive the error message Number of allowed source types exceeded.
    It can take a few minutes for all the source types to be synced. When the process is completed, the Edge Processor service returns a message indicating the number of source types that were synced successfully.
The synced source types are now listed on the Source types page in the Edge Processor service, and you can use them in your pipelines. For information about creating pipelines and applying them to Edge Processors, see Create pipelines for Edge Processors and Apply pipelines to Edge Processors.