Event metadata

When sending data to an Edge Processor through a HEC endpoint, make sure to specify a sourcetype value. If an event is not associated with a source type, then Edge Processor pipelines cannot select that event for processing.

Additionally, as a best practice for reducing inaccuracies in the Edge Processor metrics, make sure that you specify a host value. The Inbound data sources metric displayed in the Edge Processor service is based on the number of distinct host values detected in the inbound events.

If you are sending data through the services/collector endpoint, you must specify the sourcetype and host values in the body of the HTTP request. However, if you are sending data through the services/collector/raw endpoint, then you must specify the sourcetype and host values in the Uniform Resource Indicator (URI) of the HTTP request.

URI format

The HTTP request must be directed to a URI written in this format:

<protocol>://<host>:<port>/<endpoint>?<parameter>

The variables are defined as follows:

• <protocol> is either http or https.
• <host> is the host name of a specific Edge Processor instance or a load balancer.
• <port> is the number of the port that the Edge Processor instance uses to receive HEC requests.
  Note: This port number is a shared configuration setting that is specified in the Edge Processor service. See Configure shared Edge Processor settings for more information.
• <endpoint> is the HEC endpoint that you want to send the request to.
• <parameter> is one or more query string parameters that specify metadata for the event. For example, if you include the parameter sourcetype=manual, then the events that you send using this URI will have a sourcetype field containing the value manual.
  • The services/collector endpoint in Edge Processors does not support query string parameters.
  • The services/collector/raw endpoint in Edge Processors supports these parameters: host, index, source, sourcetype, and time. For more information about these parameters, see services/collector/raw in the Splunk Enterprise REST API Reference Manual.
  • To include multiple parameters, separate each parameter with an ampersand ( & ).
  • If you don't include any parameters, omit the question mark ( ? ) from the URI.

Query string parameters are typically optional, but they are mandatory when you send data to the services/collector/raw endpoint of an Edge Processor. You must use query string parameters to specify sourcetype and host values for your events.

Authorization header

When you send data to an Edge Processor using HEC, you can turn on token authentication to secure your HEC receivers in Edge Processor by using HEC tokens. This is optional. If token authentication is activated, then data sources can only send data to the Edge Processor if the HTTP request header includes a matching HEC token. To use token authentication, do the following:

• In your Edge Processor shared settings, add your HEC token. See Configure HEC token authentication in the Edge Processor service for more information.
• Insert the token you added in the shared settings into the authorization header of your HTTP request. See Get HTTP request examples with your Edge Processor information for more information.

The authorization header appears after the URI in the HTTP request in this format:

If your Edge Processor receives data that is not associated with a HEC token, and it then processes and sends that data to the Splunk platform using HEC, the Edge Processor uses the default HEC token specified in the pipeline destination. For more information about configuring a pipeline destination that uses HEC, see Send data from Edge Processors to non-connected Splunk platform deployments using HEC.

As a best practice for reducing the amount of configuration overrides in your HEC data ingestion setup, when token authentication is off in the Edge Processor settings, omit the Authorization header and HEC token from your inbound HTTP requests. Omitting the token during data ingestion allows you to rely on the pipeline destination configuration to specify the HEC token to be used when sending data out to a destination.

Edge Processor acknowledgement

When you send data to an Edge Processor using HEC, you can use the Edge Processor acknowledgement feature to verify that the Edge Processor received the data. This is optional. If you want to use Edge Processor acknowledgement, you need to add a channel ID header to your HTTP request. See About channels and sending data in the Splunk Enterprise Getting Data In manual for more information.

The channel ID header in your HTTP request needs to be in this format. Insert your channel ID in the <insert_channel_ID> placeholder:

-H "X-Splunk-Request-Channel: <insert_channel_ID>"

When you send the request, it returns an ackID value that you can use to confirm that the Edge Processor received your data. Each channel will have an ackID value unique to that specific channel only. The Edge Processor stores a maximum of 1,000,000 ackID values in memory that are not queried. Once the ackID value is queried, it will be removed. Once you exceed this limit, the Edge Processor removes the oldest ackID value. After receiving the ackID value, confirm that the Edge Processor received your data by sending a second HTTP request to the /services/collector/ack endpoint.

Send an HTTP request using the following format:

curl --location 'host_123ghnskfpi:10088/services/collector/ack?channel=<insert_channel_ID>'  -H 'Content-Type: application/json'  -d '{"acks":[<insert_ack_ID>]}'

Replace the following placeholders:

• <insert_channel_ID> with your channel ID.
• <insert_ack_ID> with the ackID value that you received from the first HTTP request.

This second request will return a value of either true or false depending on if the data was received by the Edge Processor. If the request returns a value of true, the Edge Processor has received your data and is written on the queue. If the request returns a value of false, the Edge Processor did not process and write your data on the queue or the ackID has already been removed due to reaching the maximum amount of ackID values in memory.

To see HTTP request examples with Edge Processor acknowledgment, see Get HTTP request examples with your Edge Processor information for more information.

Request body format for the services/collector endpoint

When you send data to the services/collector endpoint, the body of the HTTP request must meet the following requirements:

• The request body is a JSON object.
• The JSON object must include the following:
  • An event key that contains the data being sent by your HTTP client or logging agent.
  • A sourcetype key that contains a value for the sourcetype metadata field in the event.
  • A host key that contains a value for the host metadata field in the event.

For example, the following is a basic HTTP request that sends Hello, world! to an Edge Processor instance through the services/collector endpoint, sets the sourcetype key to manual, and sets the host key to host_1.splunk.com:

curl "http://buttercupgames.com:8088/services/collector" \
-d '{"event": "Hello, world!", "sourcetype": "manual", "host": "host_1.splunk.com"}'

The following is the same example with an authorization header:

curl "http://buttercupgames.com:8088/services/collector" -H "Authorization: Splunk 12345678-1234-1234-1234-1234567890AB" \
-d '{"event": "Hello, world!", "sourcetype": "manual", "host": "host_1.splunk.com"}'

Request body format for the services/collector/raw endpoint

When you send data to the services/collector/raw endpoint, the body of the HTTP request must be a text string that contains the data being sent by your HTTP client or logging agent.

For example, the following is a basic HTTP request that sends Hello, world! to an Edge Processor instance through the services/collector/raw endpoint. This request uses query string parameters in the URI to specify required metadata such as sourcetype and host.

curl "http://buttercupgames.com:8088/services/collector/raw?sourcetype=manual&host=host_1.splunk.com" \
-d 'Hello, world!'

The following is the same example with an authorization header:

curl "http://buttercupgames.com:8088/services/collector/raw?sourcetype=manual&host=host_1.splunk.com" -H "Authorization: Splunk 12345678-1234-1234-1234-1234567890AB" \
-d 'Hello, world!'

The services/collector/raw endpoint interprets the request body as a raw text string even if the string contains formatting syntax. For example, the following request sends {"event": "Hello, world!", "sourcetype": "manual", "host": "host_1.splunk.com"} instead of Hello, world! to the Edge Processor instance.

curl "http://buttercupgames.com:8088/services/collector/raw?sourcetype=manual&host=host_1.splunk.com" \
-d '{"event": "Hello, world!", "sourcetype": "manual", "host": "host_1.splunk.com"}'

TLS and mTLS requirements

If your Edge Processor is configured to use TLS or mTLS to secure its communications with HEC data sources, then the HTTP requests must include PEM files containing the appropriate client certificate, CA certificate, and private key for proving the identity of the data source. Depending on how your HTTP client or logging agent makes requests, there can be a variety of different ways to include these PEM files.

For example, when using a curl command to make the HTTP request, you can use the --cert, --key, and --cacert flags to specify the PEM files:

curl --cert myClientCert.pem --key myPrivateKey.pem --cacert myCACert.pem --location "https://buttercupgames:8088/services/collector" -H "Authorization: Splunk 12345678-1234-1234-1234-1234567890AB"
\ -d '{"event": "Hello, world!", "sourcetype": "manual", "host": "host_1.splunk.com"}'