Send data from Edge Processors to non-connected Splunk platform deployments using S2S

When sending data from an Edge Processor to a Splunk Enterprise deployment or a Splunk Cloud Platform deployment that is not connected to your tenant, you can choose to send that data using the Splunk-to-Splunk (S2S) protocol. S2S is the proprietary, TCP-based data transmission protocol used between Splunk software.

Start by adding a Splunk platform S2S destination in the Edge Processor service. You can configure the destination to send data to one or more indexers that are part of the same Splunk platform deployment. Then, create a pipeline that uses that destination. When you apply that pipeline to your Edge Processor, the Edge Processor starts sending the data that it receives to your Splunk platform deployment.

The specific index that the data from an Edge Processor gets routed to is determined by a precedence order of configurations. For more information, see Index precedence order when using S2S.

Note: You can also send data using the HTTP Event Collector (HEC) instead of S2S, or send data to the Splunk Cloud Platform deployment that is connected to your tenant without needing to add any destinations. For more information, see Sending data from Edge Processors to Splunk Cloud Platform or Splunk Enterprise.

Prerequisites

Before you can add a destination that sends data to the Splunk platform using S2S, you must do the following:

  • For each indexer that you want to send data to, make note of the IP address or host name of the indexer as well as the port number that it uses to receive data. You will need to provide this information when configuring the Splunk platform S2S destination.

    If you plan to send data to many indexers, prepare a list of the indexers. You can then copy-paste this list or upload a file that contains the list when configuring the destination, instead of entering the information for each indexer manually.

    The list must meet these requirements:

    • Each indexer is specified using the format <ip_address>:<port> or <hostname>:<port>
    • The indexers in the list are separated by a comma ( , ) or a new line
    • If you want to upload the list from a file, the file format must be .txt or .csv
    Note: You might be able to retrieve this list of indexers from your Splunk platform configuration files. For example, if your inputs.conf file contains a server setting that lists the relevant indexers, you can copy the value of that server setting and save it in a .txt file.
  • If you're sending data to an indexer that has the requireClientCert property set to true in the inputs.conf file, that means the indexer uses mutually authenticated TLS (mTLS) for S2S connections and requires connecting clients to authenticate themselves using TLS certificates. In this case, you must obtain the certificates for proving the Edge Processor's identity. See Obtaining TLS certificates in this topic for more information.
    Note: Splunk Cloud Platform indexers always require mTLS for S2S connections.

Configure your Edge Processor to use acknowledgement

Prerequisites

  • Collect the tenant details that Edge Processor support requires.
  • Identify every Edge Processor instance in the deployment.
  • Identify each upstream forwarding client and the applicable outputs.conf file.
  • Identify each S2S destination for which you want to enable ACK.
  • Plan a restart for each Edge Processor process, and upstream forwarding client that you change.
  • Confirm that you can search the _metrics index for events with the edge-metrics source type.
Note: Configure useACK in a [tcpout] or [tcpout:<target_group>] stanza. You cannot configure it for an individual server in a [tcpout-server:<server>] stanza.

Steps

To achieve at-least-once delivery, enable ACK across the complete data path.

  1. Reach out to pipelineprocessing@splunk.com, and ask to enable S2S ACK for the tenant. Confirm that the per-destination control is available in the Data Management UI.
  2. Apply the configuration that was provided by the pipeline processing team to every Edge Processor instance:
    1. For a standalone deployment, set ENABLE_S2S_ACK=true in the Edge Processor process environment or the systemd unit configuration.
    2. For a hybrid deployment, set ENABLE_S2S_ACK=true in the Splunk launch configuration specified by support.
  3. Restart the Edge Processor process for a standalone deployment or the Splunk service for a hybrid deployment.
  4. On each upstream forwarding client, set useACK=true in the applicable [tcpout] or [tcpout:<target_group>] stanza of outputs.conf.
  5. Restart the forwarding client to apply the outputs.conf change.
  6. In the Data Management UI, select Destinations.
  7. Select the S2S destination that you want to configure.
  8. Turn on ACK for the destination.
  9. Repeat steps 8 and 9 for every S2S destination that requires ACK.
  10. Send representative data through the complete path and verify the acknowledgment metrics.
CAUTION: Indexer acknowledgment can increase memory use for outbound queues and can produce duplicate events when a sender retransmits unacknowledged data. Plan capacity and downstream duplicate handling accordingly.

Disable E2E S2S ACK

Disable the feature in the reverse order to return to non-ACK connections.

  1. In the Data Management UI, select Destinations.
  2. Select each S2S destination that uses ACK and turn off ACK.
  3. On each upstream forwarding client, set useACK=false or remove the useACK setting from the applicable outputs.conf stanza.
  4. Restart each forwarding client that you changed.
  5. On every Edge Processor instance, set ENABLE_S2S_ACK=false or remove the setting:
    1. For a standalone deployment, update the Edge Processor process environment or the systemd unit configuration.
    2. For a hybrid deployment, update the Splunk launch configuration specified by support.
  6. Restart the Edge Processor process for a standalone deployment or the Splunk service for a hybrid deployment.
  7. Send representative data and confirm that the non-ACK connections operate normally.

Verify E2E S2S ACK

  1. Send representative data from an upstream forwarding client through Edge Processor to the S2S destination.
  2. Search the _metrics index for the edge-metrics source type.
  3. Review the incoming acknowledgment metrics:
    • ingress_ack_sent_total: Confirm that the value increases while data flows.
    • ingress_ack_failed_total: Confirm that the value remains at zero.
    • ingress_ack_channel_timeout_total: Confirm that the value remains at zero. A nonzero or increasing value can indicate acknowledgment backpressure.
  4. Review the outgoing acknowledgment metrics:
    • egress_ack_pending_batches: Confirm that the value does not increase continuously. Continuous growth can indicate a destination issue.
    • egress_ack_latency_seconds: Monitor the acknowledgment round-trip time for unexpected increases.
    • egress_ack_dropped_total: Confirm that the value remains at zero.
  5. Compare queue_size_bytes with queue_max_bytes to monitor the on-disk queue fill level.
  6. If the queue approaches capacity, investigate destination availability and acknowledgment latency before data exhausts the available queue space.

Obtaining TLS certificates

If you're sending data to an indexer that uses mTLS, then you need to have TLS certificates that the Edge Processor can use to prove its identity to the indexer. You must upload these certificates when configuring the destination in the Edge Processor service.

TLS requirements when connecting to Splunk Cloud Platform

Your Edge Processor must prove its identity using the TLS certificates provided in the universal forwarder credentials package. You can download this package from your Splunk Cloud Platform deployment by doing the following:

  1. In the Splunk Web interface for your Splunk Cloud Platform deployment, select Apps, then Universal Forwarder.
  2. Select Download Universal Forwarder Credentials.

Note the location of the credentials file. The credentials file is named splunkclouduf.spl.

TLS requirements when connecting to Splunk Enterprise

Your Edge Processor must prove its identity using the following TLS certificates, contained in separate Privacy Enhanced Mail (PEM) files:

  • A client certificate.
  • The private key associated with that client certificate. This private key must be decrypted.
  • The CA certificates used to verify the indexer.

If you don't have these PEM files, ask your Splunk Enterprise administrator for assistance. See the Secure Splunk platform communications with Transport Layer Security certificates chapter of the Securing Splunk Enterprise manual for more information.

Add a Splunk platform S2S destination

  1. In the Edge Processor service, select Destinations.
  2. On the Destinations page, select New destination, then Splunk platform using S2S.
  3. Provide a name and description for your destination.
    Field Description
    Name A unique name for your destination
    Description (Optional) A description of your destination
  4. In the Indexers area, specify one or more indexers that you want to send data to. Do any of the following:
    Action Instructions
    Specify one indexer In the input field, enter the host and port information of the indexer using the format <ip_address>:<port> or <hostname>:<port>.
    Specify an additional indexer

    1. Select Add an indexer.

    2. In the additional input field that appears, enter the host and port information for the indexer using the format <ip_address>:<port> or <hostname>:<port>.
    Specify a list of indexers

    1. Select Edit indexer list.

    2. In the Edit indexer list dialog box, enter or upload a list of indexers.

      The indexers in the list can be separated by a comma ( , ) or a new line. If you want to upload a list that is saved in a file, make sure that the file format is .txt or .csv.

      Note: Be aware that when you upload a file, the list from the file overwrites any indexers that you specified previously.

    3. Review the validation messages displayed near the bottom of the Edit indexer list dialog box and confirm that the number of valid indexers matches your expectations.

    4. Select Save valid indexers.
  5. To compress your data before sending it from an Edge Processor to this destination, turn on Data compression.

    Note: Compressing your data reduces the amount of bandwidth that is used to send data to the destination, but also causes the Edge Processor to consume more computing resources.
  6. If your indexer requires mTLS, then do the following:
    1. Select Authenticate identity using TLS certificates.
    2. Set Platform to the type of Splunk platform deployment you want to send data to.
    3. Provide the necessary certificates:
      • If you set Platform to Splunk Cloud Platform, then in the Universal forwarder credentials field, upload the splunkclouduf.spl file that you downloaded from your Splunk Cloud Platform deployment.
      • If you set Platform to Splunk Enterprise, then upload the appropriate private key and certificates in these fields:
        Field Description
        Client private key A PEM file containing the decrypted private key associated with your client certificate
        Client certificate A PEM file containing a client certificate
        CA certificates The CA certificates used to verify the indexer
  7. To finish adding the destination, select Add.

You now have a destination that you can use to send data from an Edge Processor to one or more Splunk indexers.

To start sending data from an Edge Processor to the indexers specified in the destination, create a pipeline that uses the destination you just added and then apply that pipeline to your Edge Processor. For more information, see Create pipelines for Edge Processors and Apply pipelines to Edge Processors.