How the destination for Edge Processor works

In order to send data from an Edge Processor to a storage location such as an index or an Amazon S3 bucket, you must define the location as a destination in the Edge Processor service. Each destination contains the connection information necessary for allowing an Edge Processor to send data to a given location.

The steps for adding a destination to the Edge Processor service varies depending on whether the destination is part of a Splunk Cloud Platform deployment or a Splunk Enterprise deployment:

You can confirm the destinations that are available by checking the Destinations page, and view additional details about a given destination by selecting it.

Persistent queues

Edge Processors currently provide no data delivery guarantees. The Edge Processor writes each event to a disk-backed queue for its destination before sending the event, including when the destination is available. The queue drains continuously during normal operation. If the Edge Processor cannot send data to a destination or receives more data than it can send, events accumulate in the queue and can remain there for longer periods. If the queue fills before the destination becomes available again, the Edge Processor applies backpressure and continues to attempt to add data to the queue until it restarts or shuts down. If the Edge Processor instance shuts down or restarts while data is being sent, data cannot be written to a persistent queue, which can cause data loss. After the Edge Processor sends an event to its destination, the event might remain in queue storage until the Edge Processor reclaims that space.

Edge Processor stores queue data on its host disk. In Kubernetes deployments, each Edge Processor instance stores its queue on its persistent volume. By default, the queue can hold up to 5 gigabytes of data. For information about changing the queue size, see Edit the size of your persistent queues. To create an alert for when your persistent queue uses up a certain amount of data, see Set up alerts for Edge Processor metrics.

Edge Processor does not provide application-level encryption at rest for persistent queue data. You are responsible for protecting the host and storage that hold this data, including controlling access and enabling encryption at the storage layer. Depending on your deployment, use filesystem encryption, encrypted disks or volumes (including self-encrypting drives), encrypted Kubernetes persistent volumes, or encryption provided by your cloud storage platform.

If you process personally identifiable information (PII) or other sensitive or regulated data, mask or encrypt sensitive fields before they reach the persistent queue. Do not process unmasked sensitive or regulated data through Edge Processor unless appropriate customer-managed encryption and access controls are in place. Encryption can add CPU, memory, and disk I/O overhead, which might affect throughput, latency, disk utilization, or queue-drain time depending on your workload and storage configuration.

If your pipeline uses either the branch or route command and one of the queues for your destination is full, then data may be delivered more than once for the other healthy destinations, causing data duplication.

Once the destination is available again, the Edge Processor sends queued events to the destination before processing newer data. As a result, newer data might take longer to be processed.

Edit the size of your persistent queues

Change the "Storage size" setting in the Edge Processor configuration options to adjust the persistent queue size.

You can change the size of the persistent queues used by an Edge Processor by configuring the Storage size option. This setting determines the queue size for all instances that are part of that Edge Processor.
  1. Navigate to the Edge Processors page.
  2. In the row that lists the Edge Processor that you want to modify, select the Actions icon (Image of the Actions icon) and then select Edit Edge Processor.
  3. In the Persistent queue settings section, set the Storage size field to the desired queue size in gigabytes.
  4. Select Save.