Deploy an Edge Processor pipeline

Create a destination, Edge Processor resource, and pipeline, then apply the pipeline to a registered Edge Processor instance.

Complete the steps in the Configure the Data Management API topic. Use an account with data_management_admin access for normal automation. Store credentials in a secret manager.

Set these common variables before you begin:

CODE 
export SPLUNK_MGMT_URL='https://splunk.example.com:8089'
export API_USER='dmx-automation'
export API_PASSWORD='replace-with-a-secret'
export API_BASE="$SPLUNK_MGMT_URL/servicesNS/$API_USER/dmx_cmp/v1/data"
  1. Create a destination.

    This example creates a non-TLS Splunk-to-Splunk (INDEXER) destination. Use TLS or mTLS and reachable endpoints in production. PEM material that you provide when you create or update a destination is write-only and is never returned by the API.

    JSON 
    curl --fail --user "$API_USER:$API_PASSWORD" \
  --request POST --header 'Content-Type: application/json' \
  --data '{
    "name": "edge_to_indexers",
    "kind": "INDEXER",
    "description": "Default destination for the production Edge Processor",
    "parameters": {
      "endpoints": ["indexer-a.example.com:9997", "indexer-b.example.com:9997"],
      "compressionEnabled": true,
      "tls": {"tlsOption": "NON_SECURE"}
    }
  }' \
  "$API_BASE/destinations" > destination.json

export DESTINATION_ID="$(jq -r '.id' destination.json)"
test -n "$DESTINATION_ID" && test "$DESTINATION_ID" != null

    Supported destination kinds are INDEXER, HECEXPORTER, and AWSS3. For existing compatible destinations, list them with GET $API_BASE/destinations?runtime=EDGE&count=0&offset=0 and reuse the returned id.

  2. Create the Edge Processor resource.

    The following request enables S2S and HEC input, turns off syslog, and makes the new destination the fallback for data that no applied pipeline handles.

    JSON 
    curl --fail --user "$API_USER:$API_PASSWORD" \
  --request POST --header 'Content-Type: application/json' \
  --data "{
    \"name\": \"branch-east-ep\",
    \"description\": \"Edge Processor for the east branch\",
    \"sourceConfigs\": {
      \"s2s\": {\"tlsOption\": \"NON_SECURE\"},
      \"hec\": {\"enabled\": true, \"tlsOption\": \"NON_SECURE\"},
      \"syslog\": {\"enabled\": false, \"tlsOption\": \"NON_SECURE\"}
    },
    \"defaultDestinationId\": \"$DESTINATION_ID\"
  }" \
  "$API_BASE/edge-processors" > edge-processor.json

export PROCESSOR_ID="$(jq -r '.id' edge-processor.json)"
test -n "$PROCESSOR_ID" && test "$PROCESSOR_ID" != null

    For TLS or mTLS sources, add the required server certificate and key as PEM contents. Add a CA certificate for mTLS. Do not use the illustrated NON_SECURE settings for traffic that crosses an untrusted network.

  3. Install and register an Edge Processor instance.

    Request the onboarding script, copy it to the target Linux host through a secure channel, review it, and run it under the intended service account.

    CODE 
    curl --fail --user "$API_USER:$API_PASSWORD" \
  "$API_BASE/edge-processors/$PROCESSOR_ID/onboard?enableCertificateValidation=true" \
  --output onboard-edge-processor.sh

chmod 700 onboard-edge-processor.sh
# Copy to the target host securely, then run it there:
# bash onboard-edge-processor.sh
  4. Wait for the instance to register before you deploy a pipeline.
    CODE 
    curl --fail --user "$API_USER:$API_PASSWORD" \
  "$API_BASE/edge-processors/$PROCESSOR_ID/instances?count=0&offset=0"

    The returned items array must show a healthy, running instance. If you need to reverse onboarding, request the matching /offboard script from the same Edge Processor resource and run it on that host.

  5. Create a pipeline and bind the destination.

    The pipelineBody field contains SPL2. This minimal example passes events from the source to the bound destination. Adjust the source filters and SPL2 for your data transformation.

    PYTHON 
    export PIPELINE_NAME='branch-east-pass-through'

curl --fail --user "$API_USER:$API_PASSWORD" \
  --request POST --header 'Content-Type: application/json' \
  --data "{
    \"name\": \"$PIPELINE_NAME\",
    \"runtime\": \"EDGE\",
    \"description\": \"Route selected branch events to the indexer destination\",
    \"pipelineBody\": \"\u0024pipeline = | from \u0024source | into \u0024destination;\",
    \"sourceFilters\": [
      {\"key\": \"sourcetype\", \"operator\": \"IN\", \"values\": [\"access_combined\"]}
    ],
    \"parameters\": [
      {\"name\": \"destination\", \"type\": \"DESTINATION\", \"destinationId\": \"$DESTINATION_ID\"}
    ]
  }" \
  "$API_BASE/pipelines" > pipeline.json
  6. Apply and verify the pipeline.

    Applying an Edge Processor pipeline is asynchronous. A 202 Accepted response indicates that the request was accepted; it does not confirm deployment. Poll the status until it reports APPLIED.

    JSON 
    curl --fail --user "$API_USER:$API_PASSWORD" \
  --request POST --header 'Content-Type: application/json' \
  --data "{\"processorIds\": [\"$PROCESSOR_ID\"]}" \
  "$API_BASE/pipelines/$PIPELINE_NAME/apply"

curl --fail --user "$API_USER:$API_PASSWORD" \
  "$API_BASE/pipelines/$PIPELINE_NAME/status"

    While the status is APPLYING, retry the status request. Completion is APPLIED, and the response should include the requested processor ID. Before you send production data, investigate API errors, Edge Processor host logs, input connectivity, and destination reachability.