Troubleshoot the Splunk SOAR Broker Manager

Handling potential issues with the Broker Manager

Note: This section is for Modern Automation Brokers only. For additional troubleshooting topics, see Troubleshooting the Classic Splunk SOAR Automation Broker.

Use these sequential troubleshooting steps from the Automation Broker/Broker Manager host, in the directory that contains your Broker Manager compose.yml file and splunk_data folder.

If you are not comfortable running these commands, collect the requested files and command output with help from your system administrator.

Refer to the Quick symptom guide, located later in this article, for specific symptoms and steps to check first.

Step 1: Confirm you are in the correct directory

Run the following command.

CODE 
ls

Expected results:

You see the following items listed:

  • compose.yml file

  • splunk_data folder.

If either item is missing, change into the directory where you saved the Broker Manager compose file and try again.

Step 2: Check which containers are running

Run the following command for Docker or Podman:

CODE 
docker ps -a --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'
CODE 
podman ps -a --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'

Expected result:

  • A Broker Manager container is listed.

  • An Automation Broker container is listed after the Broker Manager starts it. The name usually looks like <broker-manager-name>_ab_1.

  • The containers show a status of running or starting. Investigate the situation if the status is stopped, exited, unhealthy, or repeatedly restarting.

Step 3: Check Broker Manager and Automation Broker logs

Broker Manager logs

Find the Broker Manager container name from Step 2: Check which containers are running.

Run the following command

CODE 
docker logs <broker-manager-container-name>

Also check the persistent Broker Manager log:

CODE 
cat splunk_data/ab_1/log/broker_manager.log

Look for messages about:

  • Missing or invalid SOAR_BASE_URL

  • Missing Docker or Podman socket

  • Missing /splunk_data mount

  • Failed version lookup from Splunk SOAR

  • Image pull failures

  • upgrade_failed

Automation Broker logs

Get the Automation Broker container name from Step 2: Check which containers are running.

Then run the following command:

CODE 
docker logs <automation-broker-container-name>

Also check the following Automation Broker logs:

CODE 
cat splunk_data/ab_1/log/brokerd.log
cat splunk_data/ab_1/log/soar_brokerd_stderr.log
cat splunk_data/ab_1/log/soar_brokerd_stdout.log

Look for messages about:

  • Pairing or credential errors

  • AMQP connectivity errors

  • Proxy or TLS/certificate errors

  • Network timeouts reaching Splunk SOAR

Step 4: Check initial pairing state

Before initial pairing of the Automation Broker with your Splunk SOAR instance, place broker_registration.key at the root of splunk_data:

CODE 
ls -l splunk_data

During startup, the Broker Manager stages that key into the Automation Broker data folder. After Automation Broker pairs successfully, the key is consumed and removed.

After the Automation Broker is paired, a missing broker_registration.key is expected.

  • Do not recreate or replace the key for an already-paired broker.

  • If the broker has never paired and the key is missing, download or copy the broker registration material from Splunk SOAR again and restart the Broker Manager.

Step 5: Check if an upgrade is blocked

Run the following command:

CODE 
ls -l splunk_data/upgrade_failed

If the upgrade_failed file exists, Broker Manager previously tried an upgrade, the new container did not start in a healthy way, and the Broker Manager rolled back. The Broker Manager skips future upgrades while this upgrade_failed file exists.

Review the logs and address any issues. If you determine there are no other issues, remove the splunk_data/upgrade_failed file and try again. Removing the file without fixing the underlying issue can cause the same failed upgrade to repeat.

Step 6: Check image registry access

If you find log entries that mention an image pull or registry error, find the image and tag information in the logs or in the compose file

Use this information to run the appropriate Docker or Podman command:

CODE 
docker pull <image>:<tag>
CODE 
podman pull <image>:<tag>

If the pull fails:

  • Confirm the host has network access to the registry.

  • Confirm the image name and tag are correct.

  • If the registry requires authentication, confirm the host is logged in.

  • If your environment cannot access the registry directly, ask your administrator or Splunk Support about loading the image locally.

Quick symptom guide

Symptom What to check first
Broker Manager exits immediately Step 3 Logs, SOAR_BASE_URL, runtime socket, and /splunk_data mount
Automation Broker does not appear Step 2 Container list and Step 3 Broker Manager logs.
Broker does not appear in Splunk SOAR Step 3 Automation Broker logs, AMQP/network access, proxy settings, and SOAR broker status.
Upgrade does not proceed Step 5 upgrade_failed marker and Broker Manager logs.
Registry pull fails Step 6 Registry access and image tag.

Contacting Splunk Support

If you must open a support case, include the following information:

  • Splunk SOAR stack URL

  • Sanitized Broker Manager compose file

  • Container runtime and version

  • Output from docker ps or equivalent Podman command

  • Broker Manager logs from docker logs and splunk_data/ab_1/log/broker_manager.log

  • Automation Broker logs from docker logs, if the Automation Broker container exists

  • Whether a proxy, custom certificate, or private image registry is used

  • Whether splunk_data/upgrade_failed exists