Troubleshoot the Modern Splunk SOAR Automation Broker

Troubleshoot Podman restart loops caused by a stale common PID file

What you see

The Automation Broker starts, appears healthy until the configured systemd startup-timeout interval expires, receives SIGTERM, and might later be killed with SIGKILL. systemd then starts it again. The journal typically repeats at that interval:

container-soar_ab_example.service: Can't open PID file .../<old-container-id>/userdata/conmon.pid
container-soar_ab_example.service: start operation timed out. Terminating.
container-soar_ab_example.service: State 'stop-sigterm' timed out. Killing.
container-soar_ab_example.service: Failed with result 'timeout'.

The application log might only show that Automation Broker received SIGTERM; systemd is the component sending it.

Source:Red Hat systemd timeout documentation, systemd.service(5)

Why it happens

The Automation Broker container was removed and recreated with the same explicit name but a new container ID. Its legacy generated systemd unit still combines:

• ExecStart= that finds the replacement container by name.

• PIDFile= that contains the previous container's ID.

Podman successfully starts the replacement, but systemd waits for a PID file that can no longer exist. When the startup timeout expires, systemd terminates and restarts the service. Increasing the timeout does not fix the stale path.

Sources: Podman run --name documentation,

Confirm the issue

Run these commands as the rootless user that owns Automation Broker (sa_soar in this example):

CONTAINER="soar_ab_example"
SERVICE="container-soar_ab_example.service"
export XDG_RUNTIME_DIR="/run/user/$(id -u)"
export DBUS_SESSION_BUS_ADDRESS="unix:path=$XDG_RUNTIME_DIR/bus"
podman inspect --format \
  'ID={{.Id}}
ConmonPidFile={{.ConmonPidFile}}' \
  "$CONTAINER"
systemctl --user show "$SERVICE" \
  -p PIDFile \
  -p ExecStart \
  -p TimeoutStartUSec

If systemctl --user reports Failed to connect to bus, inspect the unit file directly:

sed -n '1,240p' \
  "$HOME/.config/systemd/user/$SERVICE"

The issue is confirmed when the unit's PIDFile= contains a different container ID from Podman's ConmonPidFile, while ExecStart= starts the current container by name. The journal shows the matching PID-file and startup-timeout errors:

journalctl --user -u "$SERVICE" \
  --since "15 minutes ago" \
  --no-pager

Sources: Podman inspect documentation, systemd.service(5) PID-file behavior

Fix the stale unit

If an installer or configuration-management system owns the service, use its repair procedure. Otherwise, regenerate the legacy unit for the current container. Run these commands as sa_soar:

REPAIR_STARTED_AT=$(date --iso-8601=seconds)
UNIT="$HOME/.config/systemd/user/$SERVICE"
WORKDIR=$(mktemp -d)
cp -a "$UNIT" \
  "${UNIT}.bak.$(date +%Y%m%d%H%M%S)"
systemctl --user stop "$SERVICE"
(
  cd "$WORKDIR"
  podman generate systemd \
    --name \
    --files \
    "$CONTAINER"
)
install -D -m 0644 \
  "$WORKDIR/$SERVICE" \
  "$UNIT"
systemctl --user daemon-reload
systemctl --user enable --now "$SERVICE"

This replaces the stale PIDFile= with the path for the current container. Do not try to solve this by increasing TimeoutStartSec; systemd would only wait longer for the wrong file. The Quadlet procedure in the next section is the preferred long-term configuration because it does not preserve a generated unit tied to an old container ID.

If systemctl --user still cannot connect to the bus, restore access to the sa_soar systemd user manager before applying the repair; changing the file alone does not reload the running manager.

Sources: Podman-generated systemd unit documentation, Red Hat systemd unit reload procedure

Verify

systemctl --user status "$SERVICE" --no-pager -l
printf 'systemd: %s\n' \
  "$(systemctl --user show "$SERVICE" -p PIDFile --value)"
printf 'Podman:  %s\n' \
  "$(podman inspect --format '{{.ConmonPidFile}}' "$CONTAINER")"

The service should remain active (running) beyond the configured TimeoutStartUSec interval shown during confirmation, and both PID-file paths should identify the current container. After that interval has passed, check the journal once for new errors:

journalctl --user -u "$SERVICE" \
  --since "$REPAIR_STARTED_AT" \
  --no-pager

The repair is complete if there are no new Can't open PID file or start operation timed out messages. Investigate SELinux denials or application errors separately if they remain; they do not explain a confirmed stale PID-file mismatch.