Monitor Containers with Docker Visibility

Use the Machine Agent to monitor application nodes running inside Docker containers and to identify container issues that impact application performance. By viewing and comparing APM metrics with the underlying container and server/machine metrics, you can easily answer the question: Is my application problem purely an application problem, or is the root cause in the container or the server?

Note: Container monitoring requires a Server Visibility license >=4.3.3, for both the Controller Tenant and the Machine Agent.

You should deploy the Machine Agent inside a Docker container. The Machine Agent collects metrics for Docker containers on the same host, and server and machine metrics for the host itself. The Controller Tenant shows all monitored containers, for each host, and the container and host IDs, for each container.

In a BRIDGE networking mode, the containers take on the container ID as the hostname. If networking is in host mode, then the containers take on the node name of the host ID. This means every container on that node has the same host ID. In this case, you need to use the unique host ID settings. See Register the Container ID as the Host ID. When using Docker Visibility, if the unique host ID setting is not configured to use container ID in host network mode, the Machine Agent automatically registers the container using the container ID as the host ID. If you have an older version of the Controller Tenant or Machine Agent, we recommend that you upgrade to Machine Agent version 20.7 or later. See Using Docker Visibility with Kubernetes.

With Controller Tenant >=20.11.0:

  • If the Machine Agent >=20.7.0, the Machine Agent automatically registers the container using the container ID as the host ID. No further action is needed.
  • If the Machine Agent <=20.6.0 is configured incorrectly, the Controller Tenant rejects the misconfigured container registration.

By default, the Machine Agent only monitors containers that have a running APM Agent. You can change this by setting the sim.docker.monitorAPMContainersOnly property on the Controller Tenant. See Controller Settings for Server Visibility.

Note: To deploy a Machine Agent on a host outside a Docker container, create a symbolic link: ln -s / /hostroot on the host. This symbolic link enables the Machine Agent to collect host metrics with Docker container metrics. When you deploy a Machine Agent inside a Docker container for monitoring, the symbolic link is automatically created when the volume mounts. To grant more restrictive permissions, enter this command to create symbolic links: ln -s /proc /hostroot/proc; ln -s /sys /hostroot/sys; ln -s /etc /hostroot/etc. You can make these links read-only because the Splunk AppDynamics Agent does not need write privileges to these directories.

This diagram illustrates how to deploy container monitoring:

Container Monitoring

  • Install the Machine Agent in a standalone container. The Machine Agent collects hardware metrics for each monitored container, as well as Machine and Server metrics for the host , and forwards the metrics to the Controller Tenant.
  • The Machine Agent can monitor all containers that are running on that host, subject to established limits, and will report runtime metrics and metadata for every container. Additionally, if any of the containers have an APM Agent installed , then the Machine Agent also correlates the container metadata and runtime metrics with the associated APM Node.