Server Visibility Requirements and Supported Environments

To enable Server Visibility, review these system requirements guidelines.

System Requirements

Server Visibility requires a supported version of the Machine Agent to be installed on the monitored server. See Machine Agent Requirements and Supported Environments.

Server Visibility features are available for AIX, Linux, Windows, and Solaris. You need a Server Visibility license to enable and use Server Visibility features. Server Visibility is enabled by default on Splunk AppDynamics Controllers. You must explicitly enable Server Visibility on your Machine Agent to start sending the expanded set of Server Visibility metrics.

You can install and run Machine Agents on other supported platforms, however, Server Visibility features are not available. For OS platforms like AIX or HP-UX, we recommend using the unbundled Machine Agent ZIP without the JRE.

Limitations

  • Total number of Machine Agents and containers reporting to a single Controller is 30,000.
  • Total number of App Agents reporting to a single Controller is 100,000.
  • Monitored servers are considered stale 30 days after they go offline. They are purged from the Controller database when that time limit is reached.
  • Docker Visibility is supported on Linux only.

Server Visibility on Windows

Review these notes:

  • It is good practice to check the server frequently to ensure that it has the latest Windows updates installed. WMI (Windows Management Instrumentation) updates are especially important because the Agent uses WMI to collect Server Visibility metrics.
  • The monitored server should have at least four cores or multiple CPUs. WMI processes can be highly resource-intensive, especially on Windows Server 2012.
  • If you do not have a Server Visibility license, or to collect Basic metrics only, configure the Agent to use the JavaHardwareMonitor extension. Server Visibility should not be enabled on the Agent.
  • The Windows Machine Agent uses these extensions by default:
    • JavaHardwareMonitor (for Basic metrics)
    • ServerMonitoring (for Server Visibility metrics)
  • The HardwareMonitor extension is not recommended on Windows.
  • To enable Server Visibility on a Windows server with a .NET APM Agent installed, you must enable .NET Compatibility Mode on both the Controller and the Machine Agent.

Controller Settings for Server Visibility

This page describes Controller Admin settings that are specific to Server Visibility. You can apply these Controller settings and permissions from either the AccountorControllertabs. Changes to the settings take effect the next time the Agent is restarted and connects to the Controller.
Attention: You need the root user password to change these settings.
You cannot set all configuration properties at the account-level. Check the Description column for each property to determine whether you can enable a specific Server Visibility property. See Roles and Permissions and Manage Controller Users and Groups.

To change the Controller Admin settings for Machine Agents, see Controller Settings for Machine Agents.

Server Visibility PropertyDescriptionDefault
sim.docker.enabled EnableDocker Visibility. SeeMonitor Containers with Docker Visibility. true
sim.docker.machine.container.limit The global limit for the number of containers to monitor per machine. You can specify this in the2023-10-19_05-04-05_Access the Administration Consoleas a Controller setting (all accounts) or as an account setting for individual accounts. The maximum limit you can specify is 120 containers. 15
sim.docker.monitorAPMContainersOnly(previously known as sim.docker.infraMode.enabled) When set to true , allows you to monitor APM containers only. If the value is set to false , monitors all the containers on the machine except the Machine Agent container. You can apply this property at an account level and enable the monitoring of containers with and without APM. This property was formerly named sim.docker.infraMode.enabled . true
sim.exceptions.stacktrace.enabled

When this setting is enabled, Controller stack traces are sent in the error response when a client request encounters an error.

Setting this to true true

 false
sim.machines.agent.process.maxClasses

The global maximum for the number of process classes collected per machine.

  • If maxClasses (global setting in Controller) is lower than maxNumberMonitoredClasses (local setting in serverMonitoringConfig.yml), the local setting is overridden and the globalMaxClasses is the effective maximum for that machine.
  • If maxClasses (global) is higher than maxNumberMonitoredClasses (local), the local setting is the effective maximum for that machine.

SeeMachine Agent Settings for Server Visibility.

Increasing this setting can affect the resource consumption of your deployment. Before you increase this setting, verify that your application environment and Controller can process the increased resource requirements.

 20
sim.machines.container.hostid.conflict.check.enabled

When using Docker Visibility, if the unique host ID container ID container ID host ID

This setting was introduced in version 20.8 of the Controller, with the default value = false true setting --network=host

 true
sim.machines.count.maxPerAccount

A maximum number of machines allowed per account. Any additional machines will not appear. If you have more than 2000 machines reporting to one Controller and need to increase this number, be aware that you might need to increase the sim.machines.registrations.maxPerSecondPerAccount

Changing this setting can affect the resource consumption of your deployment. Before you change this setting, verify your application environment and Controller can process the increased resource requirements.

 2000/account
sim.machines.deleteStaleMachines.maxLimit Set the number of machines to be deleted every 10 minutes, which is one cycle of the purger. For better performance, you can purge up to 700 machines per cycle. 100
sim.machines.dmm.defaultMode

Sets theDynamic Monitoring Mode and Server Visibilitydefault on the Controller. The allowable settings are:

  • KPI – Collect Key Performance Indicator metrics only
  • DIAGNOSTIC – Collect KPI and Diagnostic metrics
  • ADVANCED_DIAGNOSTIC –  Collect All available metrics

You must enter one of these exact strings.If you enter a different value (such as "advanced-diagnostic"), the Controller will not update the setting and some servers might not appear in the Servers list.

KPI
sim.machines.dmm.dmmAllowed With this option enabled, the Controller collects metrics from each Machine Agent based on theDynamic Monitoring Mode and Server Visibilitysetting on that Agent. If this option is disabled, the Controller collects all available metrics from each Agent. false
sim.machines.dotNet.HostIdConflictCheckAllowed

With this option enabled, the Controller throws an exception if a Machine Agent tries to register with the Controller while both of the following conditions are true:

  1. A .NET Agent on the same machine is already registered with the same <host-id> specified for the Machine Agent, and
  2. .NET Compatibility Modeis disabled on either the Controller or the Machine Agent.

The generated exception acts as a reminder that that .NET Compatibility Mode mode must be enabled on both the Controller and the Machine Agent.

Note: Do not disable this option unless instructed by the Splunk AppDynamics customer support.
 true
sim.machines.hostidMappingAllowed You must enable this mode if you want to collect and view Machine or Server metrics on a server with Machine and .NET Agents installed. See.NET Compatibility Mode. true
sim.machines.offline.toStaleTimeoutMillis

How much time, in milliseconds, to wait before considering an offline machine to be stale and marked for deletion. If this value is too high, it prevents fresh data from coming in. If the value is too low, it means less history.

As of Controller >= 21.2.0 two properties are used in conjunction to purge stale machines. These properties are used in tandem to determine how to display theServerslist in the user interface:

  • use sim.machines.offline.toStaleTimeoutMillis (172800000 ms) to determine which machines are considered stale.
  • the sim.machines.registrations.update.frequency setting (86400000 ms) determines the frequency at which a server's lastSeenTimeStamp is updated, if there are no changes in the properties of the monitored host. This setting can not be modified.

You may wish to change the sim.machines.offline.toStaleTimeoutMillis2023-10-19_05-04-05_Access the Administration Consoleas a Controller setting (all accounts) or as an account setting for individual accounts. If this setting is modified in the Controller then a restart is required. If this setting is modified at the account level, you do not need to restart.

Under rare circumstances old logic leads to a machine being deemed eligible for purge earlier than customer configuration.

172800000 ms

sim.machines.registrations.hashCache.enabled Prevent subsequent registration requests from fetch data and compare if nothing has changed in the request payload. true
sim.machines.registrations.maxPerSecondPerAccount

Set the maximum number of registrations allowed per second per account. This setting controls the rate at which concurrent registrations are processed and prevents one account on a multi-tenant Controller from monopolizing resources needed to register the Agents.

The default value assumes that the Machine Agents are started evenly across a minute. If you are running 2000 or more machines, you may need to increase this setting to 600/sec. A suggested formula is: ( (number of Machine Agents, version 4.2 or higher / 60) ).

If you use a deployment script that starts all your Agents within a few seconds of each other, you may need to further adjust the max registrations per second.

Changing this setting can affect the resource consumption of your deployment. Before you change this setting, verify your application environment and Controller can process the increased resource requirements.

60
sim.machines.registrations.update.frequency Updates the timestamp of the machine on a specific frequency, regardless of stale timeout value. Determines the frequency at which a server's lastSeenTimeStamp is updated, if there are no changes in the properties of the monitored host. This setting can not be modified.

As of Controller >= 21.2.0 two properties are used in conjunction to purge stale machines. These properties are used in tandem to determine how to display the sim.machines.offline.toStaleTimeoutMillisServerslist in the user interface. See

Under rare circumstances old logic leads to a machine being deemed eligible for purge earlier than customer configuration.

86400000 ms

sim.machines.reuse.enabled

Reuse SIM Machine entities to handle the ephemeral environment. Support is currently limited to Docker container machines.

If set to false

 true
sim.machines.simAllowed This setting allows you to enable or disable Server Visibility from the Controller. true
sim.machines.stale.purgeIntervalMillis An interval in milliseconds that determines when stale machines are deleted from the Controller database. If the duration of this value is too short, it might overload the server. If the value is too high, then stale machines are deleted more slowly.

21600000 ms

(6 hours)

sim.machines.tags.maxPerAccount The maximum number of unique tags per account. 1000
sim.machines.tags.maxPerMachine The maximum number of tags per machine. 50
sim.metrics.metricBrowser.machineMetricMappings.enabled Enables or disables the Metric Browser to display the metrics reported per machine. When this feature is enabled, tier level aggregated metrics, available under Tier node in the Server metric browser, are not visible. With Streamlined Browsing enabled, theHardware ResourcesandCustom Metricsfolders do not display. true
sim.metrics.store.customMetrics.simNodeEnabled Enables or disables the persistence of custom metric values to machines monitored by Server Visibility. If the setting is:
  • true : The collector stores and displays custom metric values for applications and machines. Custom metrics are displayed in Metric Browsers for both Applications and Server Visibility.
  • false : The collector stores and displays custom metric values for applications only. Custom metrics are displayed in the Metric Browser for Applications only (not for Server Visibility).

Disable this setting if you are collecting a lot of custom metrics. This will prevent potential spikes in the number and rate of metric values stored for individual machines.

 true
sim.metrics.trackRegisteredEnabled EnableStreamlined Browsingfor node metrics. Use this option when you are browsing metrics for tiers that contain multiple nodes. false
sim.perAccountProperty.enabled Enables a user to configure properties at an account level. true

sim.perAccountProperty.syncFrequencyInMillis

The interval in milliseconds for the per Account Property data to sync with a database.300000

sim.perAccountProperty.syncInBackground

Enables syncing the per Account Property data with a database in the background. For better performance, set this value to true . true
sim.processes.bulkDelete.enabled Delete processes in bulk instead of fetching one by one. For better performance, set this value to true. true
sim.processes.delete.maxCount Set the maximum number of processes to be deleted every 10 minutes, which is one cycle of the purger. For better performance, set the delete maxCount processes value to 55000 per cycle. 10000
sim.processes.count.maxPerAccount The maximum number of processes stored per account. 300000
sim.processes.count.maxPerMachine

The maximum number of processes that each Machine Agent can monitor.

Increasing this setting can affect the resource consumption of your deployment. Before you increase this setting, verify that your application environment and Controller can process the increased resource requirements.

 1000
sim.processes.creation.maxConcurrent

The maximum number of processes that can be registered simultaneously by the Controller. The default is 5000 processes. The Machine Agent will retry to register the processes if the first attempt fails. Until the process request is accepted and processed by the Controller, processes within that request will be missing from the Process Details page. The limit works as follows:

Case 1: Machine Agent sends a request with 6000 processes to register, and the limit is set to 5000, then this request is rejected until the limit is increased.

Case 2: Two Machine Agents (MA1 and MA2) each try to register 3000 processes and the limit is set to 5000. Both MA1 and MA2 requests are received by the Controller and only one of the requests will be processed (3000 + 3000 > 5000). So, the Controller can only process one request and rejects the other one.

Case 3: Two Machine Agents (MA1 and M2) each try to register 500 processes and the limit is set to 5000. Both requests arrived at the Controller and since 500 + 500 < 5000, then both requests are processed, and the processes are registered.

 5000
sim.processes.query.maxResultLimit The maximum number of processes that the UI can show (for example, in theServer Dashboard >Processes tab). Suppose the time range is 2 weeks and the Agent reported over 10000 processes during that interval. If maxResultLimit = 5000 , the UI will show 5000 processes. 5000
sim.processes.registrations.maxPerSecondPerAccount Set the maximum number of process requests handled per second per account. For better performance, set to 600/sec. On a multi-tenant Controller, the total number of Agents per second should not exceed 600/sec. 60/sec
sim.processes.stale.purgeIntervalMillis The number of milliseconds between consecutive deletes of stale processes for an account.3600000 ms (1 hour)
sim.processes.terminated.toStaleTimeoutMillis The number of milliseconds before a terminated process is considered stale and can be deleted to make space for new data. You can specify these in the2023-10-19_05-04-05_Access the Administration Consoleas a Controller setting (all accounts) or as an account setting for individual accounts.172800000 ms (2 days)

Machine Agent Hierarchy

To group servers together so that you can apply health rules to the specific server groups, use the Machine Hierarchy property. This property enables you to group servers into arbitrary hierarchies by specifying a hierarchical path to the server. You can group servers based on departments, geographical locations, such as data centers, or other organizational units. You can then create health rules that apply to these departments.

The server hierarchy displays in the Metric Browser, on the Servers list, and on Server Visibility dashboard.

You need a Server Visibility license to use this feature.

You can specify the Machine Hierarchy property using controller-info.xml, a system property, or an environment variable.

For example, to group servers into geographical locations, perhaps representing different data centers, you could group the following:

Machine Hierarchy Grouping

Use the machine-path element in the controller-info.xml configuration file to specify the tier and node:

<machine-path>AD-Financial|Node1</machine-path>

The Metric Browser view for this example:

Metric Browser View

When creating health rules, you can select one or more subgroups from the Affected Entities tab:

Affected Entities