Example: CPU usage spike

CPU utilization is likely to vary throughout the day, and high CPU usage is expected and normal under certain circumstances. CPU utilization can be especially high during nightly batch or offline model processing.

Note: When your observed spike aligns with your nightly batch or offline model processing schedule, and you observe no other indicators such as errors in logs or model failure, you can consider your spike normal.

In cases where the CPU spike(s) are not aligned with nightly batch or offline model processing, and you also observe errors in the logs and model failure, consider the spike(s) abnormal and contact support.

You can use the graphs available in the Splunk UBA Monitoring App to examine any CPU spikes. These graphs can be found under Monitoring > Systems, as shown in the following image:

This image shows the navigation bar of the Splunk UBA Monitoring App. The Monitoring tab is selected and the sub-menu item of Systems is highlighted.

The following image shows example CPU usage data as captured from the Splunk UBA Monitoring App over a past 48 hour period. Each line in the graph represents a node. You can see there are usage spikes for particular nodes. You can hover over any point in the graph to see additional information:

This image shows an example graph of CPU usage over a 48 hour period where each line represents a UBA node.

The following image uses the same 48 hour period information as shown in the previous image, but opened in the Search tab of the Splunk UBA Monitoring App, with the time span set as 1 minute. This time span setting provides a more precise measurement in terms of CPU usage. You can see that the utilization of certain nodes is spiking up to 100%:

This image shows an example graph of CPU usage over a 48 hour period as opened in the Search tab and with the time span set to one minute.

This view into the CPU usage shows healthy behavior in spite of the high spikes. The CPU utilization is rising overnight at about the same point in time, and then coming back down during the day.

The following image shows another example of a CPU usage spike at night. Again he CPU utilization is rising overnight and then coming back down during the day:

This image shows an example of a CPU usage in percentages chart. There is a spike of usage at the time around midnight.

For informational purposes only, you can open the /opt/caspida/content/Splunk-Standard-Security/jobs/scheduler/jobs.json file to determine when a model runs at night. You can also view the schedule of the following models:

Note: As a best practice, do not change the schedules of these model jobs without discussing with team stakeholders or UBA support.
Model name Schedule
External Destination Popularity "0 0 0 * * ?", // every day at midnight
Deterministic Profiling Model "0 5 0 * * ?", // every day at 12:05AM
VPN related Anomaly Detection Models "0 30 1 * * ?", // every day at 1:30AM
Beacon Assessment Model "0 0 2 * * ?", // every day at 2AM