Manage the number of threats and anomalies in your environment

The Offline Rule Executor in Splunk UBA runs nightly to trigger scheduled anomaly and threat rules, and also perform threat revalidation in real time when there are rule changes, when anomalies are removed from the system, or when anomaly scores are changed.

Note: Threat revalidation can be time consuming and cause memory issues depending on a variety of factors. These factors include the types and age of the anomalies involved in the threat, the number or anomalies and entities involved in the threat, and any custom threat rules active in your system.

The Offline Rule Executor times out after 15 minutes. If threat revalidation takes longer than 15 minutes, some computations are not generated in Splunk UBA.

Perform regular maintenance of your Splunk UBA deployment using any combination of the following tasks:

Maintenance task Guidance
Cleanup of anomalies more than 90 days old
  • If your deployment is 10 nodes or higher, you can delete up to 200,000 anomalies at one time.
  • If your deployment is fewer than 10 nodes, you can delete up to 100,000 anomalies at one time.

See Delete anomalies in Splunk UBA.

Close unwanted threats See Close threats in Splunk UBA.
Monitor the total number of anomalies in your environment
  • If your deployment is 10 nodes or higher, do not exceed 1.5 million anomalies.
  • If your deployment is fewer than 10 nodes, do not exceed 1 million anomalies.
Monitor the number of rule-based threats in your environment
  • If your deployment is 10 nodes or higher, do not exceed 2,000 rule-based threats.
  • If your deployment is fewer than 10 nodes, do not exceed 1,000 rule-based threats.
If you have threat rules which require more than one hour to run, modify the rule engine timeout period. 1. Log in to the Splunk UBA management node as the caspida user.

2. Add or edit the rule.engine.process.timeout.min property to /etc/caspida/local/conf/uba-site.properties and the desired number of minutes. The default is 60 minutes. The following example sets the timeout period to 90 minutes: 

rule.engine.process.timeout.min=90

3. In distributed Splunk UBA deployments, run the following command on the management node to synchronize the cluster: 

/opt/caspida/bin/Caspida sync-cluster /etc/caspida/local/conf

4. Run the following command on the management node to restart the Offline Rule Executor: 

sudo service caspida-offlineruleexec restart