Test detections on production data using Splunk Enterprise Security

Test detections on production data using Splunk Enterprise Security to evaluate their behavior and validate search results without impacting SOC analyst workflows.

When you change the status of a detection for testing, search results are written to separate indexes such as test_risk or test_notable instead of the production index and surfaced in a dedicated Detection test queue on the analyst queue in Mission Control. Collecting the search results for detections that are undergoing testing in a separate queue keeps the test findings separate from the production workflow. The test queue is read-only and does not support opening investigations or taking action on findings. the test queue is used only to evaluate detection search results and is not a substitute for operational analyst workflows.

Setting a detection to Testing status lets you validate a detection in a production environment without turning it on. Turning on a detection in production can introduce noise to analyst queues and disrupt triage workflows. The Testing status provides a controlled path to evaluate detections on real data before moving them to production.

Use case1: Test a detection against production data without disrupting analyst workflows

Set a detection to Testing status and turn it on to generate findings from live production data. Test findings are written to isolated indexes and surfaced only in the Detection Test Queue, so SOC analysts working in the analyst queue on Mission Control are not impacted. You can evaluate detection fidelity, tune thresholds, and assess false positive rates before turning on the detection for production.

Use case 2: Compare multiple versions of a detection simultaneously

Create two or more versions of the same detection and set one or more of them to Testing status. For example, you might compare a version of the detection that uses static tuning with a lookup table against a version of the detection that uses machine learning with the Splunk Machine Learning Toolkit (MLTK). Both versions can run in parallel, allowing you to evaluate their performance simultaneously before deciding which version to move to production.

Use case 3: Keep a production version running while testing a new version

When you save a new version of a detection and set it to Testing status, any previous version of that detection that was turned on remains in its current On state. This allows you to run a version of the detection that might be potentially moved to production through the testing workflow without any interruption to the detection that is active in production.

Use case 4: Evaluate historical and forward-looking findings for a detection

When testing a detection in the detection editor, you can look back at historical data to estimate findings before turning on the detection. Once the detection is placed in Testing status, Splunk Enterprise Security calculates forward-looking findings and intermediate findings, as if the detection was turned on. This gives you a realistic projection of how the detection might perform in production.

Turn on testing for a detection

Follow these steps to turn on testing for a detection in the detection editor:

  • In Splunk Enterprise Security, select Security content and then select Content management and open a detection that you want to test in the detection editor.
  • Go to the Status drop-down menu and select Testing to test the selected detection.
  • Navigate to the Mission Control page and open the Detection test queue to review findings that are created by the detection.
    Note: Findings generated while a detection is in Testing status are written to the test_risk or the test_notable indexes and are visible only in the Detection Test Queue and do not appear in the production analyst queues.

Create and test a new version of the detection

Follow these steps to test a modified version of a detection while keeping the existing production version running:

  1. In Splunk Enterprise Security, select Security content and then select Content management and open a detection that you want to test in the detection editor.
  2. Modify the detection as needed.
  3. Save the modified detection as a new version.
  4. Go to the Status drop-down menu and select Testing to test the selected detection.
  5. Review the test findings in the Detection Test Queue.
    Note: The previous version of the detection that was turned on remains in the On status and continues to create production findings. The new version runs independently in Testing status, and pushes results to the isolated test indexes for review in the Detection Test Queue.

Limitations when using detections in testing status

Following are some limitations to keep in mind when using detections in testing status:

  • Detection as code deployments: When managing detection configurations, if you use detections as code through source control or automated deployment pipelines, note that the detections in Testing status are saved to the SA-TestModeControl app instead of the app where they are usually located.

  • Workload management is not supported: Detections in Testing status do not integrate with workload management and do not run at a low scheduler priority compared to detections in production. To avoid unnecessary load on your environment, consider configuring a less frequent schedule for detections in Testing status unless you are actively monitoring results.

  • Finding-based detection index scope: For finding-based detections (FBDs), the testing feature by default reads from either the test indexes or the production indexes, not both. However, you can manually update the SPL logic of the finding-based detection to read from specific indexes or a combination of both. For example, you can configure a test version of a finding-based detection to read from both the test and production indexes, while the production version of the detection reads only from the production indexes. This allows you to observe how intermediate findings or risk during testing accumulates compared to production before moving the detection to production.

Note: Search head and search head cluster deployments use infrastructure reverse proxy with default timeout of 5 minutes for REST API calls. If testing takes more than 5 minutes to complete, you might receive an HTTP 504 error due to a premature timeout. You can decrease Test Timeout option to prevent this error. However, if more results are required, then reverse proxy timeout must be increased, which might require changing stack configuration.