Self-Healing Pipelines for CIM compliance

Configure the Self-Healing Pipelines feature to maintain CIM compliance.

Overview

The Self-Healing Pipeline feature in the Splunk Ingest Monitoring app automatically detects when ingested data deviates from the Common Information Model (CIM), and provides AI-powered remediation.

When data fields are missing, incorrectly mapped, or contain invalid values, downstream workflows, such as Splunk Enterprise Security correlation searches, dashboards, and alerts, can fail without warning. The Self-Healing Pipeline identifies these issues early and guides you through the resolution process to ensure your security and analytics pipelines remain reliable.

Key concepts

Field coverage

Field coverage measures what percentage of events that contain a non-null value for a specific CIM field. For example, if 950 out of 1,000 Authentication events have a populated src field, the field coverage for "src" is 95%. Low field coverage indicates that a field is missing from many events, which can cause CIM-dependent searches and dashboards to return incomplete results.

CIM value validation

CIM value validation measures what percentage of a field's values match the expected values defined by the CIM. For example, action fields in the Authentication datamodel must be one of the following: success, failure, pending, error.

A low CIM value validation value means the field is present, but contains values that don't match the expected values. This can cause lookups and correlations to fail.

Add-on overlay
An add-on overlay is a supplementary Splunk app that adds or corrects field extractions, transformations, and evaluations that are defined in props.conf and transforms.conf. Instead of modifying your existing add-on directly, the overlay app layers its configuration on top, preserving your original add-on, while fixing the compliance issues.

What you can do with Self-Healing Pipelines

Use the Self-Healing Pipeline feature to perform the following tasks:

Detect CIM non-compliance

Monitor your data sources for missing fields, low field coverage, and incorrect field values as defined in CIM.
Get AI-powered remediation

Receive automatically generated fixes for your add-on configuration files, like props.conf and transforms.conf.
Review changes visually

Compare your current add-on configuration against the proposed fixes in a side-by-side difference view.
Preview new field extractions Preview which CIM fields will be extracted after the proposed fix is applied without installing the overlay add-on.
Download ready-to-install packages Download a corrected add-on overlay package that you can deploy directly to your Splunk platform environment.

Self-Healing Pipeline supports the following features:

Monitor and alert on non-compliance: Track sourcetypes mapped to the CIM and receive alerts when they do not meet compliance standards
Configure granular detection Detect non-compliance at the data model, dataset, and sourcetype levels for each alert rule.
Analyze and generate fixes Use AI to collect information, analyze non-compliance, and generate add-on overlays.
Install overlays manually Download and install add-on overlay packages directly to your environment.

Example use cases

You can use Self-Healing Pipeline in the following scenarios:

  1. Scenario 1: An add-on update introduces a field mapping regression. The update triggers an alert for CIM non-compliance, and AI analysis identifies the incorrect mapping, and proposes a fix.

  2. Scenario 2: A new data source is onboarded but its fields don't conform to CIM. The alert detects missing or incorrectly formatted fields and generates an add-on overlay to add the correct extractions.

  3. Scenario 3: Field coverage degrades over time due to source-side changes. Continuous scheduled monitoring catches the drift and notifies you before downstream detections are affected.

Limitations of Self-Healing Pipelines

Self-Healing Pipelines have the following limitations:

Detection scope limitations

Detection is limited to data models, fields, and validation rules defined in the Splunk Common Information Model . Custom data models and custom fields are not supported.

The system identifies missing fields and CIM value validation. It doesn't evaluate whether a field's value is semantically reasonable for that field. For example, the system does not verify that a user field follows an expected format such as an email address, account name, MAC address, or IP-like identifier. It checks only whether the value violates explicit prescribed values defined in the CIM.

Detection operates at the sourcetype field only. Custom condition filters such as source, host, or eventtype are not supported. To monitor multiple source types, create a separate alert rule for each source type.

You must enable CIM Data Model Acceleration for each data model you want to monitor. Without acceleration, you can't create alert rules. The system uses the acceleration summary to identify available data sources, or source types. When data model acceleration is off, the acceleration summary is empty, which prevents alert creation for that data model.

Self-Healing Pipeline uses static alert thresholds rather than relative or anomaly-based detection. Because not every event is expected to have all CIM fields extracted, static thresholds might produce false positives in some environments.
AI remediation limitations

The system collects one problematic sample event from each event type in the target data source and passes it to the AI for analysis. While the system attempts to fetch a broad set of events, it might not capture all event variations. This means the AI might address only part of the problem, and you might need multiple iterations to resolve all non-compliant event types.

The AI has access only to props.conf and transforms.conf. It cannot inspect other extraction mechanisms, such as automatic lookups or lookup table contents, and attempts to infer mappings based on available configuration.

You can only review AI-generated fixes before downloading them. To update the AI-generated fixes, open and edit the files in a text editor.
AI token limitations AI token limits and other resource constraints are located at the Splunk tenant level and not tracked at the service/product level.

LLMs (per stack, uniform):

  • Soft limit: 1M total tokens/24hr rolling window. Account team notified, and service continues.

  • Hard limit: 250M total tokens/24hr rolling window. Service is blocked, if exceeded.

Overlay add-on compatibility with other add-ons

When a vendor releases an updated version of an add-on that natively resolves Common Information Model (CIM) issues, your existing overlay add-on may continue to override the updated add-on's files. This persistence can result in configuration conflicts.

Before upgrading a vendor add-on, review the vendor's release notes to confirm whether the original CIM parsing issues have been natively resolved.

Scenario Action
Release notes confirm all CIM fields that are patched by the overlay add-on, are now natively fixed. Safe to remove the overlay add-on. Uninstall to avoid configuration conflicts.
Release notes confirm some CIM fields patched by the overlay are natively fixed Keep the overlay add-on installed. Removing it will re-expose the fields the vendor has not yet fixed.
Release notes confirm no CIM fields patched by the overlay are natively fixed. Keep the overlay add-on installed. The vendor update does not address the CIM issues.

For more information, see the Install apps on your Splunk Cloud Platform deployment topic in the Splunk Cloud Platform admin manual.

Feature availability and scope

This feature is available on primary search heads, as well as on Enterprise Security (ES) search heads.

Ingest Monitoring app on primary search heads versus Enterprise Security search heads

  • Primary search heads: The full Ingest Monitoring app features are available, including dashboards, scheduled searches, and all alert templates including CIM alerts.

  • Enterprise Security (ES) search heads: The app is trimmed down to only the Self-Healing Pipeline feature. Dashboards, scheduled searches, and other alert templates are not available on ES search heads.

Any alert rules configured on a search head are local to that node and won't be available on the other search heads. This means the CIM alerts rules created or triggered from an ES search head are not accessible on your primary search head. Additionally, CIM alerts rules created or triggered from your primary search head are not accessible on your ES search head.

Prerequisites

Before using the Self-Healing Pipeline feature, verify that your environment meets the following requirements:

  1. Splunk platform environment:

    Make sure you have admin permissions in this Splunk Cloud environment. The Ingest Monitoring app (splunk-ingest-monitoring v1.4.0) is installed.

  2. Common Information Model (CIM) Add-on:

    The Splunk Common Information Model (CIM) app (Splunk_SA_CIM) is installed on your Splunk Cloud Platform instance.

  3. Data model acceleration

    Acceleration is enabled on the CIM data models you want to monitor (for example: Authentication, Network Traffic, Endpoint, and so on). To verify, perform the following steps:

    1. In Splunk, navigate to Settings > Data Models..

    2. Find the relevant data model and confirm that the Acceleration is turned on.

Get started

Enable the Ingest Monitoring app.

Perform the following steps to enable the Ingest Monitoring app:

  1. Log in to your Splunk Cloud Platform instance with an sc_admin or an admin account.

  2. Open the Ingest Monitoring app.

  3. If this is your first time, complete the guided setup and select the indexes you want to monitor.

  4. Select Submit to enable monitoring.

  5. Make sure there is a Data Management navigation pane on the left side of the app.

After setting up, the app begins collecting ingestion metrics and enables the alert system.

Verify the CIM non-compliance template is available

Perform the following steps to verify that the template is available:

  1. In the left navigation page of the Ingest Monitoring app, select Alerts.

  2. Select New alert rule.

  3. On the templates page, confirm that the CIM non-compliance template appears as an option.

Create a CIM non-compliance alert

The Ingest Monitoring alert system provides the Common Information Model (CIM) non-compliance detection feature, which enables the following results:

  • Field extraction coverage, which triggers an alert when the ratio of non-extracted suggested fields is below the threshold.

  • CIM value validation coverage, which triggers an alert when the ratio of non-expected values appearing in events is below the threshold.

Follow the guided setup to can customize the alert rule by setting up alert frequency, expiry, throttle, and severity.

Provide alert details

Configure the alert details to define the name, description, and data source for your CIM non-compliance alert.

  1. On the Alerts page, select Create alert.

  2. Select the CIM non-compliance template.

  3. Enter a name for the alert, such as Authentication CIM Compliance - firewall.

  4. (Optional) Add a description.

  5. Choose the CIM data model and dataset to monitor, such as Authentication.Authentication or Network_Traffic.All_Traffic.

  6. After choosing a data model, the source type drop-down menu populates with values that are mapped to that model. Select the source type you want to monitor.
    Note: Only accelerated data models are supported. If the data model you select isn't accelerated, the sourcetype dropdown in the next step will display no options.

  7. After choosing a data model, a table populates all required and suggested CIM fields for the selected dataset.

  8. By default, all fields are selected. Deselect any fields you do not want to monitor.

  9. Select Next.

Set alert trigger rules

Perform the following steps to set alert trigger rules:

  1. Set the field coverage threshold, which triggers the alert when the percentage of events containing a monitored field drops below the set value. By default, this threshold is set to 80%.

  2. Set the CIM value compliance threshold, which triggers the alert when the percentage of field values matching the expected CIM values drops below the set value. By default, this threshold is set to 80%.

  3. Configure the alert schedule and throttle:
    1. In the Alert schedule section, define how often the alert runs. Select one of the available options:

      • every 4 hours

      • daily

      • weekly

    2. (Optional) Deselect Exclude weekends (Saturdays and Sundays) to skip weekend evaluations.

    3. In the Expires after section, determine how long the triggered alert remains active. The default is 24 hours.

    4. Select Throttle to suppress duplicate alerts within a time window. The default is 60 minutes for CIM alerts.

  4. Select Next.

Set alert actions

Perform the following steps to set alert actions:

  1. Set the severity level for the alert. Select one of the available options:

    • Info

    • Low

    • Medium

    • High

    • Critical

  2. Select Next.

Review and activate your alert

Perform the following steps to review and activate your alert:

  1. Review all configured settings: data model and dataset, source type, field coverage and CIM value validation thresholds, schedule, expiry, throttle settings, and severity.

  2. Select Create to save the alert.

The alert is now scheduled and runs according to your configured schedule. When the alert condition is met, such as when field coverage or CIM value validation falls below the threshold, it appears in the Triggered alerts tab.

Triggered alert grouping

Triggered alerts from the same alert rule that haven't been investigated yet are grouped together. This lets users review and remediate them as a set instead of one at a time. The triggered alert grouping works as follows:

  • When a group has not yet been remediated, every new triggered alert from that alert rule joins the group. You can open the group to review information, such as when each alert was triggered, sample events, and the specific non-compliant values highlighted for each field.

  • When you start remediation from a group, all X triggered alerts in that group move to a "Remediation in progress" status. Any new triggered alerts from that alert rule after this point start a new group.

  • Once the AI finishes the analysis and remediation, the group's status changes to Done.

  • The triggered alerts in a Done group are not removed immediately. They remain visible until the alert rule's expiry setting is reached, at which point they disappear.

  • The expiry setting configured on the alert rule determines how long triggered alerts remain visible in their group before they are gone. For more information, see the Set alert trigger rules section of this manual.

CIM non-compliance alert setup examples by data source

The following table lists the best practice configurations for the Common Information Model (CIM) data model, datasets, and fields needed to monitor for each supported data source. Use this as a reference when configuring CIM compliance alert rules in Splunk Ingest Monitoring.

Data model and dataset Source type Fields
Performance.All_Performance.CPU aws:cloudwatch dest, cpu_load_percent
Network_Sessions.All_Sessions.VPN cisco:asa dest_ip, user, vendor_product
Network_Resolution.DNS crowdstrike:events:sensor answer, message_type, query, reply_code, reply_code_id, vendor_product
Change.All_Changes.Account_Management google:gcp:pubsub:audit:admin_activity action, change_type, command, dest, dvc, object, object_attrs, object_category, object_id, object_path, result, result_id, src, status, user, vendor_product
Network_Traffic.All_Traffic flowintegrator action, bytes, bytes_in, bytes_out, dest, dest_port, dvc, rule, src, src_port, transport, vendor_product
Data_Access.Data_Access o365:management:activity action, app, dest, object, object_category, src, vendor_account, user, vendor_product
Authentication. Authentication ms:aad:signin action, app, dest, src, user
Authentication. Authentication pan:globalprotect action, app, dest, src, src_user, user
Authentication. Failed_Authentication OktaIM2:log action, app, dest, src, src_user, user

Investigate and remediate CIM non-compliance

Investigate and remediate CIM non-compliance issues to ensure configurations meet required standards.

Overview

When CIM non-compliance alerts trigger, investigate and remediate CIM non-compliance issues to ensure configurations meet required standards.

Review a triggered alert group

Perform the following steps to access the remediation workflow:

  1. In the Ingest Monitoring app left navigation pane, select Alerts.

  2. Select the Triggered alerts tab.

  3. Find and select the triggered CIM non-compliance alert in the list. The details panel opens.

  4. Select Investigate.

    The Investigation page opens, showing the group's stats, such as when each alert in the group was triggered, sample events, and the non-compliant values highlighted for each monitored field.

  5. When you are ready, select Start remediation.

    The Remediation wizard opens, and it starts the AI analysis step.

Analyze alerts

The system automatically performs the following actions:

  1. Retrieves the triggered alert results, including failed fields, failure reasons, and coverage percentages.

  2. Collects sample events from your data to identify the actual field values.

  3. Identifies the relevant add-on stanza configurations in props.conf and transforms.conf.

  4. Sends the information to the AI analysis service.

A progress indicator appears, displaying the AI's reasoning and validation progress for each stage of the process. When AI analysis completes, the system provides a summary of non-compliant fields and the reason for the failure, such as missing or incorrect values. It then describes which fields are fixable and which are not, along with a confidence level for each fixable field, and proposes fixes based on your add-on configuration.

Review the AI analysis and select Continue to approve the fixes.

Best practice remediations

Review the AI-generated configuration fixes in this step.

Proposed changes are grouped by target file, such as props.conf or transforms.conf. Use the tab bar to switch between files. A side-by-side difference editor shows your current configuration on the left and the proposed fix on the right, with changes highlighted. Expand the AI reasoning panel to see a detailed explanation for each change.

A field extraction preview table appears, based on the new props.conf and transforms.conf, displaying which monitored fields are now compliant, and which are still not compliant.

Review the proposed changes carefully before you proceed.

When you are satisfied with the changes, select Continue.

Download artifacts

You can download the generated add-on configuration files.

The left panel lists all generated configuration files with their file paths.

  1. Select any file to preview its contents in the code editor on the right.

  2. Select Download TA package to download the complete add-on overlay as a .tar.gz package ready for deployment. You can also download individual files by selecting the download icon next to each file in the list.

  3. Select Continue after downloading the package.

Install and deploy the add-on overlay package

Next, install and deploy the add-on overlay package to your Splunk Platform environment.

Deploy the overlay to the same search head where you ran the detection and remediation. The generated overlay add-on only works on the search head where the detection and remediation ran. Deploy the overlay add-on back to that same search head. An overlay generated on one search head will not fix compliance issues on a different search head.

Avoid complications with future fixes by installing the add-on overlay package before you perform a run on the same alert rules.

The uploaded app appears in the table on the Uploaded Apps page. The app name and version are displayed when the package passes all cloud vetting checks.

For more details, see the Deploy and manage private apps topic in the Splunk developer documentation.

Select Done to return to the Triggered Alerts list.

Manage alert rules

You can manage alert rules in the following ways.

View alert rules

Go to the Alerts page and select the Alert Rules tab to see all configured alerts. Each row shows the alert name, type, schedule, severity, and enabled status.

Edit an alert rule

Perform the following steps to edit an alert rule:

  1. Select an alert rule row.

  2. In the details panel, select Edit.

  3. Modify settings through the same guided setup used during creation.

  4. Select Save to apply your changes.

Activate or deactivate an alert rule

Perform the following steps to deactivate an alert rule:

  1. Select the alert you want to manage.

  2. From the drop-down menu, select Activate or Deactivate. Deactivated alerts stop running on their schedule but retain their configuration.

Delete an alert rule

Select the more icon associated with the alert, and select Delete. This permanently removes the alert and its trigger history.