AI troubleshooting agent and remediation plan in Splunk Observability Cloud

Use AI troubleshooting agent and remediation plan to analyze root causes, test hypotheses, and resolve alerts.

When your systems experience failures and require troubleshooting, AI troubleshooting agent automates root cause analysis, allowing engineers to identify failures faster. After identifying potential root causes, you can move on to the AI remediation plan to see guided steps for remediation.

Availability

The AI troubleshooting (also known as RCA) agent and remediation plan is currently available to customers in all non-Governmental realms of Splunk Observability Cloud. Reach out to your Splunk representative if you are interested in using this feature.

Use AI troubleshooting agent to identify potential root causes

AI troubleshooting agent refers to the root cause analysis powered by Splunk AI. Alerts and detectors determine whether services, business transactions, and infrastructure components are healthy.

For alerts related to Splunk APM services, APM business transaction errors, and Kubernetes in Infrastructure Monitoring, Splunk Observability Cloud triggers the AI troubleshooting agent to perform root cause analysis and display suspected root causes when the user accesses the alert.
Note: To learn more about alerts, see Introduction to alerts and detectors in Splunk Observability Cloud. The AI troubleshooting agent and remediation plan currently support detectors and alerts with standard, default metrics. Custom metrics are not supported at this time.
Note: RCA is 'Opt In' by default for an Org. Users can still execute RCA by manual selecting the 'Run root cause analysis button'. To automatically enable for your org, contact Splunk Customer Support.
When systems begin to break down, alerts in Splunk Observability Cloud initiate contact with the humans responsible for maintaining your systems by sending notifications to designated users through email, Slack, ServiceNow, etc.

When users open the alert through an external notification, or directly via an Alert page in Splunk Observability Cloud, they find a summary of the alert with the following tabs:

  • Overview

  • Root Cause Analysis

  • Evidence

On the Overview tab, you see the following:

  • Summary of the alert, including the rule triggered

  • Impact analysis, which defines the blast radius of the problem regarding applications and services

  • Primary root cause

  • Additional troubleshooting information

Troubleshooting is integrated with the AI Assistant chat experience. For example, the root cause analysis execution status or 'chain of thoughts' appears in the chat. The RCA may take several minutes based on amount of evidence (e.g., logs, dependencies) to analyze.

On the Root Cause Analysis tab, you can rate the overall investigation by selecting the thumbs up or thumbs down in the chat; Splunk uses this information to improve or reinforce the results.

Root cause analysis continues to run even if the user exits the page. Root cause analysis is run once per user, so upon re-entering the alert page, the results are available immediately. There is an option to regenerate (e.g., latency with logs) on the Root Cause Analysis tab.

To minimize Mean Time to Resolution (MTTR), you can do the following when you receive an alert:

  1. Review the alert Overview tab to determine the domain and impact radius of the problem, the primary root cause, the alert and detector information, as well as additional troubleshooting information.

  2. Further investigate potential root causes by selecting Review root causes on the Overview tab.

  3. On the Root Cause Analysis tab, select the specific root cause you want to investigate. The Agent provides a relative Confidence level based on the amount of supporting evidence. Then select links found in the summaries of suspected root causes to read more detail about the suspicious behavior in the relevant Splunk Observability Cloud component.

  4. Additionally, you can view the Evidence tab to see relevant logs, exemplar traces, Splunk APM services, and other evidence contributing to the alert.

Use AI remediation plan to resolve alerts

After reviewing the suspected root causes and evidence provided by the troubleshooting agent, you are now ready to leverage the AI remediation plan in Splunk Observability Cloud. AI remediation plans are currently available for Kubernetes-related Alerts.

Navigate to Action plan by selecting View AI-generated action plan below the suspected root cause in the Root Cause Analysis tab to see the guided steps for remediation. The AI remediation plan generates a set of hypotheses and root causes along with the associated actions.

To test the hypotheses and resolve the alert, you can do the following:

  1. Select a hypothesis and a root cause to remediate.

    You will see a high-level graph describing the selected hypothesis, root cause, and the associated steps you need to take to resolve the incident.

  2. Copy and run the suggested kubectl command or code blocks in your terminal or source code location.

  3. Submit the output so the plan can determine next steps.

  4. As you iterate through the action plan, mark each step as completed. Alternatively, you can undo steps as needed.

  5. When you complete all the steps, you receive a summary of your actions.

After going through the remediation flow, you can mark the alert as resolved and close the incident if the outcome is satisfactory.

How To Make RCA More Effective

For RCA to work optimally, Splunk Observability Cloud needs enough context to connect the alert with the affected service or Kubernetes resource, logs, and related entities.

  1. Use standard dimensions on alerts. For service alerts, include service name and environment, such as service.name / sf_service and deployment.environment / sf_environment. For Kubernetes alerts, include cluster, namespace, and the affected resource, such as pod, deployment, container, node, job, or cronjob.

  2. Correlate logs with services, environments, traces, and Kubernetes resources. Logs are most useful when they include the same entity fields used by alerts and telemetry. For APM, include service name, environment, trace ID, and span ID where applicable. For Kubernetes, include cluster, namespace, pod name, container name, workload, and node where applicable. Make sure these log fields are mapped into the entity index so RCA can reliably associate log events with the affected service or Kubernetes resource.

  3. Add infrastructure identity tags to APM spans. RCA uses span tags to derive relationships between services and infrastructure. Make sure spans include the infrastructure context of the workload that emitted them, such as Kubernetes cluster, namespace, pod, container, workload, and node where applicable. This allows RCA to connect a service symptom to the underlying Kubernetes or infrastructure entity that may be causing it.

  4. Prefer standard APM and Kubernetes telemetry conventions. Standard OpenTelemetry-style names and Splunk APM/Kubernetes dimensions improve classification and routing. Custom metrics can still be useful, but they should include enough standard dimensions to identify the affected domain and entity.

Before running RCA

Access and data mapping to confirm.

  1. Log access. Confirm the user or RCA execution context can search the relevant log indexes for the affected service or Kubernetes resource. If logs are restricted, RCA may miss key evidence.

  2. Entity index mapping. Confirm important log fields are mapped into the entity index, especially service name, environment, cluster, namespace, pod, container, workload, and node where applicable. This lets RCA associate log events with the alerting entity and related infrastructure instead of treating logs as unconnected text.