Agent Mode in Splunk AI Assistant

Introduced with Splunk AI Assistant version 2.0.0, Agent Mode allows for agentic response generation in the assistant. Agent Mode expands Splunk AI Assistant capabilities with access to a wider set of tools, including running event scans, fetching existing dashboards, and executing searches. Agent Mode can parse a user prompt into parallel tool and skills calls, and with permission, execute searches.

CAUTION: In Agent Mode, executing non-trivial searches requires explicit permission from the user. These non-trivial searches can impact consumption and affect your SVC costs.

Requirements

In Splunk AI Assistant version 2.1.0 and higher, Agent Mode is available for both Splunk Cloud customers and Splunk Enterprise customers who install the assistant through the Cloud Connected option.

See the following table for requirement specifics:

Requirement Details
Splunk version You must be running version 10.1.x or higher.
AWS region You must be in a supported AWS region to use Agent Mode. See the supported regions listed in the next section.

Splunk Enterprise customers who install Splunk AI Assistant through the Cloud Connected option must select a supported region at the time of installation. To change your region you must redo the installation process.

Cloud Connected customers also lose Context data if they change regions. It can take up to 2 weeks to build back this knowledge repository.
Model Runtime

You must also opt-in for Model Runtime.

On the Settings page and General tab, opt-in for Model Runtime by selecting "Let Splunk determine the best model to deliver the outcome based on your prompt (this may include models hosted outside Splunk Cloud Platform)".

Supported regions

Agent Mode is only supported in the following AWS regions:

  • US West

  • US East

  • Canada Central

  • EU Frankfurt

  • EU London

  • EU Paris

  • EU Dublin

CAUTION: Splunk Enterprise customers who install Splunk AI Assistant through the Cloud Connected option must select a supported region at the time of installation. To change your region you must redo the installation process.
CAUTION: Cloud Connected customer who choose to change regions lose any stored Context data. It can take up to 2 weeks to build back this knowledge repository on your new region setup.

Enable Agent Mode

If you have met the requirements for Agent Mode and want to use it, your Splunk administrator must go to the Settings page for Splunk AI Assistant and turn Agent Mode on.

On the Settings page and the General tab, select or de-select Enable Agent Mode as shown in the following image:

Note: Administrators can turn Agent Mode on or off anytime.
This image shows the Settings page of the AI Assistant app. The tab for General is selected and the toggle button to enable or disable Agent Mode is highlighted.
Note: If you are an administrator but cannot select the Agent Mode option, you might not be in a supported AWS region.

Enable and configure Agent Mode as a Cloud Connected customer

Splunk AI Assistant version 2.1.x and higher offer Agent Mode to users who install the app through the Cloud Connected option. Using Agent Mode is completely optional. Even if you select a region that supports Agent Mode when you install and activate the app, you must still complete additional steps within the app before you can begin using Agent Mode.

Complete the following steps after you install Splunk AI Assistant using the Cloud Connected option:

  1. Toggle Agent Mode on from the Settings page as shown in the previous section.

  2. Agent Mode requires bidirectional communication:

    1. Open an inbound port. The port must be 8089 which is the Splunk default management port.

    2. When you open an inbound port to your Splunk instance you must allow list certain IP addresses. Limiting the allow list to the following IPs means only Splunk can call into the Splunk stack: ​​
      • 107.23.112.14

      • 3.221.64.88

      • 44.197.225.234

      • 44.216.116.179

      • 52.207.14.28

      • 52.207.14.28

  3. On the Settings page, complete the information for the Cloud Connected Certificate and Stack Addresses sections as shown in the following image: This image shows the Settings page of Splunk AI Assistant and the General tab. The sections for Cloud Connected Certificate and Stack Address are highlighted.

    1. Cloud Connected Certificate: Upload your CA certificate to authenticate the instance and allow secure HTTPS callbacks from Splunk Cloud Services (SCS). For more information on certificates, see the next section "Cloud Connected customer certificates".

    2. Stack Address: Enter the public-facing IP address or hostname of your Splunk instance. This supports bidirectional SCS connections. Both standalone Search Head (SH) and Search Head Clusters (SHC) are supported.

      1. For standalone SH, use the IP address of the standalone SH.

      2. For SHC you can use the IP address of the load balancer. Custom domain names are also an option.

    3. Select Save​ when done.

Generate and configure a certificate to enable Agent Mode for Cloud Connected customers

See the following sections for an overview of what you need to do and why, with links to the official Splunk documentation for more detailed steps.

Note: If you already have an existing SAN certificate for the node (load balancer, Search Head) you don't need to create a new one and you can upload the existing PEM file to allow your stack to verify SCS.

Why this matters for Cloud Connected customers

With Cloud Connected, Splunk Cloud Services (SCS) initiates HTTPS connections into your on-premises Customer Managed Platform (CMP) over port 8089. For these connections to succeed, your stack needs a valid certificate on that port, and SCS needs to trust it. Generating a proper certificate and making it available to Splunk is what allows the cloud side to reach your on-premises stack securely.

Before you begin: How many certificates do you need?

You only need 1 certificate, depending on your setup:

  • Single Search Head (SH): Generate the certificate for that 1 primary search head.

  • Search Head Cluster (SHC): Generate the certificate for the load balancer only. Because traffic is routed through the load balancer, securing it covers the cluster's external connection point, so there is no need to create separate certificates for each search head.

This keeps things simple and avoids the overhead of managing certificates across multiple nodes.

Step 1: Generate or obtain a SAN certificate

Generate a certificate, or request one from your chosen Certificate Authority (CA), for your load balancer (SHC) or single search head (SH). Include all the host names SCS and other clients might use to reach the node in the SAN field, such as the fully qualified domain name (FQDN) and any short names. The SAN field is what modern TLS uses to validate the connection across multiple host names.

Your certificate must meet the following requirements:

  • Be in PEM format and comply with the X.509 standard.

  • Use an RSA private key.

  • Include the Extended Key Usage values for both serverAuth and clientAuth . This is required for KV Store compatibility.

Learn more: How to prepare TLS certificates for use with the Splunk platform

Step 2: Prepare the combined certificate file

Splunk expects your server certificate, private key, and CA certificate combined into a single PEM file, in that specific order. If you use an intermediate CA, those intermediate certificates go between the server certificate and the root CA certificate. Getting this order right is essential, as Splunk cannot use the file otherwise. Once combined, restrict the file's ownership and permissions so that only the Splunk user can access it. This protects your private key.

Learn more: How to prepare TLS certificates for use with the Splunk platform

Step 3: Configure Splunk Web to use the certificate

Configure Splunk Web to serve over HTTPS using your new certificate. This secures the connection between your browser and the Splunk web interface. You will point Splunk Web to your private key and your combined certificate file.

Learn more: Configure Splunk Web to use TLS certificates

Step 4: Configure secure inter-Splunk communication on port 8089

Update your Splunk server configuration to use the new certificate for splunkd and KV Store traffic on the management port of 8089. This is the exact port SCS uses to call into your CMP, so configuring it correctly here is what enables Cloud Connected. You point Splunk to your combined certificate and your CA's root certificate so it can validate connections.

Learn more: Configure TLS certificates for inter-Splunk communication

Step 5: Enable host name validation

Turn on TLS certificate requirements and host name validation. This ensures that the machines your instance communicates with are genuinely who they claim to be, verified through the Common Name or SAN fields in the certificate. This adds a strong layer of protection against impersonation and machine-in-the-middle attacks. Host name validation requires that certificate requirements are already enabled.

Learn more: Configure TLS certificate host name validation for secured connections between Splunk software components

Step 6: Restart and verify

Restart splunkd to apply your changes, then confirm the node is using the new certificate and that host name validation is working as expected. Reviewing the relevant log files can help you confirm everything is functioning correctly.

Step 7: Make your certificate available to Splunk Cloud Services

What to upload

Upload the certificate only. The Settings page validates your upload to confirm it is correct and secure, and checks that you have not unintentionally included a private key.

Note: Never include your private key in the uploaded file

Certificate lifecycle

  • Old certificates are automatically expired.

  • You can manually delete or re-upload certificates at any time to fit your own rotation schedule.

(Optional) Mutual TLS

If your security requirements call for both sides of a connection to authenticate one another, you can take the additional step of enabling mutually authenticated TLS. This further reduces your attack surface by requiring valid certificates on both the client and server side of each connection.

Learn more about Mutual TLS (mTLS): Configure mutually authenticated transport layer security (mTLS) on the Splunk platform

How certificates are handled

When Splunk AI Assistant connects to the Customer Managed Platform (CMP) over HTTPS, the connection occurs on port 8089 and certificate handling depends on your stack's Transport Layer Security (TLS) configuration:

  • Public CA-issued certificate​: If your stack already has a public CA-issued certificate on port 8089, no manual upload is required. Splunk AI Assistant connects seamlessly.​

  • Self-signed or internally issued certificate​: If your stack uses a self-signed or internally issued certificate, you must manually upload the certificate through Splunk AI Assistant and the Settings page. The uploaded certificate is stored in an S3 bucket on the SCS side and is used to establish the HTTPS connection on subsequent SCS to CMP requests.

How certificate and lifecycle validation work

There are 2 methods for certificate validation and lifecycle validation​:

  • Upload validation​: When you manually upload a certificate, Splunk AI Assistant and the Settings page validates the upload to ensure it is correct and secure. Specifically, it checks that a private key is not unintentionally included.​

  • Lifecycle management​: Old certificates are automatically expired. You can also manually delete or re-upload certificates at any time to accommodate custom rotation schedules.​

Learn more about certificates

See the following links to learn more about certificates at Splunk:

(Optional) Limit SVC consumption through roles and capabilities

To limit the risk of high SVC consumption through searches executed in the assistant, administrators have the option to create a new role, put limits on that role, and assign that role to specific users. This step isolates who can execute resource-heavy searches.

See the following to learn more:

Note: If this option does not suffice, administrators can turn off Agent Mode to prevent users from approving searches to run.

Tracking executed searches

Searches you execute using Agent Mode can be tracked by administrators through audit logs. You can also use Splunk AI Assistant to generate these metrics from the audit logs.

CAUTION: In Agent Mode, executing any search requires explicit permission from the user. These searches might have consumption impact and affect your SVC costs.

Limitation

The metrics for version 2.0.0 and higher of Agent Mode are not available in the AI Usage Dashboard. To learn more about the usage dashboard in the Cloud Monitoring Console (CMC) see Monitor AI Assistant usage.

Beta feature: Teach AI

Agent Mode includes some default guardrails, but you can use the beta feature of Teach AI (Beta) to provide the assistant further guidance in markdown format.

Before you begin

Review the description, requirements, and provide feedback for Teach AI (Beta) in the Splunk Voice of the Customer (VOC) portal. See https://voc.splunk.com/preview/teach_splunkai

Preview disclaimer

Beta features described in this document are provided by Splunk to you "as is" without any warranties, maintenance and support, or service-level commitments. Splunk makes this Beta feature available at its sole discretion and may discontinue it at any time. Use of Beta features is subject to the Splunk Pre-Release Agreement for Hosted Services.

Teach AI settings

From the Settings page of the assistant, choose the General tab and scroll down to the Agent Mode section where you can enable the Teach AI (Beta). Example markdown has been provided, as shown in the following image:

This image shows the Settings page of the AI Assistant app. The section for Teach AI (Beta) is highlighted. You can use Teach AI (Beta) to add your own markdown details to help guide the assistant. An example markdown has been provided for review.

Choose Edit markdown definition to make and save any changes.

The Teach AI (Beta) section also provides a Markdown preview as shown in the following image:

This image shows the Settings page of the app. When scrolled down further on the Settings page, you can see a Markdown preview of the markdown added to the Teach AI (Beta) sub-section.