Model Runtime in Splunk AI Assistant

Version 1.4.0 and higher of Splunk AI Assistant provides the option to use the large language models (LLMs) hosted in Splunk Cloud Platform or models hosted in Azure OpenAI.

When you use the Model Runtime feature, Splunk AI Assistant determines when to use a Splunk platform hosted LLM, and when to use a third-party LLM. Third-party LLMs can provide better response quality through the assistant, depending on factors such as use case and cost.
Note: Users on IL 2 FedRAMP deployments can only choose the Splunk-hosted model option.
Splunk AI Assistant version 1.4.0 or higher uses Model Runtime by default. Administrators can turn off this functionality at any time from the Settings page.
Note: You must opt-in to Model Runtime if you want to use the Agent Mode feature. To learn more, including Agent Mode requirements, see Agent Mode in Splunk AI Assistant.

Supported regions

You can only use the Model Runtime feature if you are running the assistant in a supported region. Model Runtime is supported for Splunk AI Assistant users in the following regions:

  • AWS - US West Oregon

  • AWS - US East Virginia

  • AWS - Canada Central

  • AWS - EU London

  • AWS - EU Paris

  • AWS - EU Milan

  • AWS - EU Frankfurt

  • AWS - EU Dublin

  • AWS - AP Seoul
  • AWS - AP Singapore

  • AWS - AP Mumbai

  • AWS - AP Sydney

  • AWS - AP Tokyo

  • AWS - AP Jakarta

  • AWS - SA São Paulo

  • Azure - East US (Virginia)

  • Azure - UK South (London)

  • Azure - West US (California)

  • Azure - Japan East (Tokyo)

Model Runtime feature availability and region standard

See the following table for each supported region for Model Runtime, when the feature became available in that region, and the region standard. Region standard shows if your requests to the app might be processed outside the selected region.

Region standards are defined as follows:

  • Data zone standard: App requests can route to any region within the same zone. Provides zone-level routing only.

  • Global standard App requests can route anywhere in the world. Does not provide zone-level or country-level routing guarantees.

Region Feature availability Region standard
AWS - Canada Central Available as of v2.0.0 Global
AWS - AP Mumbai Available as of v2.0.0 Global
AWS - AP Seoul Available as of v2.2.0 Global
AWS - AP Singapore Available as of v2.2.0 Global
AWS - AP Sydney Available as of v2.0.0 Global
AWS - AP Tokyo Available as of v2.0.0 Global
AWS - AP Jakarta Available as of v2.3.0 Global
AWS - EU London Available as of v2.0.0 Global
AWS - EU Frankfurt Available as of v2.2.0 Data zone
AWS - EU Milan Available as of v2.2.0 Data zone
AWS - EU Paris Available as of v2.0.0 Data zone
AWS - EU Dublin Available as of v2.3.0 Global
AWS - US West Oregon Available as of v2.0.0 Data zone
AWS - US East Virginia Available as of v2.0.0 Data zone
AWS - SA São Paulo Available as of v2.2.0 Global
Azure - East US (Virginia) Available as of v1.4.0 N/A
Azure - UK South (London) Available as of v1.4.0 N/A
Azure - West US (California) Available as of v1.4.0 N/A
Azure - Japan East (Tokyo) Available as of v1.4.0 N/A

Using the Model Runtime feature

Splunk AI Assistant can leverage an external large language models (LLM) hosted in Azure OpenAI. This LLM generates the response provided by the app when deemed necessary, and can improve the response quality.

Splunk AI Assistant leverages the additional options from the LLM based on the intent and complexity of the request. The external LLM endpoint is secure but is outside the Splunk platform data boundary. The search prompt is sent to the third-party LLM and is governed by the third-party LLM provider's data handling policy.

The Model Runtime feature includes enterprise-grade compliance and regional data boundaries. Opting in causes no disruption to Splunk AI Assistant services or responsiveness.

When you opt-in, search responses are tagged with the source as being either internal, using the Splunk platform, or external, using the third-party LLM. Administrators can view these audit log tags as needed.

Opt in or out of the Model Runtime feature

When you install version 1.4.0 or higher of Splunk AI Assistant, you are opted in to this feature by default. You can opt out or back in at any time, and the change takes effect immediately. You must have administrator privileges to opt in or out of this feature.
Note: You must opt-in for Model Runtime if you want to use the Agent Mode feature. Agent Mode allows for agentic response generation in the assistant. For all other requirements see Agent Mode in Splunk AI Assistant.
If you want to opt-in or out of this feature, navigate to the Settings page of the assistant, and the General tab, as shown n the following image:
This image shows Splunk AI Assistant and configurable options on the Settings page, and General tab. The section for Model Runtime is highlighted.
Note: This setting applies at the app level and affects all users. It cannot be set at the individual user level.

When you select "Limit to Splunk-hosted models only", the Hosted model setting appears, as shown in the following image

This image is a drilled in view of the Model Runtime setting.

The Hosted model setting is not available when you select "Let Splunk determine the best model to deliver the outcome based on your prompt".

Select the Splunk-hosted model that you want Splunk AI Assistant to use. Choose Gemma 4 or GPT OSS 120B.

Use your own LLM with AI tier

If you use Splunk AI Assistant with an AI tier deployment, you can use an LLM connection configured in the Splunk AI Toolkit for Splunk AI Assistant inference.

Note: Bring your own LLM (BYOLLM) is supported with AI tier deployments. BYOLLM is not supported with Cloud Connected deployments.

Prerequisites

Before you begin, make sure the following requirements are met:

  • Splunk AI Assistant is configured to use AI tier.

  • An eligible LLM connection is configured in the Splunk AI Toolkit and shared with Splunk AI Assistant.

  • Your role has the admin_all_objects and list_ai_commander_config capabilities. The list_ai_commander_config capability can be inherited from the mltk_admin, sc_admin, mltk_dsdl_admin, or mltk_model_admin role.

Steps

Follow these steps to use an LLM connection with Splunk AI Assistant:

  1. In Splunk AI Assistant, go to Settings and then the General tab.

  2. Under Model Runtime, select Bring your own model configured in the Splunk AI Toolkit.

  3. Select an available LLM connection from the dropdown. The dropdown lists eligible connections shared with Splunk AI Assistant from the Splunk AI Toolkit.

Splunk AI Assistant then uses the selected connection for inference.