Model Runtime in Splunk AI Assistant

Version 1.4.0 and higher of Splunk AI Assistant provides the option to use the large language models (LLMs) hosted in Splunk Cloud Platform or models hosted in Azure OpenAI.

When you use the Model Runtime feature, Splunk AI Assistant determines when to use a Splunk platform hosted LLM, and when to use a third-party LLM. Third-party LLMs can provide better response quality through the assistant, depending on factors such as use case and cost.
Note: Users on IL 2 FedRAMP deployments can only choose the Splunk-hosted model option.
Splunk AI Assistant version 1.4.0 or higher uses Model Runtime by default. Administrators can turn off this functionality at any time from the Settings page.
Note: You must opt-in to Model Runtime if you want to use the Agent Mode feature. To learn more, including Agent Mode requirements, see Agent Mode in Splunk AI Assistant.

Supported regions

You can only use the Model Runtime feature if you are running the assistant in a supported region. Model Runtime is supported for Splunk AI Assistant users in the following regions:

  • AWS - Canada Central

  • AWS - AP Mumbai

  • AWS - AP Sydney

  • AWS - AP Tokyo

  • AWS - EU London

  • AWS - EU Paris

  • AWS - US West Oregon

  • AWS - US East Virginia

  • AWS - EU Milan

  • AWS - EU Frankfurt
  • AWS - AP Seoul
  • AWS - AP Singapore
  • AWS - SA São Paulo

  • Azure - East US (Virginia)

  • Azure - UK South (London)

  • Azure - West US (California)

  • Azure - Japan East (Tokyo)

Model Runtime feature availability and region standard

See the following table for each supported region for Model Runtime, when the feature became available in that region, and the region standard. Region standard shows if your requests to the app might be processed outside the selected region.

Region standards are defined as follows:

  • Data zone standard: App requests can route to any region within the same zone. Provides zone-level routing only.

  • Global standard App requests can route anywhere in the world. Does not provide zone-level or country-level routing guarantees.

Region Feature availability Region standard
AWS - Canada Central Available as of v2.0.0 Global
AWS - AP Mumbai Available as of v2.0.0 Global
AWS - AP Seoul Available as of v2.2.0 Global
AWS - AP Singapore Available as of v2.2.0 Global
AWS - AP Sydney Available as of v2.0.0 Global
AWS - AP Tokyo Available as of v2.0.0 Global
AWS - EU London Available as of v2.0.0 Global
AWS - EU Frankfurt Available as of v2.2.0 Data zone
AWS - EU Milan Available as of v2.2.0 Data zone
AWS - EU Paris Available as of v2.0.0 Data zone
AWS - US West Oregon Available as of v2.0.0 Data zone
AWS - US East Virginia Available as of v2.0.0 Data zone
AWS - SA São Paulo Available as of v2.2.0 Global
Azure - East US (Virginia) Available as of v1.4.0 N/A
Azure - UK South (London) Available as of v1.4.0 N/A
Azure - West US (California) Available as of v1.4.0 N/A
Azure - Japan East (Tokyo) Available as of v1.4.0 N/A

Using the Model Runtime feature

Splunk AI Assistant can leverage an external large language models (LLM) hosted in Azure OpenAI. This LLM generates the response provided by the app when deemed necessary, and can improve the response quality.

Splunk AI Assistant leverages the additional options from the LLM based on the intent and complexity of the request. The external LLM endpoint is secure but is outside the Splunk platform data boundary. The search prompt is sent to the third-party LLM and is governed by the third-party LLM provider's data handling policy.

The Model Runtime feature includes enterprise-grade compliance and regional data boundaries. Opting in causes no disruption to Splunk AI Assistant services or responsiveness.

When you opt-in, search responses are tagged with the source as being either internal, using the Splunk platform, or external, using the third-party LLM. Administrators can view these audit log tags as needed.

Opt in or out of the Model Runtime feature

When you install version 1.4.0 or higher of Splunk AI Assistant, you are opted in to this feature by default. You can opt out or back in at any time, and the change takes effect immediately. You must have administrator privileges to opt in or out of this feature.
Note: You must opt-in for Model Runtime if you want to use the Agent Mode feature. Agent Mode allows for agentic response generation in the assistant. For all other requirements see Agent Mode in Splunk AI Assistant.
If you want to opt-in or out of this feature, navigate to the Settings page of the assistant, and the General tab, as shown n the following image:
This image shows Splunk AI Assistant and configurable options on the Settings page, and General tab. The section for Model Runtime is highlighted.
Note: This setting applies at the app level and affects all users. It cannot be set at the individual user level.