Model Runtime in Splunk AI Assistant

Version 1.4.0 and higher of Splunk AI Assistant provides the option to use the large language models (LLMs) hosted in Splunk Cloud Platform or models hosted in Azure OpenAI.

When you use the Model Runtime feature, Splunk AI Assistant determines when to use a Splunk platform hosted LLM, and when to use a third-party LLM. Third-party LLMs can provide better response quality through the assistant, depending on factors such as use case and cost.
Note: Users on IL 2 FedRAMP deployments can only choose the Splunk-hosted model option.
Splunk AI Assistant version 1.4.0 or higher uses Model Runtime by default. Administrators can turn off this functionality at any time from the Settings page.
Note: You must opt-in to Model Runtime if you want to use the Agent Mode feature. To learn more, including Agent Mode requirements, see Agent Mode in Splunk AI Assistant.

Requirements

Model Runtime is supported for Splunk AI Assistant users in the following regions:

  • AWS - Canada Central

  • AWS - AP Mumbai

  • AWS - AP Sydney

  • AWS - AP Tokyo

  • AWS - EU London

  • AWS - EU Dublin

  • AWS - EU Paris

  • AWS - US West Oregon

  • AWS - US East Virginia

  • Azure - East US (Virginia)

  • Azure - UK South (London)

  • Azure - West US (California)

  • Azure - Japan East (Tokyo)

Using the Model Runtime feature

Splunk AI Assistant can leverage an external large language models (LLM) hosted in Azure OpenAI. This LLM generates the response provided by the app when deemed necessary, and can improve the response quality.

Splunk AI Assistant leverages the additional options from the LLM based on the intent and complexity of the request. The external LLM endpoint is secure but is outside the Splunk platform data boundary. The search prompt is sent to the third-party LLM and is governed by the third-party LLM provider's data handling policy.

The Model Runtime feature includes enterprise-grade compliance and regional data boundaries. Opting in causes no disruption to Splunk AI Assistant services or responsiveness.

When you opt-in, search responses are tagged with the source as being either internal, using the Splunk platform, or external, using the third-party LLM. Administrators can view these audit log tags as needed.

Opt in or out of the Model Runtime feature

When you install version 1.4.0 or higher of Splunk AI Assistant, you are opted in to this feature by default. You can opt out or back in at any time, and the change takes effect immediately. You must have administrator privileges to opt in or out of this feature.
Note: You must opt-in for Model Runtime if you want to use the Agent Mode feature. Agent Mode allows for agentic response generation in the assistant. For all other requirements see Agent Mode in Splunk AI Assistant.
If you want to opt-in or out of this feature, navigate to the Settings page of the assistant, and the General tab, as shown n the following image:
This image shows Splunk AI Assistant and configurable options on the Settings page, and General tab. The section for Model Runtime is highlighted.
Note: This setting applies at the app level and affects all users. It cannot be set at the individual user level.