Model Runtime in Splunk AI Assistant for SPL

In addition to the large language models (LLMs) hosted in Splunk Cloud Platform, version 1.4.0 and higher of Splunk AI Assistant provides the option to use models hosted in Azure OpenAI.

Splunk AI Assistant determines when to use a Splunk platform hosted LLM, and when to use a third-party LLM. Third-party LLMs can provide better response quality through the assistant, depending on factors such as use case and cost.

When you install or upgrade to version 1.4.0 or higher, you are opted-in to Model Runtime by default. Administrators can turn off this functionality at any time from the Settings page.

Note: You must opt-in to Model Runtime if you want to use the Agent Mode feature that shipped with version 2.0.0. To learn more, including Agent Mode requirements, see Agent Mode in Splunk AI Assistant.

Requirements

Model Runtime functionality is available to Splunk Cloud platform users in the following regions:

  • AWS - Canada Central

  • AWS - AP Mumbai

  • AWS - AP Sydney

  • AWS - AP Tokyo

  • AWS - EU London

  • AWS - EU Paris

  • AWS - US West Oregon

  • AWS - US East Virginia

  • Azure - East US (Virginia)

  • Azure - UK South (London)

  • Azure - West US (California)

  • Azure - Japan East (Tokyo)

Opt in or out of the Model Runtime feature

When you install version 1.4.0 or higher of Splunk AI Assistant, you are opted in to this feature by default. You can opt out or back in at any time, and the change takes effect immediately. You must have administrator privileges to opt in or out of this feature.
Note: You must opt-in for Model Runtime if you are a Splunk Cloud customer in a supported AWS region and you want to use the new feature available in Splunk AI Assistant version 2.0.0 of Agent Mode.
If you want to opt-in or out of this feature, navigate to the Settings page of the assistant, and the General tab, as shown n the following image:
This image shows Splunk AI Assistant and configurable options on the Settings page, and General tab. The section for Model Runtime is highlighted.
Note: This setting applies at the app level and affects all users. It cannot be set at the individual user level.

Using the Model Runtime feature

Splunk AI Assistant can leverage an external large language models (LLM) hosted in Azure OpenAI. This LLM generates the response provided by the app when deemed necessary, and can improve the response quality.

Splunk AI Assistant leverages the additional options from the LLM based on the intent and complexity of the request. The external LLM endpoint is secure but is outside the Splunk platform data boundary. The search prompt is sent to the third-party LLM and is governed by the third-party LLM provider's data handling policy.

This feature includes enterprise-grade compliance and regional data boundaries. Opting in causes no disruption to Splunk AI Assistant services or responsiveness.

When you opt-in, search responses are tagged with the source as being either internal, using the Splunk platform, or external, using the third-party LLMs. Administrators can view these audit log tags as needed.