Connections in the AI Toolkit

Use the Connections options in the AI Toolkit to configure a connection with a large-language model (LLM) or container:

  • The LLM connection type lets you select your preferred LLM provider.

  • The Container connection type lets you choose between Docker and Kubernetes container types, which is used primarily for Splunk App for Data Science and Deep Learning (DSDL) cases.

Note: In AI Toolver version 5.7.0 and lower, of the toolkit, this tab is labeled as "Connection Management" and only includes the option to connect to an LLM.

Getting started

From the AI Toolkit app, navigate to the Connections tab and choose Connections from the drop-down menu.

From +Connection in the top right, select the Connection type of LLM or Container.

Select your preferred LLM provider or Container provider from the list view. Then complete the required fields for that provider on the resulting modal windows.

Adding an LLM connection

CAUTION: By using the LLM connection feature you acknowledge and agree that data from searches using the ai command is processed by an external LLM service provider.

LLM connection permissions

To select LLM as the Connection type, you must have the following capabilities from the mltk admin role:

  • apply_ai_commander_command

  • edit_ai_commander_config

  • list_ai_commander_config

To select Splunk hosted LLM from the LLM providers menu, you must have the following capability:

  • list_tokens_scs

After the connection has been configured, you can use the LLM with the ai command. For more information see About the ai command.

Note: The ai command does not inspect the input to the LLM. Use discretion to determine if the data you send to the LLM is suitable and appropriate.

Supported LLM providers

You can use the ai command with the following LLM providers:

Note: The Splunk hosted LLM option is available in AI Toolkit version 5.7.0 and higher and offers access to hosted language models to integrate LLM capabilities into your workflow. Available options are OpenAI GPT-OSS 120B, OpenAI GPT-OSS 20B, and Llama-3.1-FoundationAI-SecurityLLM-base-1.1-8B. You must have the list_tokens_scs capability to see this option.
  • Splunk hosted LLM

  • Custom LLM connection

  • OpenAI

  • Anthropic

  • AzureOpenAI
  • Groq
  • Gemini
  • Bedrock
  • Ollama

You can use the provider= or model= parameters in your ai command search to switch between these providers.

Bedrock configuration steps

When choosing Bedrock as the LLM provider some additional steps are required.

You must set up an AWS IAM Role and IAM User to integrate with Amazon Bedrock, and configure these credentials in the Connection Management page.

Note: Without taking these steps, when you select Bedrock as the LLM provider, you will not see any selectable options from the Model > Select Model drop-down menu.

Compete the following steps:

  1. Create an IAM Role with AmazonBedrockFullAccess Policy:

    1. Sign into your AWS account and navigate to the IAM Console - Roles page. See https://console.aws.amazon.com/iam/home#/roles

    2. Select Create Role.

    3. Select Trusted Entity Type as AWS account.

    4. Choose Another AWS account or your own account, as applicable.

    5. In the Permissions policies section, search and select AmazonBedrockFullAccess.

    6. Complete the role creation steps and note down the Role ARN as shown in the following example: arn:aws:iam::<account_id>:role/mltk-bedrock-fullaccess-role

      The following image shows an example view of a completed create roles page:

      This image shows an example of an AWS account page and role information.

  2. Create IAM User with Assume Role Permissions:

    1. In your AWS account navigate to the IAM Console - Users page. See https://console.aws.amazon.com/iam/home#/users

    2. Select Add users.

    3. Enable Access key – Programmatic access.

    4. In the Permissions step, choose Attach policies directly and select no policies.

    5. Complete the user creation. Make note of the Access Key ID and Secret Access Key.

    6. Edit the user to attach the following inline policy to allow assume-role access:

      JSON 
      {    
"Version": "2012-10-17",    
"Statement": [        
{            
"Sid": "Statement1",            
"Effect": "Allow",            
"Action": "sts:AssumeRole",            
"Resource": "arn:aws:iam::<account_id>:role/mltk-bedrock-fullaccess-role"        
}    
]
}

    7. Replace <account_id> with your actual AWS account ID. The following image shows an example view of the completed permissions page:

      This image shows an example of an AWS account page and permissions information.

    8. Add the newly created user to the Trust relationships and the role created in step 1.

Adding a Container connection

When you select Container you can create a Kubernetes or Docker external runtime connection, or enable HPA.

Note: By using the container connection feature you acknowledge and agree that data from searches initiated from this deployment will be stored and processed in an external container service provider.

Container connection permissions

To select Container as the Connection type, you must have the following capability from the mltk admin role:

  • list_ai_commander_config

To select Container as the Connection type, you must have the following capabilities from the dsdl_admin role:
Note: These capabilities allow you to define and configure external run times. For example creating a Kubernetes cluster definition in the AI Toolkit.

  • list_container_connections

  • setup_container_configuration

  • enable_hpa

To run the fit and apply commands for a specific AI Toolkit container you must have the following capabilities from the dsdl_admin role:

  • fit_mltkcontainer

  • apply_mltkcontainer