Upload and inference a pre-trained Vertex AI model in the AI Toolkit

Version 5.7.4 of the AI Toolkit introduces a new external model option of Vertex AI. The Vertex AI endpoint integration lets AI Toolkit users invoke Google Cloud Platform (GCP) Vertex AI-hosted online prediction endpoints directly from Splunk searches, dashboards, and alerts, bringing model predictions into Splunk platform workflows using the familiar ML-SPL apply command.

Using Vertex AI models follows this high-level workflow:

  • Register an external endpoint as an AI Toolkit model

  • Validate endpoint connectivity

  • Invoke the external endpoint through the SPL command of apply

  • Map predictions back into Splunk results

Pro-code users can operationalize advanced ML workloads within the Splunk platform while leveraging VertexAI's managed infrastructure for scalable inference. This eliminates GPU, CPU, and Python library limitations, allowing for inference on large, complex, or custom ML models hosted in AWS, without overloading the search head.

Note: Vertex AI models follow the same permission rules as other models you create in the AI Toolkit.

Vertex AI model permissions

See the following table the permissions needed to perform Vertex AI model feature operations:

Note: All users can run inference on registered models. Users without the edit_endpoints capability can run models but cannot register new models.
Vertex AI model inference operation Required permissions
Edit, create, test, and delete edit_endpoints, edit_storage_passwords, and list_storage_passwords
Use the apply command to invoke the Vertex AI model Search permissions and list_storage_passwords

Vertex AI model requirements

AI Toolkit uses the provided Google Cloud Platform (GCP) service account JSON to authenticate to Google Cloud, validate the endpoint, and send inference requests:

  • End users running the apply command do not directly authenticate to GCP.

  • Splunk users do not need their own GCP account or personal Google login.

You must meet the following requirements to use the Vertex AI model feature:

Requirement Description
A Google Cloud project where Vertex AI is set up A Vertex AI model must already be deployed to a Vertex AI online prediction endpoint.
A service account that the AI Toolkit can use to call the Vertex AI endpoint The service account needs the following IAM permissions:

  • aiplatform.endpoints.get for Test Connection

  • aiplatform.endpoints.predict for inference and apply

Vertex AI model syntax

Calling a Vertex AI model uses the following SPL syntax:

CODE 
…| apply <vertex_model_name> runtime="vertex"

Vertex AI mappings, OpenAPI specs, and sample SPL

AI Toolkit supports batch-style mappings using the wildcard [*] and single-record mappings without the wildcard [*]. Batch-style mappings are supported for both batch_size=1 and batch_size>1.

Overview

See the following overview for Vertex AI model mapping patterns:

  • JSON endpoints require non-empty input and output maps because the AI Toolkit needs explicit JSON schema paths.

  • CSV endpoints can use {} maps because the payload is a positional or headerless CSV.

  • CSV input_feature_map is still useful when users want to select and order specific fields for the CSV request.

  • CSV output_prediction_map is useful when users want output field names that are not auto-generated names such as predictions_0.

JSON models

For application/json models, both feature maps are required:

  • The input_feature_map tells the AI Toolkit how to build the JSON request body from Splunk fields.

  • The output_prediction_map tells the AI Toolkit which response values should become Splunk output fields.

  • The map paths must match the OpenAPI request and response schema.

JSON input feature map

See the following code block:
JSON 
{
  "square_feet": "instances[*].square_feet",
  "bedrooms": "instances[*].bedrooms",
  "bathrooms": "instances[*].bathrooms",
  "age_years": "instances[*].age_years",
  "distance_to_city_km": "instances[*].distance_to_city_km"
}
This code builds a Vertex request such as the following:
JSON 
{
  "instances": [
    {
      "square_feet": 1650,
      "bedrooms": 3,
      "bathrooms": 3,
      "age_years": 2,
      "distance_to_city_km": 3.6
    }
  ]
}

JSON output prediction map

See the following code block:
JSON 
{
  "predictions[*].scores[0]": "predicted_price"
}
This code maps predictions[*].scores[0] from the Vertex response into a Splunk field named predicted_price.

JSON OpenAPI spec

See the following example of a JSON OpenAPI spec:
JSON 
{
  "openapi": "3.0.0",
  "info": {
    "title": "Vertex house price regression",
    "version": "1.0.0"
  },
  "paths": {
    "/invocations": {
      "post": {
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "type": "object",
                "properties": {
                  "instances": {
                    "type": "array",
                    "items": {
                      "type": "object",
                      "properties": {
                        "square_feet": { "type": "number" },
                        "bedrooms": { "type": "number" },
                        "bathrooms": { "type": "number" },
                        "age_years": { "type": "number" },
                        "distance_to_city_km": { "type": "number" }
                      },
                      "required": [
                        "square_feet",
                        "bedrooms",
                        "bathrooms",
                        "age_years",
                        "distance_to_city_km"
                      ]
                    }
                  }
                },
                "required": ["instances"]
              }
            }
          }
        },
        "responses": {
          "200": {
            "description": "Prediction response",
            "content": {
              "application/json": {
                "schema": {
                  "type": "object",
                  "properties": {
                    "predictions": {
                      "type": "array",
                      "items": {
                        "type": "object",
                        "properties": {
                          "scores": {
                            "type": "array",
                            "items": { "type": "number" }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

JSON sample SPL

See the following JSON sample SPL:
CODE 
| makeresults count=2
| streamstats count as row
| eval square_feet=if(row=1,1650,950)
| eval bedrooms=if(row=1,3,2), bathrooms=if(row=1,3,1)
| eval age_years=if(row=1,2,15), distance_to_city_km=if(row=1,3.6,11.0)
| table square_feet bedrooms bathrooms age_years distance_to_city_km
| apply vertex_test_demo runtime=vertex
| table square_feet bedrooms bathrooms age_years distance_to_city_km predicted_price

Supported JSON request mapping patterns

Supported JSON request mapping patterns include batch named object mapping, batch positional array mapping, batch nested object mapping, batch nested positional mapping, single named object mapping, single nested object mapping, and single positional array mapping.

See the following table for mapping and payload examples:

Example Mapping Payload
Batch named object mapping
JSON 
{
  "square_feet": "instances[*].square_feet",
  "bedrooms": "instances[*].bedrooms"
}
JSON 
{
  "instances": [
    { "square_feet": 1650, "bedrooms": 3 },
    { "square_feet": 950, "bedrooms": 2 }
  ]
}
Batch positional array mapping
JSON 
{
  "a": "instances[*][0]",
  "b": "instances[*][1]"
}
 Payload for one row with batch_size=1:
JSON 
{
  "instances": [[1, 2]]
}
Batch nested object mapping
JSON 
{
  "a": "instances[*].x.a",
  "b": "instances[*].y.b"
}
 Payload for one row with batch_size=1:
JSON 
{
  "instances": [
    {
      "x": { "a": 1 },
      "y": { "b": 2 }
    }
  ]
}
Batch nested positional mapping
JSON 
{
  "a": "instances[*].metrics[0]",
  "b": "instances[*].metrics[1]"
}
 Payload for one row with batch_size=1:
JSON 
{
  "instances": [
    {
      "metrics": [1, 2]
    }
  ]
}
Single named object mapping
JSON 
{
  "container_cpu": "container_cpu",
  "container_memory": "container_memory"
}
JSON 
{
  "container_cpu": 0.82,
  "container_memory": 1.2
}
Single nested object mapping
JSON 
{
  "cpu_spike_percent": "system.cpu_spike_percent",
  "avg_cpu_usage": "application.avg_cpu_usage"
}
JSON 
{
  "system": { "cpu_spike_percent": 45 },
  "application": { "avg_cpu_usage": 0.31 }
}
Single positional array mapping
JSON 
{
  "query_complexity": "[0]",
  "table_rows": "[1]",
  "index_count": "[2]"
}
CODE 
[0.7, 1500000, 8]
Batch mapping with a non-instances parent key
JSON 
{
  "throughput": "data[*][0]",
  "latency": "data[*][1]"
}
JSON 
{
  "data": [
    [200, 12.0],
    [120, 30.0]
  ]
}
AI Toolkit also supports object arrays under a non-instances parent key
JSON 
{
  "req_rate": "inputs[*].req_rate",
  "err_rate": "inputs[*].err_rate",
  "p95": "inputs[*].latency.p95"
}
JSON 
{
  "inputs": [
    { "req_rate": 1200, "err_rate": 0.02, "latency": { "p95": 180 } },
    { "req_rate": 900, "err_rate": 0.05, "latency": { "p95": 250 } }
  ]
}

Supported JSON response mapping patterns

Supported JSON response mapping patterns include scalar arrays, two-dimensional arrays, object arrays, nested object fields, nested array fields, root arrays, single scalar responses, and single nested responses.

See the following table for mapping and payload examples:

Example Mapping Response
Scalar array response mapping
JSON 
{
  "predictions[*]": "prediction"
}
JSON 
{
  "predictions": [0.1, 0.2]
}
Two-dimensional array response mapping
JSON 
{
  "predictions[*][0]": "score",
  "predictions[*][1]": "confidence"
}
JSON 
{
  "predictions": [
    [0.1, 0.9],
    [0.2, 0.8]
  ]
}
Object array response mapping
JSON 
{
  "predictions[*].class": "predicted_class",
  "predictions[*].prob": "confidence"
}
JSON 
{
  "predictions": [
    { "class": "NORMAL", "prob": 0.81 },
    { "class": "AT_RISK", "prob": 0.62 }
  ]
}
Nested response mapping
JSON 
{
  "predictions[*].scores[0]": "score_0",
  "predictions[*].scores[1]": "score_1"
}
JSON 
{
  "predictions": [
    { "scores": [0.25, 0.75] },
    { "scores": [0.60, 0.40] }
  ]
}
Single nested response mapping
JSON 
{
  "score.value": "risk_score"
}
JSON 
{
  "score": {
    "value": 0.73
  }
}

CSV models

For text/csv models, feature maps are optional.

  • Using an input_feature_map is optional. If provided, the AI Toolkit uses its keys to select and order the Splunk fields before generating the CSV body.

  • If input_feature_map is empty {}, the AI Toolkit sends the current SPL fields in their DataFrame order. Use fields or table before apply to control the order.

  • Using an output_prediction_map is optional. If provided, its values are used in order as output column names.

  • If output_prediction_map is empty {}, the AI Toolkit generates output field names such as predictions_0, predictions_1, and so on.

  • CSV request and response bodies are headerless.

CSV input feature map - Option 1: Empty Map

With an empty input map {}, control request column order in SPL:
CODE 
| fields a b c
The AI Toolkit sends the following:
CODE 
1,2,3
4,5,6

CSV input feature map - Option 2: Explicit Ordering

For CSV, the the map key order is important: a, b, then c. The AI Toolkit uses those keys to select and order the DataFrame columns before writing headerless CSV:
JSON 
{
  "a": "csv_col_0",
  "b": "csv_col_1",
  "c": "csv_col_2"
}

CSV output prediction map

For CSV responses, the map keys are not interpreted as response paths. The map values are used in order as output field names:
JSON 
{
  "csv_col_0": "score",
  "csv_col_1": "n_fields"
}
If the endpoint returns the following:
CODE 
6,3
15,3
Then the AI Toolkit maps that to the following:
CODE 
score, n_fields

CSV OpenAPI spec

See the following example of a CSV OpenAPI spec:
JSON 
{
  "openapi": "3.0.0",
  "info": {
    "title": "Vertex CSV endpoint",
    "version": "1.0.0"
  },
  "paths": {
    "/invocations": {
      "post": {
        "requestBody": {
          "required": true,
          "content": {
            "text/csv": {
              "schema": {
                "type": "string"
              }
            }
          }
        },
        "responses": {
          "200": {
            "description": "CSV prediction response",
            "content": {
              "text/csv": {
                "schema": {
                  "type": "string"
                }
              }
            }
          }
        }
      }
    }
  }
}

CSV sample SPL

See the following CSV sample SPL:
CODE 
| makeresults count=2
| streamstats count as row
| eval a=if(row=1,1,4), b=if(row=1,2,5), c=if(row=1,3,6)
| fields a b c
| apply vertex_test_csv runtime=vertex
| table a b c score n_fields
This sample uses the multi-payload Vertex test model registered as vertex_test_csv.
The expected request body is as follows:
CODE 
1,2,3
4,5,6
The expected response body is as follows:
CODE 
6,3
15,3

Vertex AI model configuration steps

Configuration is a one-time, secure setup that uses IAM roles with no exposed credentials.

Complete the following steps to upload and inference a new Vertex AI model:
Note: Fields marked with an asterisk are required.

  1. Log into the AI Toolkit and navigate to the Models tab and choose Models from the drop-down menu.

  2. From the +Model button, choose Vertex AI.

  3. Input model information:

    1. Add a model name. Model names must meet the following criteria:

      1. Name must start with a letter or underscore. Name cannot start with a number.

      2. After that, name can contain only, letters, numbers, and underscores.

      3. Spaces are not allowed.

      4. Special characters are not allowed, including hyphens, periods, slashes, and colons.

      5. The model name must be unique among registered models.

    2. Optionally add a model description.

  4. Input your GCP credentials:

    1. GCP project ID: Enter the unique string used that identifies your project across all Google Cloud services. This ID is located on your Google Cloud Console dashboard.

    2. GCP region: The geographic area that hosts your cloud resources and services. This region is named on your Google Cloud Console dashboard .

    3. Vertex endpoint ID: This ID is located on your Google Cloud Console dashboard .

    4. Service account JSON: The credentials used for authentication. Contains a private key that lets an application prove its identity to Google APIs and services. Must be provided in JSON format.

  5. Select Test connection to validate the GCP project, region, endpoint ID, and service account credentials. A message appears to confirm the test is successful or not.

    1. If your connection fails, make appropriate edits to the fields and test the connection again.

  6. Input feature mapping: Maps Splunk input fields to request schema locations.

  7. Output feature mapping: Maps response values to Splunk output fields.

  8. Open API spec for inference endpoint: The endpoint request and response content types and schemas.

    • The Open API spec is required for both JSON and CSV models.

    • The OpenAPI spec must be OpenAPI version 3.0.x.

    • The spec must define a /invocations path with a post operation.

    • The request content type in the OpenAPI spec determines what the AI Toolkit sends to Vertex AI.

    • Supported content types are application/json and text/csv.

  9. Choose the Batch Size. Must be an integer. Batch size is the number of rows sent with each inference request. Default is 1 and the maximum is 10,000.

  10. Select Add Model. Once added, Vertex AI models are listed on the Models tab including model details such as algorithm, feature variables, and target fields.

Edit a Vertex AI model

You can edit your stored Vertex AI models to update inference mappings, OpenAPI schema, and batch size.

Complete the following steps:

  1. From the Models tab of the AI Toolkit app, select the Models option. from the list.

  2. From the Actions column, select Edit on the same row of the model you want to edit.

  3. On the resulting model details window, edit the Model name, Input feature mapping, Output feature mapping, Open API spec for inference endpoint, or Batch size.

  4. Select Save when done.