Python 3 migration with ITSI

In this topic

• 1. Plan the migration
• 2. Validate and upgrade dependencies
• 3. Validate custom alert actions and Python code
• 4. Perform the upgrade
• 5. Retrain Predictive Analytics models
• 6. Validate the environment after the upgrade

1. Plan the migration

Treat the Splunk platform, ITSI, related apps and add-ons, custom Python code, and external integrations as one upgrade program. Complete the following sections in order.

What changed and what can break

• ITSI runtime: ITSI 5.0 requires Python 3.13. Custom alert actions, scripts that import ITSI or ITOA modules, and code that depends on older Python behavior can fail.

• App extension points: Alert actions, modular inputs, custom search commands, scripted lookups, REST endpoints, authentication scripts, and Splunk Web components can select or inherit an incompatible interpreter.

• Custom alert-action interfaces: Legacy HTML controls can fail to render or save parameters correctly and might require migration to supported Splunk custom elements.

• Python libraries: Bundled or third-party libraries such as requests, urllib3, and boto3 can introduce missing-module or API-compatibility failures.

• Predictive Analytics: AI Toolkit or Python for Scientific Computing (PSC) changes can affect custom algorithms and saved models.

• Related apps and integrations: Incompatible content packs, ticketing add-ons, private apps, webhooks, and scripted actions can break data inputs or episode workflows.

Confirm the target and upgrade path

• Record the current ITSI, Splunk platform, and related-app versions.

• Use the Splunk products version compatibility matrix to select the supported Splunk platform release for ITSI 5.0.

• ITSI 4.19.x, 4.20.x, and 4.21.x support a direct upgrade to ITSI 5.0.x. Earlier releases require step upgrades. Confirm the path in the current upgrade matrix before scheduling the change.

• Within a supported Splunk Enterprise such as 10.2.x or 10.4.x, use the compatibility matrix to identify any stated maintenance-release exception.

• For Splunk Cloud Platform, open a Support case and coordinate the platform, ITSI, and private-app changes.

Prepare backups and rollback

Complete the ITSI pre-upgrade requirements. At minimum, create a full ITSI backup, back up the KV store, generate a diagnostic file, and preserve custom app packages, local configuration, custom algorithms, and model inputs required for recovery.

Prepare the environment

• Run the ITSI Upgrade Readiness checks and resolve blocking errors.

• Confirm that no service templates are syncing.

• Review the ITSI 5.0 release notes, known issues, removed features, and version-specific upgrade notes.

• Confirm whether the deployment is a single instance, distributed deployment, or search head cluster, and include every affected component in the change plan.

Inventory custom components and immediate ITSI dependencies

Inventory custom alert actions, ITSI or ITOA imports, direct additions of SA-ITOA/lib to sys.path, modular inputs, custom search commands, scripted lookups, REST endpoints, authentication scripts, Splunk Web components, packaged libraries, and PSC-based custom algorithms.

Use enabled notable event aggregation policies (NEAPs) as an initial dependency source:

1. In ITSI, select Configuration > Event Management > Notable Event Aggregation Policies .

2. Open each enabled policy and review its Action Rules.

3. Map each external, scripted, modular, or custom action to the app or add-on that provides it. Record the app ID, installed version, owner, and declared Python runtime.

4. Add each provider to the dependency inventory and validate.

This is a starting point, not a complete scan. It does not identify actions in disabled policies, manually invoked episode actions, or Python extension points outside NEAPs. See Overview of aggregation policies in ITSI.

Review TLS and OpenSSL compatibility

If the target platform is Splunk Enterprise 10.4 or Splunk Cloud Platform 10.4.2604 or later, TLS 1.3 is available and active alongside TLS 1.2 by default. ITSI 5.0 has been validated in TLS 1.3-only and mixed TLS 1.2/TLS 1.3 configurations across Splunk Enterprise, Splunk Cloud Platform, IPv6, and FIPS environments. Before the upgrade, verify that forwarders, ticketing systems, webhooks, proxies, custom alert actions, and other connected clients can negotiate a common TLS version and use supported certificates and cipher suites. After the platform upgrade, verify HEC event ingestion, Rules Engine and NATS health, certificate generation and rotation, and peer communication where applicable. Investigate protocol-version, cipher-suite, certificate, timeout, or readiness errors before upgrading ITSI. Do not restrict connections to TLS 1.3 until every peer supports it.

See Configure TLS protocol version support.

2. Validate and upgrade dependencies

Upgrade the Splunk platform to a release supported by ITSI 5.0, then upgrade every required dependent app or add-on to a compatible release before upgrading ITSI. Keep the dependency upgrades and the ITSI upgrade in the coordinated maintenance window.

Version source: ITSI 5.0 maintenance releases can change dependency requirements. Use the ITSI 5.0 related-app compatibility table and the product documentation linked below as the source of truth at upgrade time.

Splunk Cloud Platform: run CMC and private-app checks

1. In Splunk Web, select Apps > Cloud Monitoring Console > Health.

2. Review the health check for apps that are not configured for Python 3.13 compatibility. Also review Python version usage and Third-Party App Compatibility.

3. For every warning or critical result, select View details and record the app, status, and recommended action.

CMC is deployment-wide and does not replace the NEAP, custom-code, or private-app inventory. For private apps, cloud vetting is mandatory and uses Splunk AppInspect. An app that fails the applicable checks is not approved for installation until the reported issues are fixed or handled through the supported review process. For customer-managed Splunk Enterprise, AppInspect is recommended for quality validation but is not an installation requirement.

See Use the Health dashboard, Cloud Monitoring Console release notes, and Vet apps and add-ons for Splunk Cloud Platform.

3. Validate custom alert actions and Python code

Migrate legacy alert-action configuration interfaces

If a private app provides a legacy HTML configuration view for a custom alert action, migrate the view before upgrading to ITSI 5.0. This prevents missing controls or labels and helps ensure that saved parameter values continue to reach the action. The detailed procedure covers UCC-generated and manual migrations, deployment, and validation.

For the ITSI-specific procedure, see Migrate custom alert action HTML for ITSI 5.0

For the underlying platform component reference, see Splunk Developer Program

Declare the supported Python runtime

For each custom alert-action stanza that executes a Python script and has been validated on Python 3.13, add the following setting:

python.required = 3.13

Use python.required in every applicable configuration stanza, not only alert_actions.conf. See Specify Python compatibility and the target release's configuration-file reference.

Use Python 3.9 only as a temporary app exception

ITSI 5.0 still requires Python 3.13. If a third-party or Splunkbase app is not yet compatible with Python 3.13, a supported Splunk platform release can run an applicable extension point with python.required = 3.9 when Python 3.9 is still bundled and the app vendor supports that configuration. Place the override in the app's local directory; do not edit default.

Python 3.9 is deprecated in Splunk Enterprise 10.4. Upgrade or replace the app according to the vendor's published compatibility guidance and Splunk's Python migration guidance. This exception does not apply to Splunk Web components, which use the latest bundled interpreter.

Test the effective configuration and behavior

• Use $SPLUNK_HOME/bin/splunk btool alert_actions list --debug as the source of truth for the merged alert-action configuration and the file that supplies each value.

• Test custom code and packaged dependencies with Python 3.13 in a non-production environment.

• Trigger every custom alert action and verify its configuration UI, parameter values, external connection, result, and logs.

• Test modular inputs, custom search commands, scripted lookups, REST endpoints, authentication scripts, and supported ITSI or ITOA imports that are in scope.

• Review logs for syntax, import, dependency, TLS, and runtime errors.

4. Perform the upgrade

Customer-managed Splunk Enterprise

1. Complete the ITSI prerequisites, backups, Upgrade Readiness checks, and dependency inventory.

2. Upgrade the Splunk platform to the selected ITSI-compatible release.

3. Upgrade all required dependent apps and add-ons to the validated versions.

4. Upgrade ITSI by following Upgrade ITSI on a single instance or Upgrade ITSI in a search head cluster.

5. Deploy the required ITSI components to indexers, heavy forwarders, and the license manager as directed by the selected procedure.

6. Complete the version-specific post-installation tasks.

Splunk Cloud Platform

1. Work with Splunk Support to schedule and manage the ITSI upgrade.

2. Provide the current and target versions, topology, CMC results, private-app inventory, and ITSI dependency inventory.

3. Complete private-app vetting before the maintenance window.

5. Retrain Predictive Analytics models

If the deployment uses Predictive Analytics, confirm before the upgrade that the training data remains available. After upgrading AI Toolkit or PSC, re-import backed-up custom algorithms as required, retrain models that use upgraded PSC libraries, and retrain the Predictive Analytics models saved in ITSI services. Retraining replaces the existing model.

6. Validate the environment after the upgrade

• Confirm that ITSI reports a successful upgrade and verify the installed ITSI and Splunk platform versions.

• Run index=_internal "[itsi.migration]" and resolve migration errors.

• Review the ITSI Health Check dashboard, including ITSI Migration Status, ITSI Upgrade Readiness, Basic ITSI Information, and KV Store Collections.

• Confirm the Rules Engine is running and that notable events continue to be grouped into episodes.

• Test custom alert actions, episode actions, ticketing integrations, webhooks, ITSI or ITOA imports, content-pack objects, saved searches, and data inputs that are in scope.

• Validate Predictive Analytics models after retraining.

• Review logs for Python, module, dependency, TLS, NATS, Rules Engine, and runtime errors.

• Clear the Splunk Web browser cache if pages do not load correctly.

• Record results against the change plan's acceptance criteria before closing the maintenance window.