Plan ITSI Upgrade and KV Store Migration

CAUTION:

If you plan to upgrade Splunk Enterprise with ITSI to 10.6, upgrade ITSI to version 5.0.x first and schedule a maintenance window. Enterprise 10.6 automatically migrates the cohosted KV Store to version 1.0, which uses PostgreSQL. ITSI 5.0.x and earlier cannot use cohosted KV Store v1.0. Postpone the automatic migration until your ITSI version supports this KV Store version. The KV Store is read-only during migration; see the impact table below for details about ITSI features that can be affected. Follow the Enterprise instructions to postpone the migration and check the ITSI compatibility with related apps and add-ons for your version and deployment. Postponing migration does not confirm that ITSI 5.0.x is supported with Splunk Enterprise 10.6. Check support for your deployment before upgrading. The ITSI/ITEW Compatibility Matrix lists ITSI 5.0.x with Splunk Enterprise 10.6.x in KV Store mode. This support does not include migration to cohosted KV Store version 1.0.

Impact when KV Store is read-only

The following table describes potential impacts to ITSI features and recovery checks while KV Store is read-only during migration.

Table 1. ITSI impact and recovery checks
ITSI area Potential impact Impact rating Recovery and validation
Event Analytics and Rules Engine Events might be grouped or indexed even if ITSI does not save the related episode data. Critical Backfill might not restore events if KV Store writes fail. Check episode data and test restart behavior.
Episode actions and queues An external action might run even if ITSI cannot update or remove its queue entry. This can cause duplicate or missed actions. Critical Inspect the queue and the system that received the action before retrying it. Look for stuck, duplicate, or missing actions before you restart queue consumers.
KPI processing KPI data and summary cache might not update while searches continue to run. High Recalculate or backfill KPIs for the full maintenance period, plus extra time. Check past results and events that arrived late.
Service health Indexed service health can look current even if KV Store data behind it is old or incomplete. High Recalculate service health after KV Store writes resume, then compare the results.
Maintenance windows Window changes, including start and end times, recurring windows, and logs, might not be saved. High Restore and check current maintenance windows. Some past changes might be missing.
Adaptive thresholds Learned thresholds or custom-window settings might be old or not saved. High Rerun affected calculations and check the threshold windows.
Entities and CMDB imports Entity data, relationships, status caches, or deletion tracking might fail even if source events are indexed. High Run a full import or extend the import time range. Check that deleted entities are removed.
Dashboards and APIs Views in dashboards and APIs can show current event data alongside old or incomplete KV Store data. Medium to high Verify dashboards, APIs, episodes, and service health after ITSI restarts.
Cleanup and retention Aging or cleanup jobs might fail during a short period when KV Store is read-only. Low for a short window Rerun cleanup jobs after KV Store writes resume. Check that retention works as expected.

Use the following sequence for your deployment.

  1. Identify your deployment and KV Store type.
    Use the ITSI compatibility with related apps and add-ons to identify whether your deployment uses Splunk Enterprise cohosted KV Store, Splunk Cloud KV Store, or KV Service. The migration steps differ by deployment.
  2. Complete the ITSI 5.0.x upgrade for your deployment before you upgrade Splunk Enterprise.
    For a single-instance deployment, follow Upgrade IT Service Intelligence on a single instance. Check the ITSI compatibility with related apps and add-ons for your ITSI version and backend. For other deployment types, use the upgrade instructions for your deployment.
  3. For Splunk Enterprise, postpone the automatic cohosted KV Store migration before you upgrade to 10.6.
    In Enterprise 10.2 or 10.4, set [kvstore] postgresMigrateOnStartup=false; in a search-head cluster, push the setting to every cluster member. Follow Postpone the automatic migration, How to upgrade Splunk Enterprise, and About upgrading to 10.6: Read this first. After the platform upgrade, verify that migration has not started and KV Store data remains accessible. Keep the migration postponed until your ITSI version and KV Store backend are listed as compatible in the ITSI compatibility matrix.
  4. For Splunk Cloud, check whether your stack uses KV Store or KV Service and ask Splunk Support to confirm the upgrade sequence.
    For KV Service, a feature flag lets you postpone data migration until a maintenance window. See About upgrading to 10.6: Read this first for upgrade guidance and the Splunk Enterprise 10.6 release notes for information about the KV Service feature flag. Do not apply the Enterprise postgresMigrateOnStartup setting to Cloud KV Service.
  5. Before the Enterprise migration, confirm KV Store and search-head-cluster health, KV Store server version 7.0 or later, and more than 50% free disk space.
    Use a parallel backup to estimate migration time based on collection count and size, CPU resources, and disk I/O. Include post-migration validation in the window. See Upgrade to a cohosted KV store, Upgrade the KV Store server version, and Back up and restore the KV Store with parallelism.
  6. Stop ITSI processes that can write to KV Store before migration begins.
    On each search head that can run ITSI work, stop scheduled searches, the Rules Engine, modular and scripted inputs, work started through REST endpoints, and KV Store queue consumers. In a search-head cluster, do this on every member that can run ITSI work. Keep source data collection and indexers running if they can operate independently. Stop Splunk services only if the migration procedure for your deployment requires it. See Upgrade to a cohosted KV store for the migration procedure.
  7. Migrate the Enterprise cohosted KV Store only after confirming that your ITSI version supports KV Store v1.0.
    Follow Upgrade to a cohosted KV store. The KV Store is read-only during migration. Keep ITSI processes that write to KV Store stopped until the migration completes.
  8. Validate and restore ITSI processing after migration.
    Confirm KV Store is healthy and writable before restarting ITSI processing. Restore the searches, Rules Engine, modular and scripted inputs, REST-triggered work, and queue consumers that you stopped. Check for missing or duplicate actions before you retry them. Recalculate or backfill KPIs, service health, correlation searches, maintenance windows, adaptive thresholds, and entity data. Check episodes, dashboards, and API results. Failed operations might not run again automatically. For recovery guidance, see Back up and restore the KV Store with parallelism and Upgrade to a cohosted KV store.