Known issues in Splunk IT Service Intelligence

This version of IT Service Intelligence (ITSI) has the following known issues and workarounds.

Adaptive Thresholding

Date filed Issue number Description
2024-09-04 ITSI-37270 Use Recommended Thresholding Configuration cannot use all backfilled events
Workaround:
While backfilling the KPI customer can set the fill data gaps option other than Last available value and after backfill completes successfully they can switch the option to Last available value.
2024-04-12 ITSI-35070 On few KPIs using adaptive threshold, the results from the scheduled overnight run seem very different from the preview adaptive threshold results
Workaround:
Add {{| where not isnull(alert_value)}} before Applyat command in the AT search to remove the empty Alert valueevents.

Entity Rules

Date filed Issue number Description
2024-05-06 ITSI-35571 New entities are not added to linked services even if they match the filter conditions

Notable Events

Date filed Issue number Description
2024-06-25 ITSI-36467 Investigate the ConcurrentModificationException in rules engine process
Workaround:
Update the below changes *on each SH* to disable async execution of actions in Rules Engine.
  1. Go to $SPLUNK HOME/etc/apps/SA-ITOA/local/itsi rules engine.properties
  2. There should be a parameter Rules_engine_feature_disabled_list in the file if not available then add this param with values as given in point-3.
  3. Add the value RUN ACTION SYNC to the comma separated list of values

{noformat}rules_engine_feature_disabled_list = POLICY_EXECUTOR_ASYNC_SUB_ACTORS, POLICY_EXECUTOR_STATE_RECOVERY, SORT_NOTABLE_EVENTS, RUN_ACTION_ASYNC{noformat}

  1. Restart the Itsi event grouping real time search by going to Activity → Jobs → Set filters to "All" → search Label="itsi event grouping" and then stop the job. It will restart after a couple of mins
2024-06-20 ITSI-36397 Actions are not performed if the event is breaking the episode based on the timebased criteria and grouping into a new group.
Workaround:
Reset Policy rules check frequency delay to 60000 under $SPLUNK HOME/etc/apps/SA-ITOA/local/itsi rules engine.properties

OR

Update the action rules in the new to perform the action when Number of events in this episode is less than or equals to 2 if the action is Number of events in this episode is == 1

2024-06-10 ITSI-36103, ITSI-36215 NEAP action rules triggers false alert emails even though the conditions are not satisfied
2024-02-08 ITSI-34393 BDT event should not satisfy 'if episode is broken' action rule for inactive episodes

Notable Event Aggregation Policies

Date filed Issue number Description
2024-06-25 ITSI-36467 Investigate the ConcurrentModificationException in rules engine process
Workaround:
Update the below changes *on each SH* to disable async execution of actions in Rules Engine.
  1. Go to $SPLUNK HOME/etc/apps/SA-ITOA/local/itsi rules engine.properties
  2. There should be a parameter Rules engine feature disabled list in the file if not available then add this param with values as given in point-3.
  3. Add the value RUN ACTION ASYNC to the comma separated list of values

{noformat}rules_engine_feature_disabled_list = POLICY_EXECUTOR_ASYNC_SUB_ACTORS, POLICY_EXECUTOR_STATE_RECOVERY, SORT_NOTABLE_EVENTS, RUN_ACTION_ASYNC{noformat}

  1. Restart the Itsi event grouping real time search by going to Activity → Jobs → Set filters to "All" → search Label="itsi event grouping" and then stop the job. It will restart after a couple of mins
2024-06-20 ITSI-36397 Actions are not performed if the event is breaking the episode based on the timebased criteria and grouping into a new group.
Workaround:
Reset Policy rules check frequency delay to 60000 under $SPLUNK HOME/etc/apps/SA-ITOA/local/itsi rules engine.properties

OR

Update the action rules in the new to perform the action when Number of events in this episode is less than or equals to 2 if the action is Number of events in this episode is == 1

2024-06-10 ITSI-36103, ITSI-36215 NEAP action rules triggers false alert emails even though the conditions are not satisfied
2024-02-08 ITSI-34393 BDT event should not satisfy 'if episode is broken' action rule for inactive episodes

Service Analyzer

Date filed Issue number Description
2024-05-23 ITSI-35809 Could not retrieve health scores for the service tiles.
2024-05-01 ITSI-35514 For KPI configured with fill data gaps with null values and set severities for null value to be other than unknown, alert_value is not present in metric index results into service analyzer does not display lane for those KPIs
Workaround:
For the problematic KPI change the fill data gaps with N/A to have some custom value or set the Threshold level for the null value to Unknown
2023-01-12 ITSI-28014 On Splunk Enterprise SH Cluster, Drill-down link on Service is not functional for ITSI Dev Build
Workaround:
# Workaround from the UI (easier and recommended):
    1. With an admin user, go to {{https://<URL>:8000/en-US/manager/permissions/itsi/data/ui/views/service_definition?manager_cancel_url=%2Fmanager%2Fitsi%2Fdata%2Fui%2Fviews%3Fns%3Ditsi%26app_only%3D1%26pwnr%3D-%26search%3Ddefinition%26count%3D100&uri=%2FservicesNS%2Fnobody%2Fitsi%2Fdata%2Fui%2Fviews%2Fservice_definition}}
    2. Give READ and WRITE for ITOA_ADMIN
    3. Give READ for ITOA_TEAM_ADMIN
    4. Save
    5. Refresh service analyzer page
  1. Workaround from the CLI:
  • SSH into the SH of the cluster.
  • Go to $SPLUNK HOME/etc/apps/itsi/metadata file, and remove Local.meta file by using the command Rm -rf $SPLUNK HOME/etc/apps/itsi/metadata/local.meta .
  • Restart Splunk using $SPLUNK HOME/bin/splunk restart and open ITSI.
  • Go to Service Analyzer and open any service and you'll be able to see the drilldown.

Service Definition

Date filed Issue number Description
2024-09-10 ITSI-37299 Discrepancy in the "Per-Entity Threshold Value" graph
2024-04-22 ITSI-35260 Entities not displaying entity_types correctly in Service Definition page

Uncategorized issues

Date filed Issue number Description
2024-06-06 ITSI-36019 Discrepancy in time in the user_access_interface.log file
2024-05-31 ITSI-35922 events from source itsi_appserver.log are way longer than new increased TRUNCATE limit
Workaround:
Copy the below stanza in Apps/SA-ITOA/package/local/props.conf from Apps/SA-ITOA/package/default/props.conf and increased TRUNCATE value with 200000 .

{noformat}[source::...(/|\\)var(/|\\)log(/|\\)splunk(/|\\)itsi*] TIME_PREFIX=^ TIME_FORMAT=%Y-%m-%d %H:%M:%S,%3N%z LINE_BREAKER =([\r\n]+)\d{4}-\d{2}-\d{2}\s SHOULD_LINEMERGE = false TRUNCATE = 100000 MAX_TIMESTAMP_LOOKAHEAD = 29 sourcetype = itsi_internal_log EXTRACT-component = ^[^\[\n]*\[(?P<component>[^\]]+) EXTRACT-sub_component = ^[^\]\n]*\]\s+\[(?P<sub_component>[^:\]]+) EXTRACT-log_level = ^[^\[\n]*\s+(?P<log_level>(?:\w+))\s+\[{noformat}

2024-04-26 ITSI-35428 Customer wants to know whether Index references needs to be changed in scripts for default ITSI indexes.
2024-02-29 ITSI-34551 Breaking event does not trigger breaking action rules
2022-03-24 ITSI-22641 Premium features disabled because the ITSI license checker is not finding all the valid licenses, when they are more than 30 licenses installed
Workaround:
If the license-master has more than 30 licenses, remove the expired ones to keep the list short.