Ingest and normalize third-party alerts into ITSI

Onboard alerts from Splunk Cloud Platform or other third-party sources by creating a data integration connection in ITSI. A data integration connection normalizes raw event data in order to bring that data into ITSI Event Analytics. Follow the assisted workflow to configure and create a data integration connection.

Prerequisites

You must have the correct permissions to create a data integration connection:

Role Permissions
itoa_admin, itoa_team_admin
  • Create, update, and delete a connection
  • Activate and deactivate a connection
  • Preview results for the field mappings defined for a connection
  • View all the connections available for a data integration source (i.e. Nagios)
  • View list of data integration connections on the Deployed Integrations section
itoa_analyst
  • View all the connections available for a data integration source (i.e. Nagios)
  • View list of data integration connections on the Deployed Integrations section

Create a data integration connection

  1. From the ITSI navigation menu, select Configuration then Data Integrations.
  2. Select one of the data integration connections from the Integrations library, or select Add connection for deployed connections to add a new data connection for that specific data source.
  3. Follow the guided steps in the workflow to map your field data to the correct values and generate a data integration connection.

Upgrade to new ITSI data model

ITSI includes a standardized event data model that provides a consistent structure for all events ingested through the product with a formal schema built as an extension of the Splunk Common Information Model (CIM). With the new data model, events share a common set of core fields (event_type, signature, src, title, status, vendor_severity, producer_app, and other fields). This creates alignment across correlation rules, notable event aggregation policies, and searches that work across multiple data sources.

Events can be enriched with CMDB configuration item data using a standardized ci_* namespace — including CI classification, location, environment, ownership, operational status, and relationship mappings.

New event_fingerprint, dedup_key, and correlation_key fields replace the previous alert_identifier_fields, giving you more control over how events are categorized, de-duplicated, and grouped into episodes.

Existing data integration templates that are not currently using the updated data model can be upgraded by selecting the Upgrade button. Upgrading the template automatically updates existing field names to align with the new data model. View your updated templates by selecting the data source from the Deployed integrations tab. Existing templates will also continue to function until you choose to migrate.

The following is an example of Dynatrace events mapped to the new data model:
JSON 
{
   affectedEntities{}.entityId.id: SERVICE-737140
   affectedEntities{}.entityId.type: SERVICE
   affectedEntities{}.name: webserver01.example.com
   app: Dynatrace
   ci_id: 9f8cf8b887510fd8abec1f0d3fbb3555
   description: Response time increase
   drilldown_search_earliest_offset: -900
   drilldown_search_latest_offset: 900
   drilldown_search_search: %itsi_drilldown_search_spl%
   drilldown_search_title: %itsi_drilldown_search_label%
   drilldown_title: %itsi_drilldown_link_label%
   drilldown_uri: https://splunk.com
   dtp_alert_status: CLOSED
   dtp_category: INFO
   dtp_end_time: 1782333603
   dtp_entity_details_compressed: webserver01.example.com+SERVICE-635354+SERVICE+Response time increase+1782333535711
   dtp_entity_id: SERVICE-635354
   dtp_entity_name: webserver01.example.com
   dtp_entity_type: SERVICE
   dtp_evidence_details_info: webserver01.example.com+SERVICE-635354+SERVICE+Response time increase+1782333535711
   dtp_evidence_start_times: 1782333535711
   dtp_impacted_entity_info: Production App+APPLICATION-461256+APPLICATION
   dtp_problem_id: P-01234
   dtp_signature: Response time increase
   dtp_start_time: 1782333536
   dtp_status: CLOSED
   dtp_title: Response time degradation on webserver01.example.com
   endTime: 1782333603371
   entityTags{}.context: CONTEXTLESS
   entityTags{}.key: env
   entityTags{}.value: production
   event_field_max_length: 10000
   event_fingerprint: Response time increase-webserver01.example.com-SERVICE
   event_id: 96032634-700d-11f1-b7fe-0e96002f3029
   event_identifier_fields: event_fingerprint
   event_identifier_hash: 7878482c9d2a296750ffabea6cdc821e7bf3ea8471213d90eb02d02a5d259695
   event_identifier_string: Response time increase-webserver01.example.com-SERVICE
   event_type: alert
   evidenceDetails.details{}.displayName: Response time increase
   evidenceDetails.details{}.entity.entityId.id: SERVICE-635354
   evidenceDetails.details{}.entity.entityId.type: SERVICE
   evidenceDetails.details{}.entity.name: webserver01.example.com
   evidenceDetails.details{}.evidenceType: SLOWDOWN
   evidenceDetails.details{}.startTime: 1782333535711
   evidenceDetails.totalCount: 1
   impactAnalysis.impacts{}.estimatedAffectedUsers: 205
   impactAnalysis.impacts{}.impactType: PERFORMANCE
   impactedEntities{}.entityId.id: APPLICATION-461256
   impactedEntities{}.entityId.type: APPLICATION
   impactedEntities{}.name: Production App
   is_impacted_entity: 0
   is_use_event_time: 0
   itsi_drilldown_link_url: https://splunk.com
   itsi_drilldown_search_earliest: -900
   itsi_drilldown_search_latest: 900
   itsi_instruction: %itsi_instruction%
   itsi_schema_version: v2
   mod_time: 1782333907.887072
   orig_description: Response time increase
   orig_index: main
   orig_owner: unassigned
   orig_raw: {"displayId": "P-01234", "severityLevel": "INFO", "status": "CLOSED", "startTime": 1782333543371, "endTime": 1782333603371, "rootCauseEntity": {"entityId": {"id": "SERVICE-651755", "type": "SERVICE"}, "name": "webserver01.example.com"}, "managementZones": [], "entityTags": [{"context": "CONTEXTLESS", "key": "env", "value": "production"}], "title": "Response time degradation on webserver01.example.com", "affectedEntities": [{"entityId": {"id": "SERVICE-737140", "type": "SERVICE"}, "name": "webserver01.example.com"}], "impactedEntities": [{"entityId": {"id": "APPLICATION-461256", "type": "APPLICATION"}, "name": "Production App"}], "evidenceDetails": {"totalCount": 1, "details": [{"displayName": "Response time increase", "entity": {"entityId": {"id": "SERVICE-635354", "type": "SERVICE"}, "name": "webserver01.example.com"}, "evidenceType": "SLOWDOWN", "startTime": 1782333535711}]}, "impactAnalysis": {"impacts": [{"impactType": "PERFORMANCE", "estimatedAffectedUsers": 205}]}, "ci_id": "9f8cf8b887510fd8abec1f0d3fbb3555"}
   orig_rid: 20
   orig_sid: scheduler_c3BsdW5rLXN5c3RlbS11c2Vy__itsi__RMD555ee4c43f1bdcf8e_at_1782333900_27144
   orig_sourcetype: dynatrace:problemV2
   orig_status: 5
   orig_time: 1782333907.887072
   orig_title: Response time degradation on webserver01.example.com
   owner: unassigned
   producer_app: Dynatrace
   producer_event_id: P-01234
   rid: 20
   rootCauseEntity.entityId.id: SERVICE-651755
   rootCauseEntity.entityId.type: SERVICE
   rootCauseEntity.name: webserver01.example.com
   search_name: DATA_INTEGRATION_CS-dynatrace_default
   search_type: basic
   severity: 1
   severity_id: 1
   signature: Response time increase
   src: webserver01.example.com
   startTime: 1782333543371
   status: 5
   subcomponent: SERVICE
   title: Response time degradation on webserver01.example.com
   vendor_severity: INFO
}

Select data ingest method

  1. Set a title for your data integration connection. This title can't be changed after you create the connection.
  2. Specify where your data is stored.
    • Splunk add-on (indexed data): raw alerts exist in a Splunk index.
    • Webhook (HTTP request): alerts exist in a third-party source and will be ingested using the relevant Splunk add-on, or by using a search.
  3. If you select indexed data, provide a time range for the search to find data. The search must return at least one result.

Map data fields for ingest and configuration

  1. Input the fields from your data integration source that will map to the Splunk Common Information Model (CIM). CIM is a shared semantic model focused on extracting value from data, and ensures that your data is normalized and can integrate smoothly with ITSI Event Analytics.
    Select one of the following transformation options for each required field:
    • Composition: input multiple fields to map to the property. Select one or more fields and/or one or more text strings.
    • Mapping rule: select either the value case mapping, or the coalesce option. Value case mapping sets conditional rules for field mapping. Coalesce normalizes field names with the same value and takes the first non-null value to combine. You must set a default value to apply to the field in case these mapping rules don't return any values.
    • Regex: apply a regex expression to extract data from the field.

    Note: Note: The default field map configuration is populated from the itsi_data_integration_template.conf file.
  2. Set a transformation option and value for each of the following fields:
    Field Description
    Src The host or source of your alert data.
    Vendor severity The original vendor-specific severity or health status string for this alert. For example, critical or warning.
    Severity ID The numeric or vendor-specific severity indicator corresponding to the event severity.

    For ITSI, severity_id is one of the following values:

    1 = Info or Unknown
    2 = Normal or Cleared
    3 = Low
    4 = Medium
    5 = High
    6 = Critical

    Title The title of the notable event in Episode Review. For example, mysql-01 server cpu Load %.
    Owner The ITSI role to which the notable event is assigned in Episode Review. If using advanced mode, the value must resolve to a username in the system.
    Status The triage status to display in Episode Review. For example, New.
  3. (Optional) Configure the additional field values. For descriptions for each field value, refer to the ITSI Normalization documentation.

Schedule

  1. Configure the schedule for the data integration connection:
    • Basic: Schedule searches to run at regular intervals. Configure the search interval in the Run Every menu.
    • Cron: Schedule searches to run periodically at fixed times, dates, or intervals. Enter a schedule in Cron Schedule. For more information, see Use cron expressions for scheduling in the Alerting Manual.
  2. Identify the service(s) impacted by the alerts generated from this data integration, and set the entity lookup field used to look up corresponding entities. For example, host.
    Field Description Defaults
    Service Select one or more ITSI services to which this correlation search applies. You can only select services that belong to teams for which you have read access. None
    Entity Lookup Field The field in the data retrieved by the correlation search that is used to look up corresponding entities. For example, host. None

Throttling

A data integration connection search can create multiple alerts for the same condition. In most cases, it is best to have a single alert for the same condition for a specific duration. Throttling blocks the search from creating duplicate alerts for the same issue each time the search runs.

Configure the following fields to suppress alerts:

Field Description
Fields to group by Fields to compare to identify similar events. For example, cpu_load_percent.

If a field matches this field during the suppress period, a new alert won't be created. You can define multiple fields. The fields available depend on the search fields that the correlation search returns.

Suppress period During the suppress period, any additional event that matches any of the Fields to group by does not create a new alert. After the suppress period passes, the next matching event creates a new alert, and throttling conditions resume. Enter a relative time range in seconds. For example, 60s (60 seconds).

Preview and save data integration connection

  1. Select Preview results in the Transformed field section to preview the results of your field mappings, and view the notable events created based on your mapping rules. Additionally, you can search for fields in the Fields section.
  2. When you finish configuring your data integration connection, select Save.

View data integration connections

To see a list of all configured data integration connections in ITSI, select the Deployed integrations tab from the Data Integrations page. Alternatively, select the specific data source from the page to view the connections specific to that data source. You can deactivate, clone, or delete your connection after you create it.

You can also view your new data integration connection listed on the Searches, Reports, and Alerts page. These saved searches are prefixed with DATA_INTEGRATION_CS-.

Note: Note: When you update a data connection from ITSI, the corresponding saved search is automatically updated. However, changes made to the saved search from the Searches, Reports, and Alerts page won't be reflected.

Next steps

You can filter by events on the ITSI Episode Review page based on your data connection settings. Additionally, when you select an episode you will see the impacted services and KPIs based on your configuration settings. View the other common fields configured for each episode on the Common Fields tab on the Episode Review page.