Application Server Module troubleshooting

Verify Data Collection

Verify that the add-ons in your deployment are installed and configured correctly by checking the add-on's indices, sources or source types.

Add-on Data verification search
Tomcat index = * sourcetype = tomcat:*
Websphere index = * sourcetype = index=* sourcetype = ibm:was*

Data population can also be verified by checking to see if your tags have data.

Tag Search
appserver tag=appserver
inventory tag=appserver tag=inventory
inventory-->application tag=appserver tag=inventory tag=application
performance tag=appserver tag=performance
performance-->transactions tag=appserver tag=performance tag=transactions
performance-->runtime tag=appserver tag=performance tag=runtime
performance-->throughput tag=appserver tag=performance tag=throughput
performance-->session tag=appserver tag=performance tag=session

Verify inventory data

Index Search Data Model Search 
tag=appserver
tag=inventory
|search application_server=*
|table application_server,role,vendor_product,version_number,vendor_product_runtime,vendor_product_runtime_version,full_host_name
|dedup application_server
 
|tstats latest(Inventory.ip_address) as ip_address,
latest(Inventory.product_family) as product_family,
latest(Inventory.vendor_product) as vendor_product,
latest(Inventory.version_number) as version_number,
latest(Inventory.vendor_product_runtime) as vendor_product_runtime,
latest(Inventory.runtime_vendor) as runtime_vendor,
latest(Inventory.vendor_product_runtime_version) as vendor_product_runtime_version,
latest(Inventory.appserver_port_number) as appserver_port_number,
latest(Inventory.root_role) as role,
latest(Inventory.full_host_name) as full_host_name,
latest(Inventory.application_server) as application_server
from datamodel=AppServer.Inventory
where Inventory.ip_address=* 
by host
| fields + application_server,role,vendor_product,version_number,vendor_product_runtime,vendor_product_runtime_version,full_host_name

Verify KPI data

Search Expected Results 
tag=appserver
tag=performance
tag=session
| timechart avg(active_sessions_count) as "Active Sessions (count)"
 Numbers populate on the Statistics tab with the field Active Sessions(count) 
tag=appserver
tag=performance
tag=runtime
| timechart avg(cpu_utilization_percent) as cpu_util_pct
| eventstats avg(cpu_util_pct) as average
| eventstats min(cpu_util_pct) as minimum
| eventstats max(cpu_util_pct) as maximum
| eval _stats = "Range: ".round(minimum, 1)." - ".round(maximum, 1)."%, "."Average: ".round(average, 1)."%"
| table cpu_util_pct,average, minimum, maximum
 Numbers populate on the Statistics tab in a table with columns cpu_util_pct, average, minimum, and maximum

Entities not found

When you are unable to locate operating system entities, see the ITSI module troubleshooting section on entities.

Troubleshooting entity appearance speed

It will take about an hour to start seeing a full sparkline chart and for KPIs to become active. Learn about entity appearance speed.

Metrics not collected

If you are unable to view your data, run the data model audit to make sure your data models are processing your data.

If you are running into issues seeing your KPI data, see ITSI module troubleshooting section on KPIs.

User Permissions

If a user encounters a permission-related obstacle, the issue could be related to their assigned role.

ITSI permissions are determined by the role that each user has. Each role offers a different set of user permissions.

See ITSI module troubleshooting to learn about permissions within ITSI.

Learn about adding navigation to a Splunk app.

The Drilldown page stops functioning

If the drilldown page stops working while using ITSI:

  1. Launch a web browser and
  2. Use your console to debug the JavaScripts,
  3. Examine the results to see if any KPI templates are corrupted.

Tomcat lookup field error

If you have both the Tomcat and WebSphere add-ons installed on the same server, and encounter the following error about Tomcat. 

    
Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'tomcat:access:log' and lookup table 'tomcat_rootdir_jmxhost_lookup'.
Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'tomcat:runtime:log' and lookup table 'tomcat_rootdir_jmxhost_lookup'. [tag-asnixdev3.sv.splunk.com]
Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'tomcat:access:log' and lookup table 'tomcat_rootdir_jmxhost_lookup'. [tag-asnixdev3.sv.splunk.com]
Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'tomcat:runtime:log' and lookup table 'tomcat_rootdir_jmxhost_lookup'.

Check the $splunk/etc/apps/Splunk_TA_ibm-was/lookups/ folder to see if the file ibm_was_inventory.csv is present. If it is not there, open the WebSphere add-on installation package, and copy the ibm_was_inventory.csv file into the $splunk/etc/apps/Splunk_TA_ibm-was/lookups/ folder.