Build a chart of multiple data series

Splunk transforming commands do not support a direct way to define multiple data series in your charts (or timecharts). However, you CAN achieve this using a combination of the stats and xyseries commands.

The chart and timechart commands both return tabulated data for graphing, where the x-axis is either some arbitrary field or _time, respectively. When these commands are used with a split-by field, the output is a table where each column represents a distinct value of the split-by field.

In contrast, the stats command produces a table where each row represents a single unique combination of the values of the group-by fields. You can then use the xyseries command to redefine your data series for graphing.

For most cases, you can simulate the results of "... | chart n by x,y" with "... | stats n by x,y | xyseries x y n". (For the timechart equivalent of results, x = _time.)

Scenario

Let's say you want to report on data from a cluster of application servers. The events gathered from each server contain information such as counts of active sessions, requests handled since last update, etc. and are placed in the applications_servers index. You want to display each server instance and the number of sessions per instance on the same timechart so that you can compare the distributions of sessions and load.

Ideally, you want to be able to run a timechart report, such as: 

index=application_servers 
| timechart sum(handledRequests) avg(sessions) by source

However, timechart does not support multiple data series; so instead, you need run a search similar to the following: 

index=application_servers 
| bin _time 
| stats sum(handledRequests) as hRs, avg(sessions) as ssns by _time, source 
| eval s1="handledReqs sessions" 
| makemv s1 
| mvexpand s1 
| eval yval=case(s1=="handledReqs",hRs,s1=="sessions",ssns) 
| eval series=source+":"+s1 
| xyseries _time,series,yval

Walkthrough

Let's break this search down so it's easier to understand what each part of the search is doing:

The first thing that you need to do, before the stats command, is to separate the events by time. 

... | bin _time

The stats command is used to calculate statistics for each source value: The sum of handledRequests values are renamed as hRs, and the average number of sessions are renamed as ssns. 

... | stats sum(handledRequests) as hRs, avg(sessions) as ssns by _time, source

The following portion of the search uses the eval command to add a single-valued field called "s1" to each result from the stats command. Then, the makemv command converts the values in the s1 field into a multivalued field, where the first value is "handledRequests" and the second value is "sessions". The mvexpand command then creates separate series for each value of s1. 

... | eval s1="handledRequests sessions" | makemv s1 | mvexpand s1

Then the search uses the eval command to define a new field called "yval", and assign values to the field based on the case that it matches. So, if the value of s1 is "handledRequests", the yval field is assigned the "hRs" value. And, if the value of the s1 field is "sessions", the yval field is assigned the "ssns" value. 

... | eval yval=case(s1=="handledRequests",hRs,s1=="sessions",ssns)

Then the search uses the eval command to define a new field called "series", which concatenates the value of the source and s1 fields. 

... | eval series=source+":"+s1

Finally, the xyseries command is used to define a chart with _time on the x-axis, yval on the y-axis, and data defined by the series field. 

... | xyseries _time, yval, series