Get histogram data into Splunk Observability Cloud

You can collect histogram data using a variety of receivers, including the Prometheus receiver, and send them to Splunk Observability Cloud using the OpenTelemetry Collector.

You can collect histogram data using a variety of receivers, including the Prometheus receiver, and send them to Splunk Observability Cloud using the OpenTelemetry Collector. See Prometheus receiver.

The Splunk Distribution of OpenTelemetry Collector supports explicit-bucket histogram metrics. This allows you to send histogram metric using the OTLP/HTTP exporter or, starting from version 0.98, through the SignalFx exporter.

CAUTION: Only the SignalFx exporter included in the Splunk Distribution of the OpenTelemetry Collector has been confirmed to successfully send histogram data. The version of the SignalFx exporter in the Amazon Distribution of OpenTelemetry (ADOT), however, does not support the parameter send_otlp_histogram and, therefore, cannot be used to send histogram data.

Export histogram data with the SignalFx exporter

The version of the SignalFx exporter in the Splunk Distribution of the OpenTelemetry Collector supports the parameter send_otlp_histograms and is the recommended method to send histogram data.

The SignalFx exporter can preserve histogram bucket data. This can be used to extract various statistics from the metric at charting time, e.g., 90th percentile or mean.

To send histogram data to Splunk Observability Cloud with the SignalFx Exporter, set the send_otlp_histograms option to true. For example:

exporters:
  signalfx:
    access_token: "${SPLUNK_ACCESS_TOKEN}"
    api_url: "${SPLUNK_API_URL}"
    ingest_url: "${SPLUNK_INGEST_URL}"
    sync_host_metadata: true
    correlation:
    send_otlp_histograms: true

Export histogram data with the OTLP/HTTP exporter

Because of its broader compatibility, the OTLP/HTTP exporter is able to send histogram data to any OTLP-compliant backend. Splunk Observability Cloud does not support sending metrics via the gRPC-based OTLP exporter, so you must use OTLP/HTTP exporter.

Note: The OTLP/HTTP exporter can send logs but not to Splunk Observability Cloud. You should use the the Splunk HTTP Event Collector (HEC) if you need to export logs to Splunk Observability Cloud.

To send histogram data to Splunk Observability Cloud with the OLTP/HTTP exporter, you configure the endpoints for the metrics_endpoint and the traces_endpoint fields:

exporters:
  otlphttp:
    metrics_endpoint: https://ingest.<realm>.signalfx.com/v2/datapoint/otlp
    traces_endpoint: https://ingest.<realm>.signalfx.com/v2/trace/otlp
    headers:
      "X-SF-Token": "mytoken"
    tls:
      insecure: true
    timeout: 10s

Best practices when sending bucket histogram data

When sending bucket histogram data to Splunk Observability Cloud, follow these best practices:

Send minimum and maximum values, unless you’re sending cumulative data. The minimum value must be lower than the maximum value, otherwise the datapoint is dropped.
Use less than 31 buckets when sending custom histograms. Histograms with more than 31 buckets are dropped.
Make sure that bucket boundaries don’t overlap or repeat. Order the bucket boundaries when sending them.
Send values as signed integer, float, or numeric string in decimal or fixed-point notation. Splunk Observability Cloud stores them as 64-bit integers.
Check that the sum of all histogram buckets is equal to the count field, and that the size of bucket boundaries is equal to the bucket count minus 1. Histograms that don’t comply with these criteria are dropped.
When sending cumulative data, for example from Prometheus, use delta aggregation temporality. See Considerations on delta aggregation temporality for instructions on how to configure delta temporality in your system.

Considerations on delta aggregation temporality

When handling cumulative histograms, you must set the delta aggregation temporality flag. If you do not, the cumulative histograms will lack minimum and maximum values. This might cause a percentile calculation to give an incorrect value.

To activate delta aggregation temporality in your instrumentation, set the OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE environment variable to delta. See the compliance matrix in the OpenTelemetry Specification repository to check SDK support for your language.

Send histogram data using the API

If you need to bypass the OpenTelemetry Collector, send histogram data directly to Splunk Observability Cloud using the /v2/datapoint/otlp endpoint of the ingest API. The endpoint accepts data in OTLP, serialized as Protobuf, over HTTP. The gRPC scheme is not supported.

To learn how to send histogram metric data using the API, see /datapoint/otlp in the Splunk Developer Portal.

Migrate your dashboards, functions, charts, and detectors

To migrate your existing dashboards, functions, charts, and detectors to histograms, follow these steps:

Make sure that you’re sending histogram data using the Splunk Distribution of OpenTelemetry Collector version 0.98 or higher. Lower versions can’t send histogram data in OTLP format using the SignalFx exporter.
Edit your charts to use the new histogram() function. See histogram() in the SignalFlow reference documentation.

Troubleshooting

If you are a Splunk Observability Cloud customer and are not able to see your data in Splunk Observability Cloud, you can get help in the following ways.

Available to Splunk Observability Cloud customers

Submit a case in the Splunk Support Portal.
Contact Splunk Support.

Available to prospective customers and free trial users

Ask a question and get answers through community support at Splunk Answers.
Join the Splunk #observability user group Slack channel to communicate with customers, partners, and Splunk employees worldwide. To join, see Chat groups.

Documentation