SignalFx exporter
Use the SignalFx exporter to send logs and metrics from the OpenTelemetry Collector to SignalFx endpoints.
The SignalFx exporter is a native OpenTelemetry component that allows the OpenTelemetry Collector to send metrics and events to SignalFx endpoints. The supported pipeline types are metrics and logs. See Process your data with pipelines for more information.
While the SignalFx Smart Agent has reached End of Support, native OpenTelemetry components such as the Smart Agent receiver, the SignalFx receiver, and the SignalFx exporter are available and supported. For information on the receivers, see Smart Agent receiver and SignalFx receiver.
Get started
By default, the Splunk Distribution of OpenTelemetry Collector includes the SignalFx exporter in the metrics and logs/signalfx pipelines.
Sample configurations
The following example shows the default configuration of the SignalFx exporter for metrics and events ingest:
# Metrics + Events
signalfx:
access_token: "${SPLUNK_ACCESS_TOKEN}"
api_url: "${SPLUNK_API_URL}"
ingest_url: "${SPLUNK_INGEST_URL}"
# Use instead when sending to gateway (http forwarder extension ingress endpoint)
#api_url: http://${SPLUNK_GATEWAY_URL}:6060
#ingest_url: http://${SPLUNK_GATEWAY_URL}:9943
sync_host_metadata: trueWhen adding the SignalFx exporter, configure both the metrics and logs pipelines. Make sure to also add the SignalFx receiver as in the following example:
service:
pipelines:
metrics:
receivers: [signalfx]
processors: [memory_limiter, batch, resourcedetection]
exporters: [signalfx]
logs:
receivers: [signalfx]
processors: [memory_limiter, batch, resourcedetection]
exporters: [signalfx]Send histogram metrics in OTLP format
The Splunk Distribution of OpenTelemetry Collector supports OTLP histogram metrics starting from version 0.98 and higher. See Explicit bucket histograms for more information.
To send histogram data to Splunk Observability Cloud with the SignalFx exporter, set send_otlp_histograms: true in your Collector values.yaml file. For example:
exporters:
signalfx:
access_token: "${SPLUNK_ACCESS_TOKEN}"
api_url: "${SPLUNK_API_URL}"
ingest_url: "${SPLUNK_INGEST_URL}"
sync_host_metadata: true
correlation:
send_otlp_histograms: trueDefault metric filters
To prevent unwanted custom metrics, the SignalFx exporter excludes a number of metrics by default. See List of metrics excluded by default for more information.
To override default exclusions and include metrics manually, use the include_metrics option. For example:
exporters:
signalfx:
include_metrics:
- metric_names: [cpu.interrupt, cpu.user, cpu.system]
- metric_name: system.cpu.time
dimensions:
state: [interrupt, user, system]The following example instructs the exporter to send only the cpu.interrupt metric with a cpu dimension value and both per core and aggregate cpu.idle metrics:
exporters:
signalfx:
include_metrics:
- metric_name: "cpu.idle"
- metric_name: "cpu.interrupt"
dimensions:
cpu: ["*"]List of metrics excluded by default
Metrics excluded by default by the SignalFx exporter are listed in the default_metrics.go file. The following snippet shows the latest version of the list:
# DefaultExcludeMetricsYaml holds a list of hard coded metrics that's added to the
# exclude list from the config. It includes non-default metrics collected by
# receivers. This list is determined by categorization of metrics in the SignalFx
# Agent. Metrics in the OpenTelemetry convention that have equivalents in the
# SignalFx Agent that are categorized as non-default are also included in this list.
exclude_metrics:
# Metrics in SignalFx Agent Format.
- metric_names:
# CPU metrics.
- cpu.interrupt
- cpu.nice
- cpu.softirq
- cpu.steal
- cpu.system
- cpu.user
- cpu.utilization_per_core
- cpu.wait
# Disk-IO metrics.
- disk_ops.pending
# Virtual memory metrics
- vmpage_io.memory.in
- vmpage_io.memory.out
# Metrics in OpenTelemetry Convention.
# CPU Metrics.
- metric_name: system.cpu.time
dimensions:
state: [idle, interrupt, nice, softirq, steal, system, user, wait]
- metric_name: cpu.idle
dimensions:
cpu: ["*"]
# Memory metrics.
- metric_name: system.memory.usage
dimensions:
state: [inactive]
# Filesystem metrics.
- metric_name: system.filesystem.usage
dimensions:
state: [reserved]
- metric_name: system.filesystem.inodes.usage
# Disk-IO metrics.
- metric_names:
- system.disk.merged
- system.disk.io
- system.disk.time
- system.disk.io_time
- system.disk.operation_time
- system.disk.pending_operations
- system.disk.weighted_io_time
# Network-IO metrics.
- metric_names:
- system.network.packets
- system.network.dropped
- system.network.tcp_connections
- system.network.connections
# Processes metrics
- metric_names:
- system.processes.count
- system.processes.created
# Virtual memory metrics.
- metric_names:
- system.paging.faults
- system.paging.usage
- metric_name: system.paging.operations
dimensions:
type: [minor]
# k8s metrics.
- metric_names:
# matches all container limit metrics but k8s.container.cpu_limit and k8s.container.memory_limit
- /^k8s\.container\..+_limit$/
- '!k8s.container.memory_limit'
- '!k8s.container.cpu_limit'
# matches all container request metrics but k8s.container.cpu_request and k8s.container.memory_request
- /^k8s\.container\..+_request$/
- '!k8s.container.memory_request'
- '!k8s.container.cpu_request'
# matches any node condition but k8s.node.condition_ready
- /^k8s\.node\.condition_.+$/
- '!k8s.node.condition_ready'
# kubelet metrics
# matches (container|k8s.node|k8s.pod).memory...
- /^(?i:(container)|(k8s\.node)|(k8s\.pod))\.memory\.available$/
- /^(?i:(container)|(k8s\.node)|(k8s\.pod))\.memory\.major_page_faults$/
- /^(?i:(container)|(k8s\.node)|(k8s\.pod))\.memory\.page_faults$/
- /^(?i:(container)|(k8s\.node)|(k8s\.pod))\.memory\.rss$/
- /^(?i:(k8s\.node)|(k8s\.pod))\.memory\.usage$/
- /^(?i:(k8s\.node)|(k8s\.pod))\.memory\.working_set$/
# matches (k8s.node|k8s.pod).filesystem...
- /^k8s\.(?i:(node)|(pod))\.filesystem\.available$/
- /^k8s\.(?i:(node)|(pod))\.filesystem\.capacity$/
- /^k8s\.(?i:(node)|(pod))\.filesystem\.usage$/
# matches (container|k8s.node|k8s.pod).cpu...
- /^(?i:(container)|(k8s\.node)|(k8s\.pod))\.cpu\.time$/
- /^(?i:(container)|(k8s\.node)|(k8s\.pod))\.cpu\.utilization$/
# matches k8s.node.network.io and k8s.node.network.errors
- /^k8s\.node\.network\.(?:(io)|(errors))$/
# matches k8s.volume.inodes, k8s.volume.inodes and k8s.volume.inodes.used
- /^k8s\.volume\.inodes(\.free|\.used)*$/Filter metrics using service or environment
To filter metrics by service or environment, send traces to Splunk Observability Cloud using an OTLP exporter, such as the OTLP/HTTP exporter. Splunk Observability Cloud uses the OTLP trace data to correlate services and environments with related metrics.
correlation setting, is deprecated and has no effect.
Drop histogram metrics
In case of high cardinality metrics, dropping histogram buckets might be useful. To drop histogram metrics, set drop_histogram_buckets to true.
When drop_histogram_buckets is activated, histogram buckets are dropped instead of being translated to datapoints with the _bucket suffix. Only datapoints with _sum, _count, _min, and _max are sent through the exporter.
Migrate from translation rules
Migration guide for translation rules
translation_rules is deprecated, use existing OpenTelemetry Collector processors instead. For more details on the replacement options refer to the migration guide in GitHub.
Deprecated rules (for reference only)
Use the translation_rules field to transform metrics or produce custom metrics by copying, calculating, or aggregating other metric values without requiring an additional processor.
Translation rules currently allow the following actions:
-
aggregate_metric: Aggregates a metric through removal of specified dimensions. -
calculate_new_metric: Creates a new metric via operating on two consistuent ones. -
convert_values: Convert float values to int or int to float for specified metric names. -
copy_metrics: Creates a new metric as a copy of another. -
delta_metric: Creates a new delta metric for a specified non-delta one. -
divide_int: Scales a metric’s integer value by a given factor. -
drop_dimensions: Drops dimensions for specified metrics, or globally. -
drop_metrics: Drops all metrics with a given name. -
multiply_float: Scales a metric’s float value by a given float factor. -
multiply_int: Scales a metric’s int value by a given int factor. -
rename_dimension_keys: Renames dimensions for specified metrics, or globally. -
rename_metrics: Replaces a given metric name with specified one. -
split_metric: Splits a given metric into multiple new ones for a specified dimension.
Default translation rules and generated metrics
The SignalFx exporter uses the translation rules defined in translation/constants.go by default.
The default rules create metrics which are reported directly to Infrastructure Monitoring. If you want to change any of their attributes or values, you need to either modify the translation rules or their constituent host metrics.
By default, the SignalFx exporter creates the following aggregated metrics from the Host metrics receiver:
-
cpu.idle -
cpu.interrupt -
cpu.nice -
cpu.num_processors -
cpu.softirq -
cpu.steal -
cpu.system -
cpu.user -
cpu.utilization -
cpu.utilization_per_core -
cpu.wait -
disk.summary_utilization -
disk.utilization -
disk_ops.pending -
disk_ops.total -
memory.total -
memory.utilization -
network.total -
process.cpu_time_seconds -
system.disk.io.total -
system.disk.operations.total -
system.network.io.total -
system.network.packets.total -
vmpage_io.memory.in -
vmpage_io.memory.out -
vmpage_io.swap.in -
vmpage_io.swap.out
In addition to the aggregated metrics, the default rules make available the following "per core" custom hostmetrics. The CPU number is assigned to the dimension cpu:
-
cpu.interrupt -
cpu.nice -
cpu.softirq -
cpu.steal -
cpu.system -
cpu.user -
cpu.wait
Settings
The following table shows the configuration options for the SignalFx exporter:
included
https://raw.githubusercontent.com/splunk/collector-config-tools/main/cfg-metadata/exporter/signalfx.yaml
access_token_passthrough setting if you’re using a SignalFx receiver with the same setting. Only use the SignalFx receiver with the SignalFx exporter when activating this setting.
Troubleshooting
If you are a Splunk Observability Cloud customer and are not able to see your data in Splunk Observability Cloud, you can get help in the following ways.
Available to Splunk Observability Cloud customers
-
Submit a case in the Splunk Support Portal.
-
Contact Splunk Support.
Available to prospective customers and free trial users
-
Ask a question and get answers through community support at Splunk Answers.
-
Join the Splunk community #observability Slack channel to communicate with customers, partners, and Splunk employees worldwide.