Kafka metrics receiver

Collects Kafka metrics such as brokers, topics, partitions, and consumer groups from Kafka server, and converts them to OTLP format.

The Kafka metrics receiver collects Kafka metrics (such as brokers, topics, partitions, and consumer groups) from a Kafka server and converts them to OTLP format. The supported pipeline type is metrics. See Process your data with pipelines for more information.

Note: Out-of-the-box dashboards and navigators are not supported for the Kafka metrics receiver.

Get started

Follow these steps to configure and activate the component:

Deploy the Splunk Distribution of the OpenTelemetry Collector to your host or container platform:
Configure the receiver as described in the next section.
Restart the Collector.

Sample configuration

To activate the Kafka metrics receiver, add kafkametrics to the receivers section of your configuration file, as shown in the following example:

receivers:
  kafkametrics:
    protocol_version: 2.0.0
    scrapers:
      - brokers
      - topics
      - consumers

To complete the configuration, include the receiver in the metrics pipeline of the service section of your configuration file. For example:

service:
  pipelines:
    metrics:
      receivers: [kafkametrics]

Configuration options

These settings are required:

protocol_version. No default. The Kafka protocol version, for example 2.0.0.
scrapers. No default. Any combination of the following scrapers can be enabled:
- topics
- consumers
- brokers

The following settings are optional:

brokers. localhost:9092 by default. The list of brokers to read from.
resolve_canonical_bootstrap_servers_only. false by default. Determines whether to resolve then reverse-lookup broker IPs during startup.
topic_match. ^[^_].*$) by default. Regex pattern of topics to filter on metrics collection. The default filter excludes internal topics, which start with _.
group_match. .*) by default. Regex pattern of consumer groups to filter on for metrics.
client_id. otel-metrics-receiver by default. Consumer client Id.
collection_interval. 1m by default. Frequency of metric collection/scraping.
initial_delay. 1s by default. Determines how long this receiver waits before starting.
auth. No default. Use one of the following:
- plain_text. It has the following fields:
  - username. The username to use.
  - password. The password to use.
- tls. It has the following fields:
  - ca_file. Use only if insecure is set to false. Path to the CA cert. For a client it verifies the server certificate.
  - cert_file. Use only if insecure is set to false. Path to the TLS cert to use for TLS required connections.
  - key_file. Use only if insecure is set to false. Path to the TLS key to use for TLS required connections.
  - insecure. false by default. Turns off the verification of the server’s certificate chain and host name, InsecureSkipVerify in the tls configuration.
  - server_name_override. Indicates the name of the server requested by the client in order to support virtual hosting.
- kerberos. It has the following fields:
  - service_name. Kerberos service name.
  - realm. Kerberos realm.
  - use_keytab. If true, the keytab is used instead of the password.
  - username. The Kerberos username used to authenticate with KDC.
  - password. The Kerberos password used to authenticate with KDC.
  - config_file. Path to Kerberos configuration, for example /etc/krb5.conf.
  - keytab_file. Path to the keytab file, for example /etc/security/kafka.keytab.
  - disable_fast_negotiation. false by default. Turns off the PA-FX-FAST negotiation (Pre-Authentication Framework - Fast). Some common Kerberos implementations do not support PA-FX-FAST negotiation.

Configuration example: Set authentication and the collection interval to 5 seconds

This example sets the collection interval for all scrapers to 5 seconds and configures TLS authentication:

receivers:
  kafkametrics:
    brokers: 10.10.10.10:9092
    protocol_version: 2.0.0
    scrapers:
      - brokers
      - topics
      - consumers
    auth:
      tls:
        ca_file: ca.pem
        cert_file: cert.pem
        key_file: key.pem
    collection_interval: 5s

Settings

The following table shows the configuration options for the Kafka metrics receiver:

included

https://raw.githubusercontent.com/splunk/collector-config-tools/main/cfg-metadata/receiver/kafkametrics.yaml

Metrics

The following metrics, resource attributes, and attributes are available.

included

https://raw.githubusercontent.com/splunk/collector-config-tools/main/metric-metadata/kafkametricsreceiver.yaml

Activate or deactivate specific metrics

You can activate or deactivate specific metrics by setting the enabled field in the metrics section for each metric. For example:

receivers:
  samplereceiver:
    metrics:
      metric-one:
        enabled: true
      metric-two:
        enabled: false

The following is an example of host metrics receiver configuration with activated metrics:

receivers:
  hostmetrics:
    scrapers:
      process:
        metrics:
          process.cpu.utilization:
            enabled: true

Note: Deactivated metrics aren’t sent to Splunk Observability Cloud.

Billing

If you’re in a MTS-based subscription, all metrics count towards metrics usage.
If you’re in a host-based plan, metrics listed as active (Active: Yes) on this document are considered default and are included free of charge.

Learn more at Infrastructure Monitoring subscription usage (Host and metric plans).

Troubleshooting

If you are a Splunk Observability Cloud customer and are not able to see your data in Splunk Observability Cloud, you can get help in the following ways.

Available to Splunk Observability Cloud customers

Submit a case in the Splunk Support Portal.
Contact Splunk Support.

Available to prospective customers and free trial users

Ask a question and get answers through community support at Splunk Answers.
Join the Splunk community #observability Slack channel to communicate with customers, partners, and Splunk employees worldwide.