Configure the Prometheus receiver to collect Seldon Core metrics

Configure the Splunk Distribution of the OpenTelemetry Collector to send Seldon metrics to Splunk Observability Cloud.

1. Install the Collector for Kubernetes using Helm.
2. To manually activate the Prometheus receiver for Seldon Core, make the following changes to your Collector values.yaml configuration file.
   1. Add prometheus/seldon to the receivers section. For example:
      YAMLCopy

      agent: config: receivers: # Add this dedicated receiver for Seldon prometheus/seldon: config: scrape_configs: - job_name: seldon-core kubernetes_sd_configs: - namespaces: names: - default # Add namespaces where Seldon is deployed - seldon-system # Controller metrics # - production # Add other namespaces as needed role: pod relabel_configs: # Only scrape pods with seldon-app label - action: keep regex: (.+) source_labels: - __meta_kubernetes_pod_label_seldon_app # Set scrape target to pod IP + port 8000 - replacement: $1:8000 source_labels: - __meta_kubernetes_pod_ip target_label: __address__ scrape_interval: 30s

      agent:
        config:
          receivers:
            # Add this dedicated receiver for Seldon
            prometheus/seldon:
              config:
                scrape_configs:
                - job_name: seldon-core
                  kubernetes_sd_configs:
                  - namespaces:
                      names:
                      - default           # Add namespaces where Seldon is deployed
                      - seldon-system     # Controller metrics
                      # - production      # Add other namespaces as needed
                    role: pod
                  relabel_configs:
                  # Only scrape pods with seldon-app label
                  - action: keep
                    regex: (.+)
                    source_labels:
                    - __meta_kubernetes_pod_label_seldon_app
                  # Set scrape target to pod IP + port 8000
                  - replacement: $1:8000
                    source_labels:
                    - __meta_kubernetes_pod_ip
                    target_label: __address__
                  scrape_interval: 30s

   2. Add prometheus/seldon to the metrics pipeline of the service section. For example:
      YAMLCopy

      service: pipelines: metrics: receivers: [prometheus/seldon]

      service:   
        pipelines:   
          metrics:   
            receivers: [prometheus/seldon]

3. To receive histogram metrics as native histograms in Splunk Observability Cloud, make the following changes to your Collector values.yaml file.
   Note:

   The send_otlp_histograms: true setting is required to use histogram functions in Splunk Observability Cloud and enables histograms to be received as a single metric. If this setting is not enabled, histogram metrics are received as multiple metrics separated by their function type. For example, the seldon_api_executor_server_requests_seconds histogram metric would be received as the following metrics:

   • seldon_api_executor_server_requests_seconds_bucket

   • seldon_api_executor_server_requests_seconds_count

   • seldon_api_executor_server_requests_seconds_sum

   For more information on histogram metrics, see Histogram metrics in Splunk Observability Cloud.

   1. Add send_otlp_histograms: true to the otlphttp/splunk exporter section. For example:
      YAMLCopy

      exporters: otlp: endpoint: https://ingest.<realm>.signalfx.com headers: X-SF-Token: ${SPLUNK_ACCESS_TOKEN} otlphttp/splunk: metrics_endpoint: https://ingest.<realm>.signalfx.com/v2/datapoint/otlp headers: X-SF-Token: ${SPLUNK_ACCESS_TOKEN} # Enable native histogram support send_otlp_histograms: true

      exporters:
        otlp:
          endpoint: https://ingest.<realm>.signalfx.com
          headers:
            X-SF-Token: ${SPLUNK_ACCESS_TOKEN}
        otlphttp/splunk:
          metrics_endpoint: https://ingest.<realm>.signalfx.com/v2/datapoint/otlp
          headers:
            X-SF-Token: ${SPLUNK_ACCESS_TOKEN}
          # Enable native histogram support
          send_otlp_histograms: true

   2. Add the otlphttp/splunk exporter to the metrics pipeline of the service section. For example:
      YAMLCopy

      service: pipelines: metrics: receivers: [receiver_creator, prometheus/seldon] exporters: [otlphttp/splunk]

      service:
        pipelines:
          metrics:
            receivers: [receiver_creator, prometheus/seldon]
            exporters: [otlphttp/splunk]

4. Restart the Splunk Distribution of the OpenTelemetry Collector.