Troubleshoot data collection

Fix problems with collecting data from your Microsoft SQL Server database server or service.

Database platform isn't visible on the UI

Check these settings:

  • All the metrics that the Splunk Distribution of OpenTelemetry Collector collects by default are activated. Since these metrics are activated by default, you don't need to take any action unless you've deactivated them.

  • Your YAML override file is compatible with the latest changes in the Helm chart.

  • (For Microsoft SQL Server databases) Your OpenTelemetry collector's receivers.sqlserver.server is set to the fully qualified domain name (FQDN) or IP address of your Microsoft SQL Server database service. For example, in the context of a Kubernetes internal DNS system, if the service name is mssql-server and the namespace is mssqltest , receivers.sqlserver.server should be mssql-server.mssqltest.svc.cluster.local.

  • When navigating to a database instance from the Database Monitoring Overview page, you see at least six tabs. If only you only see three tabs (Queries, Query samples, and Query metrics), it means that your OpenTelemetry collector's metrics pipeline, which sends database infrastructure metrics collected by the database receiver, is not configured properly. For help with metric pipeline configuration, see the documentation for Microsoft SQL Server receiver or Oracle Database receiver.

Database platform metrics are duplicated on the UI

When deploying the Splunk Distribution of OpenTelemetry Collector in a Kubernetes cluster, it is crucial to understand the implications of configuring external service receivers (like sqlserver) under different deployment types.

Problem: Duplicate collection with agent: (DaemonSet)

If you configure a database receiver (such as sqlserver) under the agent: section of the Helm chart, the Splunk Distribution of OpenTelemetry Collector DaemonSet will deploy an agent pod on every node in your Kubernetes cluster. Each of these agent pods will then independently attempt to scrape the same external database instance.

This results in:

  • Duplicate metrics and events being sent to Splunk Observability Cloud.

  • Inaccurate data, inflated usage, and potential rate-limiting issues.

  • Unnecessary load on the database server from multiple scraping instances.

How to detect duplicate collection:

On the Infrastructure > Datastores > Database Server Instances (OTel) page, you can see two instances with the same name but different hosts. While not a 100% guarantee, this is a sign that the receiver is configured under the agent: component and collects data from one instance twice.

Solution: Use clusterReceiver: (Deployment)

To avoid duplicate collection for external services like databases, always configure their receivers under the clusterReceiver: section. The clusterReceiver component is deployed as a Kubernetes Deployment, allowing you to specify a single replica (replicas: 1). This ensures that only one instance of the collector is responsible for scraping the database, preventing any duplication.

See Helm chart architecture and components documentation for more information about clusterReceiver.

Example of correct configuration

The following configuration demonstrates the recommended approach:

  • agent: is enabled for node-specific collection (such as host metrics and kubelet stats).

  • clusterReceiver: is enabled with replicas: 1 and configured specifically for the sqlserver receiver, ensuring unique collection.

# collector-values-override.yaml (Illustrative Snippet)
splunkObservability:
  realm: "YOUR_REALM"
  accessToken: YOUR_ACCESS_TOKEN
  ingestUrl: YOUR_INGEST_URL

clusterName: dashbase-cluster

agent:
  enabled: true # Keep agent enabled for node-level metrics
  config:
    exporters:
      signalfx:
        api_url: YOUR_API_URL
    # Ensure no database receivers are configured here to avoid duplication.
    # Agent should focus on node-specific data (hostmetrics, kubeletstats, etc.)

clusterReceiver: # Use clusterReceiver for external database scraping
  enabled: true
  replicas: 1 # Crucial: ensures only one instance scrapes the database
  config:
    exporters:
      signalfx:
        api_url: YOUR_API_URL
      # Exports dbmon events as logs
      otlphttp/dbmon:
        headers:
          X-SF-Token: YOUR_ACCESS_TOKEN
          X-splunk-instrumentation-library: dbmon
        logs_endpoint: YOUR_INGEST_URL/v3/event
        sending_queue:
          batch:
            flush_timeout: 15s
            max_size: 10485760 # 10 MiB
            sizer: bytes
    receivers:
      sqlserver:
        collection_interval: 10s
        username: YOUR_DB_USERNAME
        password: YOUR_DB_PASSWORD # REMINDER: Use Kubernetes Secrets for production!
        server: YOUR_DB_SERVER_ADDRESS # Replace with your actual DB host
        port: 1433
        resource_attributes:
          sqlserver.instance.name:
            enabled: true
        events:
          db.server.query_sample:
            enabled: true
          db.server.top_query:
            enabled: true
    service:
      pipelines:
        metrics:
          receivers:
            - sqlserver
          processors:
            - memory_limiter
            - batch
            - resourcedetection
            - resource
          exporters:
            - signalfx
        logs/dbmon:
          receivers:
            - sqlserver
          processors:
            - memory_limiter
            - batch
            - resourcedetection
          exporters:
            - otlphttp/dbmon