Configure external Kafka

Configure Splunk AppDynamics Virtual Appliance to connect to self-managed Kafka brokers.

Provide an external Kafka deployment that is compatible with Apache Kafka 4.2.0 and meets the requirements for authentication, ACLs, topic configuration, replication, and consumer groups.

Virtual Appliance validates Kafka readiness (networking, brokers, listeners, identities, certificates). Topics and per-consumer ACLs are created during deployment external Kafka validation and reconciliation.

Storage Requirements

Ensure that you have sufficient storage on Kafka deployment to create required PVCs.
Profile Storage
Medium 6Gi
Extra Large 6Gi
  1. Prepare Kafka according to the contract in the target Virtual Appliance package.

    The contract defines the compatible Kafka version, required topics, topic settings, consumer groups, principals, and ACLs.

    • Provide at least three available broker replicas for application topics.
    • Provide separate bootstrap endpoint lists for mTLS and SASL/SCRAM-SHA-512 clients.
    • Make broker statefulServices.kafka.external. bootstrapServers values resolvable and reachable from OpenShift pods.
    • Use server certificates that cover every bootstrap and advertised broker name.
    • Provide the complete CA chain that validates every broker certificate.
    Note: The Kafka contract used to validate and configure the external Kafka deployment is available at appd-charts/charts/kafka-external/files/kafka-contract.json. This JSON file defines the required topics, configuration, authentication, permissions, and minimum capacity.
  2. Create the Kafka bootstrap admin identity.

    Create a SCRAM-SHA-512 administrator account with permission to inspect the Kafka state and add or update the required topics, configurations, and ACLs.

    Note: The credentials are exposed only to the prerequisite reconciliation job, it will be removed later.
  3. Create consumer service identities.
    Consumer Authentication Identity
    Controller SASL/SCRAM-SHA-512 kafka-controller-user
    Schema Registry mTLS User:CN=schema-registry-user
    Data Ingestion Services, Events, Metrics, and Snapshots mTLS User:CN=kafka-dis-user
    EUM mTLS User:CN=kafka-eum-user
    Secure Application mTLS User:CN=kafka-secureapp-user
    Anomaly Detection mTLS User:CN=kafka-aiops-user
  4. Create the validation-only mTLS principal User:CN=kafka-validator-audit.

    Grant only the permissions required to describe the Kafka cluster, required topics, topic configurations, and ACLs. Do not grant produce, consume, or superuser access.

  5. Verify Kafka network and TLS access from OpenShift.

    OpenShift workloads must reach every configured mTLS and SCRAM bootstrap endpoint and every broker endpoint returned in Kafka metadata.

    Confirm forward and return routes, DNS, firewall rules, NetworkPolicies, listener ports, and certificate SANs. Route mTLS clients to mTLS listeners and the Controller to the SCRAM listener.

    Important: Access to a load balancer or bootstrap port does not prove access to every broker.
  6. Configure the Kafka endpoints in config/external-kafka.yaml.

    Preserve schemaVersion: 1 and the validation settings supplied with the package.

    CODE 
    statefulServices:
  kafka:
    schemaVersion: 1
    external:
      bootstrapServers:
        mtls:
          - "broker-1.example.internal:9093"
          - "broker-2.example.internal:9093"
          - "broker-3.example.internal:9093"
        scram:
          - "broker-1.example.internal:9094"
          - "broker-2.example.internal:9094"
          - "broker-3.example.internal:9094"

    Enter each endpoint as host:port. Do not include a scheme, path, username, password, or certificate. Keep the packaged validator image and default timeout settings.

  7. Configure Kafka secrets.

    Add the broker CA, bootstrap admin identity credentials, application certificates and private keys, Controller password and audit certificate and private key to `config/secrets.yaml`. Virtual Appliance creates and distributes the Kubernetes Secrets.

    Use leaf-first PEM client certificate chains and unencrypted PEM private keys. Do not change kafka.controllerUser.password; that value belongs to embedded Kafka.

  8. Verify external Kafka during installation.

    The read-only validator checks DNS, TCP, TLS, host name identity, authentication, broker metadata, topics, topic settings, principals, and ACLs.

    If validation fails, inspect the output of the appdcli around Checking external Kafka configuration for errors and warnings.
    CODE 
    ./appdcli setup external-kafka

    Correct Kafka, the network, the configuration, or the source Secrets, and then run the installation again. Do not delete or relabel a Secret to bypass an ownership error.