Monitoring and autoscaling for ingestion separation

Autoscaling and monitoring reference for ingestion separation.

Use the Horizontal Pod Autoscaler to scale IngestorCluster replicas based on demand, and Grafana dashboards to monitor ingestion and indexing metrics.

Horizontal Pod Autoscaler

To automatically adjust the number of replicas to serve the ingestion traffic effectively, use the Horizontal Pod Autoscaler (HPA) to scale the workload based on actual demand. HPA monitors specified metrics and removes replicas when traffic decreases, or adds replicas when traffic exceeds the capacity of current resources.

The following example configures a HorizontalPodAutoscaler named ingestor-hpa to scale the IngestorCluster custom resource named ingestor. With average utilization set to 50, the HPA keeps the average CPU utilization of the pods at 50%. The replicas scale from a minimum of 3 to a maximum of 10.

CODE 
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ingestor-hpa
spec:
  scaleTargetRef:
    apiVersion: enterprise.splunk.com/v4
    kind: IngestorCluster
    name: ingestor
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

For more information, see Horizontal Pod Autoscaling on Kubernetes Docs.

Grafana monitoring dashboard

Install and configure Grafana on the cluster to monitor ingestion and indexing resources on a dashboard with diagrams and metrics such as number of replicas, resource consumption, network I/O, and persistent volume usage.

The following Grafana dashboard JSON presents ingestion and indexing data:

JSON 
{
  "id": null,
  "uid": "splunk-autoscale",
  "title": "Splunk Ingestion & Indexer Autoscaling with I/O & PV",
  "schemaVersion": 27,
  "version": 12,
  "refresh": "5s",
  "time": { "from": "now-30m", "to": "now" },
  "timezone": "browser",
  "style": "dark",
  "tags": ["splunk","autoscale","ingestion","indexer","io","pv"],
  "graphTooltip": 1,
  "panels": [
    { "id": 1, "type": "stat", "title": "Ingestion Replicas",
      "gridPos": {"x":0,"y":0,"w":4,"h":4},
      "targets":[{"expr":"kube_statefulset_replicas{namespace=\"default\",statefulset=\"splunk-ingestor-ingestor\"}"}],
      "options": {"reduceOptions":{"calcs":["last"]},"orientation":"horizontal","colorMode":"value","graphMode":"none","textMode":"value"}},
    { "id": 2, "type": "stat", "title": "Indexer Replicas",
      "gridPos": {"x":4,"y":0,"w":4,"h":4},
      "targets":[{"expr":"kube_statefulset_replicas{namespace=\"default\",statefulset=\"splunk-indexer-indexer\"}"}],
      "options": {"reduceOptions":{"calcs":["last"]},"orientation":"horizontal","colorMode":"value","graphMode":"none","textMode":"value"}},
    { "id": 3, "type": "timeseries", "title": "Ingestion CPU (cores)",
      "gridPos": {"x":8,"y":0,"w":8,"h":4},
      "targets":[{"expr":"sum(rate(container_cpu_usage_seconds_total{namespace=\"default\",pod=~\"splunk-ingestor-ingestor-.*\"}[1m]))","legendFormat":"CPU (cores)"}]},
    { "id": 4, "type": "timeseries", "title": "Ingestion Memory (MiB)",
      "gridPos": {"x":16,"y":0,"w":8,"h":4},
      "targets":[{"expr":"sum(container_memory_usage_bytes{namespace=\"default\",pod=~\"splunk-ingestor-ingestor-.*\"}) / 1024 / 1024","legendFormat":"Memory (MiB)"}]},
    { "id": 5, "type": "timeseries", "title": "Ingestion Network In (KB/s)",
      "gridPos": {"x":0,"y":8,"w":8,"h":4},
      "targets":[{"expr":"sum(rate(container_network_receive_bytes_total{namespace=\"default\",pod=~\"splunk-ingestor-ingestor-.*\"}[1m])) / 1024","legendFormat":"Net In (KB/s)"}]},
    { "id": 6, "type": "timeseries", "title": "Ingestion Network Out (KB/s)",
      "gridPos": {"x":8,"y":8,"w":8,"h":4},
      "targets":[{"expr":"sum(rate(container_network_transmit_bytes_total{namespace=\"default\",pod=~\"splunk-ingestor-ingestor-.*\"}[1m])) / 1024","legendFormat":"Net Out (KB/s)"}]},
    { "id": 7, "type": "timeseries", "title": "Indexer CPU (cores)",
      "gridPos": {"x":16,"y":4,"w":8,"h":4},
      "targets":[{"expr":"sum(rate(container_cpu_usage_seconds_total{namespace=\"default\",pod=~\"splunk-indexer-indexer-.*\"}[1m]))","legendFormat":"CPU (cores)"}]},
    { "id": 8, "type": "timeseries", "title": "Indexer Memory (MiB)",
      "gridPos": {"x":0,"y":12,"w":8,"h":4},
      "targets":[{"expr":"sum(container_memory_usage_bytes{namespace=\"default\",pod=~\"splunk-indexer-indexer-.*\"}) / 1024 / 1024","legendFormat":"Memory (MiB)"}]},
    { "id": 9, "type": "timeseries", "title": "Indexer Network In (KB/s)",
      "gridPos": {"x":8,"y":12,"w":8,"h":4},
      "targets":[{"expr":"sum(rate(container_network_receive_bytes_total{namespace=\"default\",pod=~\"splunk-indexer-indexer-.*\"}[1m])) / 1024","legendFormat":"Net In (KB/s)"}]},
    { "id": 10, "type": "timeseries", "title": "Indexer Network Out (KB/s)",
      "gridPos": {"x":16,"y":12,"w":8,"h":4},
      "targets":[{"expr":"sum(rate(container_network_transmit_bytes_total{namespace=\"default\",pod=~\"splunk-indexer-indexer-.*\"}[1m])) / 1024","legendFormat":"Net Out (KB/s)"}]},
    { "id": 11, "type": "timeseries", "title": "Ingestion Disk Read (KB/s)",
      "gridPos": {"x":0,"y":16,"w":8,"h":4},
      "targets":[{"expr":"sum(rate(container_fs_reads_bytes_total{namespace=\"default\",pod=~\"splunk-ingestor-ingestor-.*\"}[1m])) / 1024","legendFormat":"Disk Read (KB/s)"}]},
    { "id": 12, "type": "timeseries", "title": "Ingestion Disk Write (KB/s)",
      "gridPos": {"x":8,"y":16,"w":8,"h":4},
      "targets":[{"expr":"sum(rate(container_fs_writes_bytes_total{namespace=\"default\",pod=~\"splunk-ingestor-ingestor-.*\"}[1m])) / 1024","legendFormat":"Disk Write (KB/s)"}]},
    { "id": 13, "type": "timeseries", "title": "Indexer PV Usage (GiB)",
      "gridPos": {"x":0,"y":20,"w":8,"h":4},
      "targets":[{"expr":"kubelet_volume_stats_used_bytes{namespace=\"default\",persistentvolumeclaim=~\".*-indexer-.*\"} / 1024 / 1024 / 1024","legendFormat":"Used GiB"},
                 {"expr":"kubelet_volume_stats_capacity_bytes{namespace=\"default\",persistentvolumeclaim=~\".*-indexer-.*\"} / 1024 / 1024 / 1024","legendFormat":"Capacity GiB"}]},
    { "id": 14, "type": "timeseries", "title": "Ingestion PV Usage (GiB)",
      "gridPos": {"x":8,"y":20,"w":8,"h":4},
      "targets":[{"expr":"kubelet_volume_stats_used_bytes{namespace=\"default\",persistentvolumeclaim=~\".*-ingestor-.*\"} / 1024 / 1024 / 1024","legendFormat":"Used GiB"},
                 {"expr":"kubelet_volume_stats_capacity_bytes{namespace=\"default\",persistentvolumeclaim=~\".*-ingestor-.*\"} / 1024 / 1024 / 1024","legendFormat":"Capacity GiB"}]}
  ]
}

For more information on installing the monitoring stack, see the kube-prometheus-stack Helm chart.