EUM Pods Restart Frequently

EUM pods in Virtual Appliance restart frequently or terminate with the OOMKilled status.

This issue occurs if the following conditions exist:
  • The default liveness probe is set to one second. Therefore, aggressive liveness probe timeouts.
  • Insufficient CPU or memory resources for high traffic environments.
Open a case with Splunk AppDynamics support to increase liveness probe timeout and JVM heap size.
Important: The following steps are intended for Splunk AppDynamics Support personnel only. Do not attempt to perform these actions unless explicitly directed by a support representative. Unauthorized modifications may impact your application stability or support eligibility.
  1. Increase the liveness and readiness probe.
    JSON 
    kubectl patch statefulset eum-ss -n cisco-eum --type='json' -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/0/livenessProbe/timeoutSeconds", "value": 30},
  {"op": "replace", "path": "/spec/template/spec/containers/0/livenessProbe/failureThreshold", "value": 10},
  {"op": "replace", "path": "/spec/template/spec/containers/0/livenessProbe/periodSeconds", "value": 60},
  {"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/timeoutSeconds", "value": 30},
  {"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/failureThreshold", "value": 10}
]'
  2. Increase the JVM heap size depending on the Virtual Appliance profile.
    Small
    CODE 
    Xms: 3G
Xmx: 5G
    Medium
    CODE 
    Xms: 4G
Xmx: 8G
    Large
    CODE 
    Xms: 4G
Xmx: 11G
    Extra Large
    CODE 
    Xms: 5G
Xmx: 14G

    Example Commands:

    An example command to find the EUM_PROCESSOR_OPTS environment variable on the Stateful set.
    CODE 
    kubectl get statefulset eum-ss -n cisco-eum -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}={.value}{"\n"}{end}' | grep -n EUM_PROCESSOR
    • Add the following code with the JVM heap size if the EUM_PROCESSOR_OPTS environment variable is missing:
      JSON 
      kubectl patch statefulset eum-ss -n cisco-eum --type='json' -p='[
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/env/-",
    "value": {
      "name": "EUM_PROCESSOR_OPTS",
      "value": "-Xms<XMS_VALUE> -Xmx<XMX_VALUE>"
    }
  }
]'
    • Edit the following code with correct JVM heap size in the EUM_PROCESSOR_OPTS environment variable:
      JSON 
      kubectl patch statefulset eum-ss -n cisco-eum --type='json' -p='[
  {
    "op": "replace",
    "path": "/spec/template/spec/containers/0/env/7",
    "value": {
      "name": "EUM_PROCESSOR_OPTS",
      "value": "-Xms<XMS_VALUE> -Xmx<XMX_VALUE>"
    }
  }
]'