Manage Splunk POD

Manage your Splunk POD cluster using the Splunk Kubernetes Installer to retrieve credentials, access network information, and manage apps.

Use the Kubernetes Installer for Splunk POD to manage your Splunk POD cluster. Get credentials, access live network connections, install and update apps such as Splunk Enterprise Security (ES), and more.

Access Splunk Web

To access the Splunk Web interface and other management consoles, retrieve the auto-generated credentials and identify the correct service ports.

  1. On the bastion node in the installation directory, run this command to retrieve your admin password and the default HTTP Event Collector (HEC) token:
    CODE 
    ./kubernetes-installer-standalone -static.cluster <your_cluster_config>.yml -get.creds
  2. Access Splunk component interfaces using the IP address of any worker node and the corresponding port:
    Component Port URL Example
    SHC 1 UI (pod-medium and pod-large only) 443 https://<ANY_WORKER_IP>
    SHC 2 UI (pod-medium and pod-large only) 8100 https://<ANY_WORKER_IP>:8100
    Standalone SH 1 UI (pod-small only) 8000 https://<ANY_WORKER_IP>:8000
    Standalone SH 2 UI (pod-small only) 8001 https://<ANY_WORKER_IP>:8001
    Cluster Manager UI 1443 https://<ANY_WORKER_IP>:1443
    License Manager UI 2443 https://<ANY_WORKER_IP>:2443
    Monitoring Console UI 3443 https://<ANY_WORKER_IP>:3443

    About search head UI access

    The pod-small option does not support search head UI access on port 443. Port 443 provides UI access for search head clusters in pod-medium and pod-large only.

    For search head UI access in pod-small, you must specify at least one standalone search head in your cluster configuration. You can access the UI for standalone SH 1 and SH 2 on Port 8000 and 8001, respectively.

    For search head UI access in pod-medium and pod-large, you must specify at least one search head cluster in your cluster configuration. You can access the UI for SHC 1 and SHC 2 on 443 and 8100, respectively.

    For details on configuring standalone search heads in pod-small and search head clusters in pod-medium and pod-large, see the Static cluster configuration file reference.

View local documentation

Access documentation and installation-specific network information directly from your web browser.

Access Splunk POD documentation and installation-specific network information, including URLs and port mappings for search heads and other components, directly from your web browser.
  1. From the bastion node, start the local web server:
    CODE 
    ./kubernetes-install-standalone –web [--web.port 8080]
  2. Open a web browser, then go to any worker IP address on the designated port (default is 8080):
    CODE 
    http://<BASTION IP>:8080/docs
Note: The local web server must be running to access the local documentation.

Manage apps

The Splunk Kubernetes Installer uses the SOK app framework to manage apps within the Splunk POD cluster.

The Splunk Kubernetes Installer uses the SOK app framework to manage apps within the Splunk POD cluster. The app framework installs new or updated apps at 15-minute intervals. For more information, see the App Framework Resource Guide.

  1. Copy the app package (.spl, .tgz, .tar.gz) to the bastion server.
  2. In the static cluster configuration file, add the app's file path using the scope for the target tier.

    Use the following YAML structures as templates for your configuration:

    • Indexer tier (clustermanager.apps.cluster)
      CODE 
      clustermanager:
  apps:
    cluster: ["./path/to/app.tgz"]
    • Cluster manager (clustermanager.apps.local):
      CODE 
      clustermanager:
  apps:
    local: ["./path/to/app.tgz"]
    • Search head clusters (non-premium) (searchheadcluster[].apps.cluster):
      CODE 
      searchheadcluster:
- name: core-shc
    apps:
      cluster: ["./path/to/app.tgz"]
    • Search head clusters (premium/ES) (searchheadcluster[].apps.premium):
      CODE 
      searchheadcluster:
- name: es-shc
    apps:
      premium: ["./path/to/app.tgz"]
    • Search head cluster deployer (searchheadcluster[].apps.local):
      CODE 
      searchheadcluster:
- name: core-shc
    apps:
      local: ["./path/to/app.tgz"]
    • Standalone search heads (non-premium) (standalone[].apps.local):
      CODE 
      standalone:
- name: sh
    apps:
      local: ["./path/to/app.tgz"]
    • Standalone search heads (premium/ES) (standalone[].apps.premium):
      CODE 
      standalone:
- name: es-sh
    apps:
      premium: ["./path/to/app.tgz"]

    For more information on app scopes, see Deploy Splunk POD.

  3. Rerun the installer:
    CODE 
    ./kubernetes-installer-standalone -static.cluster <your_cluster_config>.yml -deploy

    Key app management considerations:

    • Deletion: The installer does not support app deletion. To remove an app's functionality, disable it through the Splunk UI or apps.conf.
    • Downgrades: SOK does not officially support app downgrades. If you accidentally install an older version, rerun the installer with the correct version.
    • Duplicates: If you list multiple files in the same scope that map to the same internal app name, the installer uses only the last file in the list.

Install Splunk Enterprise Security

To install the Enterprise Security (ES) premium app on your Splunk POD cluster, add the following items to your static cluster configuration file.

To install the Splunk Enterprise Security (ES) premium app on the Splunk POD cluster, add the following details to the static cluster configuration YAML file:

  • standalone stanza (pod-small): Specify a name of your choice in the standalone stanza, for example es-sh. Splunk POD small supports ES installation on standalone search heads only.
  • searchheadcluster stanza (pod-medium and pod-large): Specify a name of your choice in the searchheadcluster stanza, for example es-shc. Splunk POD medium and large support ES installation on search head clusters only.
  • Premium apps: List the ES app package under the standalone[].apps.premium scope.
  • Indexer apps: List the apps to install on indexers (e.g. Splunk_TA_ForIndexers) under the clustermanager.apps.cluster scope.
Update the static cluster configuration file with the appropriate stanzas and app packages.

Example: pod-small with ES. (Deploys 2 standalone search heads: one standard and one with ES):

CODE 
clustermanager:
 apps:
 cluster: ["./Splunk_TA_ForIndexers_8.1.1.tgz"]

standalone:
- name: core-sh
    apps:
      local: ["./performance-insights-for-splunk_118.tgz"]
 - name: es-sh
    apps:
      local: []
 premium: ["./splunk_app_es-8.1.1.tgz"]

Example: pod-medium and pod-large with ES. (Deploys 2 SHCs: one standard and one with ES):

CODE 
searchheadcluster:
# Port 443
- name: core-shc
    apps: 
 cluster: ["./performance-insights-for-splunk_118.tgz"]

# Port 8100
- name: es-shc
    apps: 
      cluster: []
 premium: ["./splunk_app_es-8.1.1-176740.tgz"]

For details on configuring standalone search heads and search head clusters, see Static cluster configuration file reference.

Create indexes

You cannot create indexes through the Splunk UI in a POD environment. Instead, you must define them within a custom app.

  1. Create an indexes.conf file and bundle it inside a custom app.
  2. Define homePath, coldPath, and thawedPath in the file.
    The system automatically applies SmartStore settings and a one-year (31,536,000 seconds) frozen retention period.
    CAUTION: Do not specify a replication factor (RF) or remotePath for any of your indexes. These values will be set for you in Splunk Enterprise.
  3. Deploy: Install the custom app using the clustermanager.apps.cluster scope.

Manage knowledge objects

The system handles knowledge objects differently based on their origin.

The system handles knowledge objects differently based on their origin.

  • Non-Local App Content (/default): The installer copies this content from the bastion host to SeaweedFS. You can upgrade this content, but you cannot delete it.
  • Local App Content (/local): Users create this content through the Splunk UI. The system replicates these objects among search heads (in a Search Head Cluster) and protects them from installer overwrites.
    Note: The installer framework does not back up local content externally.

Manage Universal Forwarders

The Kubernetes Installer does not manage or interact with Splunk Universal Forwarders (UF).

Manage Universal Forwarders (UF) independently of the installer. This includes setting up a secure S2S connection that allows forwarders to send data to the cluster.

Key considerations for UF management:

  • Manual management: Use agent management (deployment server) on the bastion node or a third-party tool (such as Ansible) to manage forwarder configurations. See About agent management.
  • Security requirements: Forwarders require specific security apps and TLS certificates to communicate with the cluster.
  • Load balancing: Manually configure the outputs.conf server list with multiple worker node IPs or DNS names to ensure high availability.

For details on establishing secure S2S connections for Universal Forwarders, see Network routing and ingress for Splunk POD.

Use Federated Search in Splunk POD

Splunk POD supports Federated Search in standard mode, allowing you to run searches across multiple clusters.

Use Federated Search in standard mode to run searches across Splunk POD clusters or from a Splunk POD cluster to a remote deployment. No specific configuration changes to Splunk POD or the Splunk Kubernetes Installer are required to enable this feature.

Follow the Splunk documentation to configure Federated Search from the Splunk UI on the local deployment. Use the endpoint https://<any federated provider POD worker IP>:443. No configuration is required on the Splunk POD clusters acting as federated providers.

Federated Search limitations

  • Because every Splunk POD cluster uses the same server names for indexers (for example, splunk-idx-indexer-0), it is difficult to isolate search results for a specific cluster without using index filtering.
  • Default IP-based routing requires that Federated Search links use specific IP addresses. If a worker node goes offline or is replaced, you must manually re-establish the link to the cluster.

Monitor Splunk POD

Tools and ports used for monitoring Splunk Enterprise components and the POD environmemt.

For comprehensive monitoring of Splunk Enterprise components, including search heads, indexers, cluster manager, license manager, and so on, use the Splunk Monitoring Console. This is available on port 3443 of any worker node (https://<worker_node>:3443).

For more information, see About the Monitoring Console.