Manage Splunk POD

Manage your Splunk POD cluster using the Kubernetes Installer to retrieve credentials, access network information, and manage apps.

Use the Kubernetes Installer for Splunk POD to manage your Splunk POD cluster. Get credentials, view network configuration details, install and update apps such as Splunk Enterprise Security (ES), and more.

Access Splunk Web

To access the Splunk Web interface and other management consoles, retrieve the auto-generated credentials and identify the correct service ports.

  1. From the installation directory on the bastion node, retrieve your admin password and the default HTTP Event Collector (HEC) token:
    CODE 
    ./kubernetes-installer-standalone -static.cluster <your_cluster_config>.yml -get.creds
  2. Access Splunk component interfaces using the IP address of any worker node and the corresponding port:
    Component Port URL Example
    SHC 1 UI (pod-medium and pod-large only) 443 https://<ANY_WORKER_IP>
    SHC 2 UI (pod-medium and pod-large only) 8100 https://<ANY_WORKER_IP>:8100
    Standalone SH 1 UI (pod-small only) 8000 https://<ANY_WORKER_IP>:8000
    Standalone SH 2 UI (pod-small only) 8001 https://<ANY_WORKER_IP>:8001
    Cluster Manager UI 1443 https://<ANY_WORKER_IP>:1443
    License Manager UI 2443 https://<ANY_WORKER_IP>:2443
    Monitoring Console UI 3443 https://<ANY_WORKER_IP>:3443

    About search head UI access

    The pod-small option does not support search head UI access on port 443. Port 443 provides UI access for search head clusters in pod-medium and pod-large only.

    For search head UI access in pod-small, you must specify at least one standalone search head in your cluster configuration. You can access the UI for standalone SH 1 and SH 2 on Port 8000 and 8001, respectively.

    For search head UI access in pod-medium and pod-large, you must specify at least one search head cluster in your cluster configuration. You can access the UI for SHC 1 and SHC 2 on 443 and 8100, respectively.

    For details on configuring standalone search heads in pod-small and search head clusters in pod-medium and pod-large, see the Static cluster configuration file reference.

View local documentation

Access documentation and installation-specific network information directly from your web browser.

Access Splunk POD documentation and installation-specific network information, including URLs and port mappings for search heads and other components, directly from your web browser.
  1. From the bastion node, start the local web server:
    CODE 
    ./kubernetes-installer-standalone --web [--web.port 8080]
  2. Open a web browser, then go to any worker IP address on the designated port (default is 8080):
    CODE 
    http://<BASTION IP>:8080/docs
Note: The local web server must be running to access the local documentation.

Manage apps

The Kubernetes Installer for Splunk POD uses the SOK app framework to manage apps within the Splunk POD cluster.

The Kubernetes Installer for Splunk POD uses the SOK (Splunk Operator for Kubernetes) app framework to manage apps within the Splunk POD cluster. The app framework installs new or updated apps at 15-minute intervals. For more information, see the SOK App Framework Resource Guide on GitHub.

  1. Copy the app package (.spl, .tgz, .tar.gz) to the bastion server.
  2. In the static cluster configuration file, add the app's file path using the scope for the target tier.

    Use the following YAML structures as templates for your configuration:

    • Standalone search head (non-ES): standalone[].apps.local
      CODE 
      standalone:
- name: sh
  apps:
    local: ["./path/to/app.tgz"]
    • Standalone search head (ES): standalone[].apps.premium
      CODE 
      standalone:
- name: es-sh
  apps:
    premium: ["./path/to/app.tgz"]
    • Search head cluster (non-ES): searchheadcluster[].apps.cluster
      CODE 
      searchheadcluster:
- name: core-shc
  apps:
    cluster: ["./path/to/app.tgz"]
    • Search head cluster (ES): searchheadcluster[].apps.premium
      CODE 
      searchheadcluster:
- name: es-shc
  apps:
    premium: ["./path/to/splunk_app_es-<version>.tgz"]
    • Search head cluster deployer: searchheadcluster[].apps.local
      CODE 
      searchheadcluster:
- name: core-shc
  apps:
    local: ["./path/to/app.tgz"]
    • Indexer tier: clustermanager.apps.cluster
      CODE 
      clustermanager:
  apps:
    cluster: ["./path/to/app.tgz"]
    • Cluster Manager: clustermanager.apps.local
      CODE 
      clustermanager:
  apps:
    local: ["./path/to/app.tgz"]
    • License Manager: licensemanager.apps.local
      CODE 
      licensemanager:
  apps:
    local: ["./path/to/app.tgz"]

    For more information on app scopes, see Deploy Splunk POD.

  3. Rerun the installer:
    CODE 
    ./kubernetes-installer-standalone -static.cluster <your_cluster_config>.yml -deploy

    Key app management considerations:

    • Deletion: The installer does not support app deletion. To remove an app's functionality, deactivates it through the Splunk UI or apps.conf.
    • Downgrades: SOK does not officially support app downgrades. If you accidentally install an older version, rerun the installer with the correct version.
    • Duplicates: If you list multiple files in the same scope that map to the same internal app name, the installer uses only the last file in the list.

Install Splunk Enteprise Security

Install and configure Enterprise Security (ES) on a Splunk POD cluster.

Splunk POD supports Splunk Enterprise Security (ES) on any search head tier (standalone or search head cluster).

To install ES on the Splunk POD cluster, add the following details to the static cluster configuration YAML file:

  • standalone stanza (pod-small): Specify a name of your choice in the standalone stanza, for example "es-sh". Splunk POD small supports ES installation on standalone search heads only.
  • searchheadcluster stanza (pod-medium, pod-large, pod-xlarge): Specify a name of your choice in the searchheadcluster stanza, for example "es-shc". Splunk POD medium and large support ES installation on search head clusters only.
  • ES app: List the ES app package under the standalone[].apps.premium scope. The premium scope applies to ES only.
  • Indexer apps: List the apps to install on indexers (e.g. Splunk_TA_ForIndexers) under the clustermanager.apps.cluster scope.

Configure static cluster configuration file for ES

Example: pod-small with ES. Deploys 2 standalone search heads (1 standard, 1 with ES):

CODE 
clustermanager:
 apps:
   cluster: ["./Splunk_TA_ForIndexers_8.1.1.tgz"]

standalone:
- name: core-sh
  apps:
    local: ["./performance-insights-for-splunk_118.tgz"]
- name: es-sh
  apps:
    local: []
    premium: ["./splunk_app_es-8.1.1.tgz"]

Example: pod-medium, pod-large, or pod-xlarge with ES. Deploys 2 SHCs (1 standard, 1 with ES):

CODE 
searchheadcluster:
# Port 443
- name: core-shc
  apps: 
    cluster: ["./performance-insights-for-splunk_118.tgz"]

# Port 8100
- name: es-shc
  apps: 
    cluster: []
    premium: ["./splunk_app_es-8.1.1-176740.tgz"]

For details on configuring standalone search heads and search head clusters, see Static cluster configuration file reference.

For more information, see Install Splunk Enterprise Security in a Splunk POD cluster.

Install Splunk IT Service Intelligence

Install and configure ITSI on a Splunk POD cluster.

Splunk POD supports the Splunk IT Service Intelligence (ITSI) premium app on any search head tier (standalone or search head cluster).

To install ITSI on the Splunk POD cluster, add the following details to the static cluster configuration YAML file:

  • standalone stanza (pod-small): Specify a name of your choice in the standalone stanza, for example "itsi-sh".
  • searchheadcluster stanza (pod-medium, pod-large, pod-xlarge): Specify a name of your choice in the searchheadcluster stanza, for example "itsi-shc".
  • Cluster Manager apps: List the required ITSI indexer app under the clustermanager.apps.cluster scope.
  • License Manager apps: List the required ITSI license apps under the licensemanager.apps.local scope.
  • Licenses: Specify the ITSI license file in the licenses array.

Installing ITSI in a Splunk POD cluster involves three high-level tasks:

  1. Unpack the ITSI bundle and repackage the individual ITSI sub-apps.
  2. Download OpenJDK 17 and embed it into a custom Splunk app named jdk.
  3. Map the ITSI component apps and the JDK app to specific Splunk tiers in the Splunk POD static cluster configuration file.

Unpack the ITSI bundle and repackage individual sub-apps

The ITSI app bundle includes all necessary ITSI sub-apps. Each sub-app must be packaged as a separate tarball before installation. Note that .spl files are the same as .tgz files.

  1. Download the ITSI app package.
  2. Unpack the ITSI tarball:
    CODE 
    tar -xzvf splunk-it-service-intelligence_<version>.spl
    Note: Splunk software uses the .spl file extension for app packages. This format is a gzip-compressed tarball, extractable with standard tar commands.
  3. Compress each sub-app directory found in the unpacked bundle into an individual tarball:
    CODE 
    tar -czvf <subapp>.tgz <subapp>

    Repeat for each sub-app in the unpacked bundle. You can delete the original sub-app directories after repackaging. Note the file paths of the sub-app tarballs for use in the static cluster configuration file.

Prepare the JDK custom app

ITSI requires a Java runtime. To provide it, create a custom Splunk app called jdk that contains an OpenJDK 17 installation. The folder must be named jdk .

  1. Create the app directory structure:
    CODE 
    mkdir -p jdk/default jdk/metadata

    The resulting directory structure:

    CODE 
    jdk/
├── default/
│   └── app.conf
└── metadata/
    └── default.meta
  2. Create the jdk/default/app.conf file with the following content:
    CODE 
    [install]
state = enabled
is_configured = true

[ui]
show_in_nav = false
  3. Create the jdk/metadata/default.meta file with the following content:
    CODE 
    []
access = read : [ * ], write : [ admin ]
export = system

[savedsearches]
owner = admin

[governance]
access = read : [ * ], write : [ * ]
  4. Download the OpenJDK 17 package from a verified source and unpack it. For example:
    CODE 
    tar -xzvf openjdk-17_linux-x64_bin.tar.gz
  5. Copy the unpacked JDK contents into the jdk custom Splunk app:
    CODE 
    cp -r <openjdk-directory>/* jdk/

    The resulting app structure:

    CODE 
    jdk/
├── bin/        ← java, javac, etc.
├── conf/
├── default/
│   └── app.conf
├── include/
├── jmods/
├── legal/
├── lib/
├── metadata/
│   └── default.meta
└── release
  6. Package the completed app into a tarball:
    CODE 
    tar -czvf jdk.tgz jdk

    Note the file paths of the sub-app tarballs for use in the static cluster configuration file.

Map ITSI apps to the static cluster configuration file

Map the ITSI sub-app tarballs and the jdk.tgz app to the correct tiers in the static cluster configuration file.

  1. Add the search head tier apps. List all ITSI sub-apps except SA-ITSI-Licensechecker, plus the jdk.tgz app, under the appropriate search head stanza.

    Example: pod-small with ITSI. Deploys 2 standalone search heads (1 standard, 1 with ITSI):

    CODE 
    standalone:
  - name: itsi-sh
    apps:
      local:
        - "./jdk.tgz"
        - "./itsi.tgz"
        - "./DA-ITSI-APPSERVER.tgz"
        - "./DA-ITSI-DATABASE.tgz"
        - "./DA-ITSI-EUEM.tgz"
        - "./DA-ITSI-LB.tgz"
        - "./DA-ITSI-OS.tgz"
        - "./DA-ITSI-STORAGE.tgz"
        - "./DA-ITSI-VIRTUALIZATION.tgz"
        - "./DA-ITSI-WEBSERVER.tgz"
        - "./SA-IndexCreation.tgz"
        - "./SA-ITOA.tgz"
        - "./SA-ITSI-AI-Summarization.tgz"
        - "./SA-ITSI-AlertCorrelation.tgz"
        - "./SA-ITSI-AT-Recommendations.tgz"
        - "./SA-ITSI-ATAD.tgz"
        - "./SA-ITSI-CustomModuleViz.tgz"
        - "./SA-ITSI-DriftDetection.tgz"
        - "./SA-ITSI-MetricAD.tgz"
        - "./SA-UserAccess.tgz"

    Example: pod-medium, pod-large, and pod-xlarge with ITSI. Deploys 2 SHCs (1 standard, 1 with ITSI):

    CODE 
    searchheadcluster:
  - name: itsi-shc
    apps:
      cluster:
        - "./jdk.tgz"
        - "./itsi.tgz"
        - "./DA-ITSI-APPSERVER.tgz"
        - "./DA-ITSI-DATABASE.tgz"
        - "./DA-ITSI-EUEM.tgz"
        - "./DA-ITSI-LB.tgz"
        - "./DA-ITSI-OS.tgz"
        - "./DA-ITSI-STORAGE.tgz"
        - "./DA-ITSI-VIRTUALIZATION.tgz"
        - "./DA-ITSI-WEBSERVER.tgz"
        - "./SA-IndexCreation.tgz"
        - "./SA-ITOA.tgz"
        - "./SA-ITSI-AI-Summarization.tgz"
        - "./SA-ITSI-AlertCorrelation.tgz"
        - "./SA-ITSI-AT-Recommendations.tgz"
        - "./SA-ITSI-ATAD.tgz"
        - "./SA-ITSI-CustomModuleViz.tgz"
        - "./SA-ITSI-DriftDetection.tgz"
        - "./SA-ITSI-MetricAD.tgz"
        - "./SA-UserAccess.tgz"

  2. Add the Cluster Manager apps. List SA-IndexCreation under the clustermanager.apps.cluster scope:

    CODE 
    clustermanager:
  apps:
    cluster:
      - "./SA-IndexCreation.tgz"

  3. Add the License Manager apps. List SA-ITSI-Licensechecker and SA-UserAccess under the licensemanager.apps.local scope:

    CODE 
    licensemanager:
  apps:
    local:
      - "./SA-ITSI-Licensechecker.tgz"
      - "./SA-UserAccess.tgz"

  4. Add the ITSI license. Specify the ITSI license file in the licenses array:

    CODE 
    licenses: ["enterprise.lic", "itsi.lic"]

For details on configuring standalone search heads and search head clusters, see Static cluster configuration file reference.

Create indexes

You cannot create indexes through the Splunk UI in a POD environment. Instead, you must define them within a custom app.

  1. Create an indexes.conf file and bundle it inside a custom app.
  2. Define homePath, coldPath, and thawedPath in the file.
    The system automatically applies SmartStore settings and a one-year (31,536,000 seconds) frozen retention period.
    CAUTION: Do not specify a replication factor (RF) or remotePath for any of your indexes. These values will be set for you in Splunk Enterprise.
  3. Deploy: Install the custom app using the clustermanager.apps.cluster scope.

Manage knowledge objects

The system handles knowledge objects differently based on their origin.

The system handles knowledge objects differently based on their origin.

  • Non-Local App Content (/default): The installer copies this content from the bastion host to SeaweedFS. You can upgrade this content, but you cannot delete it.
  • Local App Content (/local): Users create this content through the Splunk UI. The system replicates these objects among search heads (in a Search Head Cluster) and protects them from installer overwrites.
    Note: The installer framework does not back up local content externally.

Manage Universal Forwarders

The Kubernetes Installer does not manage or interact with Splunk Universal Forwarders (UF).

Because Splunk Universal Forwarders (UF) typically exist outside the Kubernetes cluster boundary, they are managed independently from POD lifecycle operations. This includes setting up a secure S2S connection that allows forwarders to send data to the cluster.

Key considerations for UF management:

  • Manual management: Use agent management (deployment server) on the bastion node or a third-party tool (such as Ansible) to manage forwarder configurations. See About agent management.
  • Security requirements: Forwarders require specific security apps and TLS certificates to communicate with the cluster.
  • Load balancing: Manually configure the outputs.conf server list with multiple worker node IPs or DNS names to ensure high availability.

For details on establishing secure S2S connections for Universal Forwarders, see Network routing and ingress for Splunk POD.

Use Federated Search in Splunk POD

Splunk POD supports Federated Search in standard mode, allowing you to run searches across multiple clusters.

Use Federated Search in standard mode to run searches across Splunk POD clusters or from a Splunk POD cluster to a remote deployment. No specific configuration changes to Splunk POD or the Splunk Kubernetes Installer are required to enable this feature.

Follow the Splunk documentation to configure Federated Search from the Splunk UI on the local deployment. Use the endpoint https://<any federated provider POD worker IP>:443. No configuration is required on the Splunk POD clusters acting as federated providers.

Federated Search limitations

  • Because every Splunk POD cluster uses the same server names for indexers (for example, splunk-idx-indexer-0), it is difficult to isolate search results for a specific cluster without using index filtering.
  • Default IP-based routing requires that Federated Search links use specific IP addresses. If a worker node goes offline or is replaced, you must manually re-establish the link to the cluster.

Monitor Splunk POD

Tools and ports used for monitoring Splunk Enterprise components and the POD environmemt.

For comprehensive monitoring of Splunk Enterprise components, including search heads, indexers, cluster manager, license manager, and so on, use the Splunk Monitoring Console. This is available on port 3443 of any worker node (https://<worker_node>:3443).

For more information, see About the Monitoring Console.

Splunk POD also provides access to Perses dashboards for Prometheus data, where you can view metrics related to the Kubernetes environment and non-Splunk services. Perses dashboards are available on port 3000 of any worker node (https://<worker_node>:3000). See Network routing and ingress for Splunk POD.