kubectl-splunk plugin

A kubectl plugin for managing Splunk instances in Kubernetes pods.

About kubectl-splunk

The kubectl-splunk plugin runs Splunk commands inside Splunk pods on Kubernetes and supports REST calls, file copy, interactive shells, credentials, and configuration. Release and build metadata for the package appear on PyPI and in the splunk-operator repository.

kubectl-splunk is a kubectl plugin that runs Splunk commands directly inside Splunk pods in a Kubernetes cluster. It simplifies management of Splunk instances deployed as StatefulSets or Deployments by providing a command-line interface.

The plugin supports these capabilities:

  • Run Splunk CLI commands inside Splunk pods.
  • Run Splunk REST API calls through port-forwarding.
  • Select pods on the command line or in a configuration file.
  • Retrieve Splunk admin credentials from pods automatically.
  • Open an interactive shell in Splunk pods.
  • Copy files to and from Splunk pods.
  • Handle authentication with credential storage.
  • Customize configuration and verbosity.
  • Run on multiple platforms.
  • Support shell auto-completion.

Feature summary

  • Run Splunk commands: Run Splunk CLI commands inside the pod.
  • REST API support: Run Splunk REST API calls through port-forwarding.
  • Pod selection: Specify a pod on the command line, with an environment variable, or in a configuration file. If you do not specify a pod, the script prompts when multiple pods exist.
  • Automatic credential retrieval: Defaults to the admin user and reads the password from the pod when you do not provide one.
  • Interactive shell: Start a shell session inside the Splunk pod.
  • Copy files: Transfer files to and from Splunk pods.
  • Authentication handling: Handle Splunk credentials securely, including saving them.
  • Configuration flexibility: Use configuration files or environment variables for defaults.
  • Verbosity control: Adjust logging levels.
  • Caching: Cache pod information to improve performance.
  • Auto-completion: Shell auto-completion for commands and options.
  • Secure logging: Passwords and other sensitive data are not logged.

Prerequisites

Before you use kubectl-splunk, install Python 3, kubectl, and required Python packages, and confirm access to the cluster and Splunk CLI in pods. Meet these requirements before you install or run kubectl-splunk.

  • Install Python 3 on your system.
  • Install and configure kubectl, the Kubernetes command-line tool.
  • Confirm you have access to the Kubernetes cluster where Splunk runs.
  • Confirm the Splunk CLI is available inside the Splunk pods.
  • Install required Python packages:
    CODE 
    pip install requests argcomplete

Install kubectl-splunk

Install kubectl-splunk from PyPI with pipx or pip inside a virtual environment, then verify the installation. Use pipx for an isolated environment that avoids dependency conflicts with other Python tools. You can also install with pip inside a virtual environment.

Install with pipx (recommended)

  1. Install pipx if it is not already installed:
    CODE 
    python3 -m pip install --user pipx
python3 -m pipx ensurepath

    You might need to restart your shell or run source ~/.bashrc or source ~/.zshrc to update your PATH.

  2. Install kubectl-splunk with pipx:
    CODE 
    pipx install kubectl-splunk

Alternative: pip in a virtual environment

  1. Create and activate a virtual environment:
    CODE 
    python3 -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
  2. Install the package:
    CODE 
    pip install kubectl-splunk

Verify installation

Confirm that kubectl-splunk is on your PATH:

CODE 
kubectl splunk --help

You should see the help output, which confirms a successful install.

Usage

Use kubectl splunk with a global options prefix, a mode, and mode-specific options. Modes include exec, rest, cp, and interactive.

Basic command structure

CODE 
kubectl splunk [global options] <mode> [mode options]

Modes of operation

  • exec: Run a Splunk command inside a Splunk pod.
  • rest: Run a Splunk REST API call through port-forwarding.
  • cp: Copy files to or from a Splunk pod.
  • interactive: Start an interactive shell inside a Splunk pod (use the --interactive flag).

Global options

Global options, authentication options, and mode-specific options for kubectl splunk exec, rest, and cp.

  • -n, --namespace: Kubernetes namespace (default: default or from config or environment).
  • -l, --selector: Label selector for Splunk pods (default: app=splunk or from config or environment).
  • --context: Kubernetes context.
  • -P, --pod: Exact pod name. Set this value in config or environment, or pass it on the command line.
  • -i, --interactive: Start an interactive shell in the Splunk pod.
  • --splunk-path: Path to the Splunk CLI in the container (default: splunk or from config or environment).
  • --local-port: Local port for port-forwarding in REST mode (default: 8000 or from config or environment).
  • -v: Increase verbosity (-v, -vv, or -vvv).
  • --version: Print the program version and exit.

Authentication options

  • -u, --username: Splunk username (default: admin).
  • -p, --password: Splunk password. The plugin prompts for the password or detects it automatically if you omit it.
  • --insecure: Turn off SSL certificate verification (for example for self-signed certificates).
  • --save-credentials: Save credentials securely for future use.

exec mode

  • Usage: kubectl splunk exec [splunk_command]
  • splunk_command: Splunk command to run (for example list user).

rest mode

  • Usage: kubectl splunk rest METHOD ENDPOINT [options]
  • METHOD: HTTP method (GET, POST, PUT, DELETE).
  • ENDPOINT: Splunk REST API path (for example /services/server/info).
  • --data: Request body for POST or PUT.
  • --params: Query parameters (for example "key1=value1&key2=value2").

cp mode

  • Usage: kubectl splunk cp SRC DEST
  • SRC: Source file path.
  • DEST: Destination file path.
  • Use : to mark the remote path in the pod (for example :/path/in/pod).

Examples

Example kubectl splunk commands for exec, search, namespace and selector, pod selection, interactive shell, copy, REST, authentication, verbosity, and context.

Run a Splunk command

CODE 
kubectl splunk exec status

Run a Splunk search command

CODE 
kubectl splunk exec search "index=_internal | head 10"

Specify namespace and label selector

CODE 
kubectl splunk -n splunk-namespace -l app=splunk exec status

Specify a pod directly

CODE 
kubectl splunk --pod splunk-idxc-indexer-0 exec status

Or use the short form:

CODE 
kubectl splunk -P splunk-idxc-indexer-0 exec status

Start an interactive shell

CODE 
kubectl splunk --interactive

Copy files to a pod

CODE 
kubectl splunk cp /local/path/file.txt :/remote/path/file.txt

Copy files from a pod

CODE 
kubectl splunk cp :/remote/path/file.txt /local/path/file.txt

Run a REST API call

CODE 
kubectl splunk rest GET /services/server/info --insecure

Create a search job (POST request)

CODE 
kubectl splunk rest POST /services/search/jobs --data "search=search index=_internal | head 10" --insecure

Use authentication and save credentials

CODE 
kubectl splunk -u admin --save-credentials exec list user

Increase verbosity

CODE 
kubectl splunk -vv exec status

Specify Kubernetes context

CODE 
kubectl splunk --context my-cluster exec status

Configuration file

Set defaults in ~/.kubectl_splunk_config, override them with environment variables, and understand how the plugin selects a pod.

Create a configuration file to set default values.

File: ~/.kubectl_splunk_config

Example content:

CODE 
[DEFAULT]
namespace = splunk-namespace
selector = app=splunk
splunk_path = splunk
pod_name = splunk-idxc-indexer-0  # Default pod name
local_port = 8000                 # Default local port for REST mode
  • namespace: Default Kubernetes namespace.
  • selector: Default label selector for Splunk pods.
  • splunk_path: Path to the Splunk CLI inside the container.
  • pod_name: Default pod name when you do not pass one on the command line.
  • local_port: Default local port for port-forwarding in REST mode.

Environment variables

You can set environment variables to override defaults:

  • KUBECTL_SPLUNK_NAMESPACE: Default namespace.
  • KUBECTL_SPLUNK_SELECTOR: Default label selector.
  • KUBECTL_SPLUNK_PATH: Default Splunk CLI path.
  • KUBECTL_SPLUNK_POD: Default pod name.
  • KUBECTL_SPLUNK_LOCAL_PORT: Default local port for REST mode.

Example:

CODE 
export KUBECTL_SPLUNK_NAMESPACE=splunk-namespace
export KUBECTL_SPLUNK_SELECTOR=app=splunk
export KUBECTL_SPLUNK_POD=splunk-idxc-indexer-0
export KUBECTL_SPLUNK_LOCAL_PORT=8000

Pod selection behavior

The script chooses a pod in this order:

  1. Command-line argument: --pod or -P takes highest priority.
  2. Environment variable: When you set KUBECTL_SPLUNK_POD and you did not pass a pod on the command line, the script uses that value.
  3. Configuration file: When you did not pass a pod and you did not set KUBECTL_SPLUNK_POD, the script reads pod_name from ~/.kubectl_splunk_config.
  4. Interactive selection: When you did not specify a pod and multiple pods match, the script lists them and prompts you to choose.
  5. Automatic selection: When only one pod matches, the script selects it.

Authentication: default credentials

Default Splunk credentials, password retrieval from the pod, and secure credential storage in ~/.kubectl_splunk_credentials.

  • Username: Defaults to admin if you do not set one.
  • Password: If you do not provide a password, the script tries to read it from /mnt/splunk-secrets/password in the pod.
  • Automatic password retrieval: The script reads the password from the pod when the pod allows reading that file.

Usage without credentials:

CODE 
kubectl splunk exec list user

Authentication: credential storage

  • Save credentials: Use --save-credentials to store credentials for future use.
  • Credentials file: The plugin writes credentials to ~/.kubectl_splunk_credentials with mode 600.

Provide credentials once:

CODE 
kubectl splunk -u admin --save-credentials exec list user

Subsequent commands reuse the saved credentials automatically.

Attention: The plugin handles passwords securely and does not print them in logs or command output.

REST API mode details

Use rest mode to run Splunk REST API calls through port-forwarding.

CODE 
kubectl splunk rest METHOD ENDPOINT [--data DATA] [--params PARAMS] [options]
  • METHOD: HTTP method (GET, POST, PUT, DELETE).
  • ENDPOINT: Splunk REST API path (for example /services/server/info).
  • --data: Data for POST or PUT.
  • --params: Query parameters (for example "key1=value1&key2=value2").
  • --insecure: Turn off SSL certificate verification.

Examples

Get server info:

CODE 
kubectl splunk rest GET /services/server/info --insecure

Create a search job:

CODE 
kubectl splunk rest POST /services/search/jobs --data "search=search index=_internal | head 10" --insecure

Copy mode details

Use cp mode to copy files to or from a Splunk pod.

Copy to pod:

CODE 
kubectl splunk -P splunk-idxc-indexer-0 cp /local/path/file.txt :/remote/path/file.txt

Copy from pod:

CODE 
kubectl splunk -P splunk-idxc-indexer-0 cp :/remote/path/file.txt /local/path/file.txt
  • Use : to mark the remote path in the pod.
  • cp mode requires a single pod. Specify it with --pod, -P, an environment variable, or the configuration file.
  • If multiple pods match and you did not specify a pod, the script prompts you to choose.

Interactive shell details

Start an interactive shell in a Splunk pod with --interactive:

CODE 
kubectl splunk --interactive
  • If you set the pod on the command line, in an environment variable, or in the config file, the script uses that pod.
  • If multiple pods match and you did not specify a pod, the script prompts you to choose.

Logging and verbosity

Control verbosity with -v, manage the pod cache file under /tmp, and configure argcomplete-based shell completion for kubectl-splunk.

Adjust logging with the -v flag:

  • -v: Warnings and errors.
  • -vv: Informational messages.
  • -vvv: Debug messages.

Example:

CODE 
kubectl splunk -vv exec status
Attention: Logs mask sensitive data such as passwords.

Caching

For 5 minutes after each run, the plugin caches the pod name from that execution. The plugin stores the cache in /tmp/kubectl_splunk_cache.json.

  • If you set the pod name on the command line, in an environment variable, or in the configuration file, the cache uses that pod.
  • If you do not set a pod name and multiple pods match, the script caches the pod you select.
  • The cache expires after 5 minutes.

Clear the cache:

CODE 
rm /tmp/kubectl_splunk_cache.json

Auto-completion

Install the argcomplete package:

CODE 
pip install argcomplete

Activate global completion:

CODE 
activate-global-python-argcomplete --user

Add the following to your shell initialization file (for example .bashrc, .bash_profile, or .zshrc):

CODE 
eval "$(register-python-argcomplete kubectl-splunk)"

Reload your shell configuration:

CODE 
source ~/.bashrc  # or your shell config file

Troubleshooting

Resolve ambiguous options, missing pods, multiple pod prompts, copy mode limits, permissions, cache issues, and password retrieval failures.

Ambiguous option error

If you see an error such as ambiguous option: --p could match --pod, --password, use full option names: --pod or -P for the pod, and --password or -p for the password.

Pod not found

Verify the pod name and namespace. List pods with kubectl get pods -n <namespace>.

Multiple pods

If multiple pods match and you did not specify a pod, the plugin prompts you to choose. To skip the prompt, set the pod with --pod, -P, an environment variable, or the configuration file.

Copy mode limitations

cp mode requires a single pod. Specify the pod on the command line, in an environment variable, or in the configuration file.

Permission denied

Confirm you have permission to access the Kubernetes cluster and the Splunk pods.

Caching issues

If you suspect the script cached an outdated pod name, delete the cache file.

Password retrieval failure

Confirm you can run commands in the pod and read /mnt/splunk-secrets/password when the file exists.

License

The kubectl-splunk project uses the Apache License 2.0. Contribute through GitHub by forking, branching, testing, and opening pull requests.

This project is licensed under the Apache License 2.0.

Contributing

Contributions are welcome. Submit issues and pull requests through the project GitHub repository.

  1. Fork the repository: Use the Fork action on the GitHub repository page.
  2. Clone your fork:
    CODE 
    git clone https://github.com/splunk/splunk-operator.git
cd tools/kubectl-splunk
  3. Create a feature branch:
    CODE 
    git checkout -b feature/your-feature-name
  4. Make your changes and follow the project coding standards.
  5. Run tests before you commit:
    CODE 
    python -m unittest discover -s tests
  6. Commit and push to your fork, then open a pull request on the upstream repository.

Coding standards

  • Follow PEP 8 for Python style.
  • Write meaningful docstrings for modules, classes, and functions.
  • Keep code documented and maintainable.

Reporting issues

Open an issue on the GitHub Issues page with enough detail to reproduce the problem.

Feedback

Thank you for using kubectl-splunk. This plugin helps you manage Splunk on Kubernetes.