Configure SmartStore access with IRSA

Configure SmartStore S3 access using AWS IAM Role for Service Account (IRSA).

Splunk Enterprise 9.0.5 and higher supports SmartStore access using AWS IAM Role for Service Account (IRSA) for fine-grained access control to S3 resources.

Understand the following concepts before configuring IRSA:

  • AWS Identity and Access Management (IAM) provides fine-grained access control where you can specify who can access which AWS service or resources, ensuring the principle of least privilege.
  • Kubernetes pods receive an identity through a Kubernetes Service Account. When you create a Service Account, a JWT token is automatically created as a Kubernetes Secret. This Secret can then be mounted into pods and used by that Service Account to authenticate to the Kubernetes API Server.
  • AWS introduced IAM Roles for Service Accounts (IRSA), using AWS Identity APIs, an OpenID Connect (OIDC) identity provider, and Kubernetes Service Accounts to apply fine-grained access controls to Kubernetes pods.
  • In Kubernetes, the ProjectedServiceAccountToken feature allows a fully compliant OIDC JWT token issued by the TokenRequest API to be mounted into the pod as a Projected Volume. The relevant Service Account Token Volume Projection flags are enabled by default on an EKS cluster.
  • AWS created an identity webhook that comes preinstalled in an EKS cluster. This webhook listens to create pod API calls and can inject an additional token into splunkd pods. You can install this webhook into self-managed Kubernetes clusters on AWS using this guide.
  1. Follow the steps in IAM Roles for Service Accounts to create an IAM Role for Service Account.
  2. Ensure the IAM Role has only the least amount of privilege necessary for SmartStore to work.
  3. Annotate the service account with the specific IAM Role, and add the service account to the Splunk custom resource where required.

    The following example adds the service account to a Standalone instance:

    CODE 
    apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: test
spec:
  serviceAccount: oidc-service-account
  smartstore:
    defaults:
        volumeName: test-cluster-bucket
    indexes:
      - name: main
        remotePath: $_index_name
        volumeName: test-cluster-bucket
      - name: cloudwatch
        remotePath: $_index_name
        volumeName: test-cluster-bucket
    volumes:
      - name: test-cluster-bucket
        path: test-cluster-bucket/smartstore
        endpoint: https://s3-us-west-2.amazonaws.com
  4. Verify that the IAM service account is used only in the custom resources where it is required.

When the Splunk pod runs, the AWS webhook service injects two new environment variables, AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN, along with the JWS Token file. The splunk pod reads these environment variables to get temporary AWS credentials from the AWS IAM service to access SmartStore buckets.

OIDC key management

Proper key management of OIDC is outside of the Splunk installation. You are responsible for using a properly configured OIDC with certificates from a trusted CA.

Self-signed certificate

The OIDC should not use self-signed certificates. Instead, use an existing PKI infrastructure. For example, have the OIDC certificate issued and signed by your organization CA with proper certificate signature chains and key expiration policies.

Sharing OIDC token file

Ensure that the token file specified in the AWS_WEB_IDENTITY_TOKEN_FILE location is only accessible inside of the pod and is not mapped or shared outside of the pod.