App Framework limitations and troubleshooting

Limitations and troubleshooting for the App Framework.

Known limitations of the App Framework and troubleshooting guidance for app deployment issues.

Limitations

The App Framework does not preview, analyze, verify versions, or enable Splunk Apps and Add-ons. The administrator is responsible for previewing the app or add-on contents, verifying the app is enabled, and that the app is supported with the version of Splunk Enterprise deployed in the containers. For Splunk app packaging specifications see Package apps for Splunk Cloud or Splunk Enterprise in the Splunk Enterprise Developer documentation. The app archive files must end with .spl or .tgz; all other files are ignored.

  1. The App Framework has no support to remove an app or add-on once deployed. To disable an app, update the archive contents located in the App Source, and set the app.conf state to disabled.
  2. The App Framework defines one worker per CR type. For example, if you have multiple clusters receiving app updates, a delay while managing one cluster delays the app updates to the other cluster.

Troubleshooting

The AppFramework feature stores data about the installation of applications in Splunk Enterprise Custom Resources Status subresource.

The field cr.status.AppDeploymentContext.AppsSrcDeployStatus stores the AppFramework deployment statuses. Further, each Application under every Application source has detailed deployment information in cr.status.AppDeploymentContext.AppsSrcDeployStatus.AppDeploymentInfo.

Phase information

The process of installing an application is divided into multiple sequential phases. Each Application has its current phase information in cr.status.AppDeploymentContext.AppsSrcDeployStatus.AppDeploymentInfo.PhaseInfo.

Phase 1 - App package download

Status Code Description
101 App Package is pending download
102 App Package download is in progress
103 App Package download is complete
199 App Package is not downloaded after multiple retries

Phase 2 - App package copy

Status Code Description
201 App Package is pending copy
202 App Package copy is in progress
203 App Package copy is complete
298 Downloaded App Package is missing on Operator pod PVC
299 App Package is not copied after multiple retries

Phase 3 - App package install

Status Code Description
301 App Package is pending install
302 App Package install is in progress
303 App Package install is complete
398 Copied App Package is missing on Splunk Enterprise pod PVC
399 App Package is not copied after multiple retries

Example of Standalone with successful Application install. CR spec:

CODE 
apiVersion: enterprise.splunk.com/v4
kind: Standalone
metadata:
  name: test
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  replicas: 1
  appRepo:
    appsRepoPollIntervalSeconds: 100
    defaults:
      volumeName: volume_app_repo_us
      scope: local
    appSources:
    - name: dummy
      location: dummy/
      volumeName: volume_app_repo_us
    volumes:
    - name: volume_app_repo_us
      storageType: s3
      provider: aws
      path: test/cspl_1250_apps/
      endpoint: https://s3-us-west-2.amazonaws.com
      region: us-west-2
      secretRef: s3-secret

CR status:

JSON 
bash# kubectl get stdaln -o yaml | grep -i appSrcDeployStatus -A 33
      appSrcDeployStatus:
        dummy:
          appDeploymentInfo:
          - appName: a.tgz
            appPackageTopFolder: testapp
            auxPhaseInfo:
            - phase: install
              status: 303
            deployStatus: 3
            isUpdate: false
            objectHash: ab78...89
            phaseInfo:
              phase: install
              status: 303
            repoState: 1
          - appName: b.tgz
            appPackageTopFolder: newapp
            auxPhaseInfo:
            - phase: install
              status: 303
            deployStatus: 3
            isUpdate: false
            objectHash: 8745a....876
            phaseInfo:
              phase: install
              status: 303
            repoState: 1
      appsRepoStatusPollIntervalSeconds: 100
      appsStatusMaxConcurrentAppDownloads: 5
      bundlePushStatus: {}
      isDeploymentInProgress: false
      lastAppInfoCheckTime: 1719277376
      version: 1

Bundle push status

The AppFramework uses a bundle push to install applications in clustered environments such as IndexerCluster and SearchHeadCluster.

Status Code Description
0 Bundle push is uninitialized, to be scheduled
1 Bundle Push is pending, waiting for all the apps to be copied to the Pod
2 Bundle Push is in progress
3 Bundle Push is complete

Search Head Cluster example. CR spec:

CODE 
apiVersion: enterprise.splunk.com/v4
kind: SearchHeadCluster
metadata:
  name: shc
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  replicas: 3
  appRepo:
    appsRepoPollIntervalSeconds: 100
    defaults:
      volumeName: volume_app_repo_us
      scope: cluster
    appSources:
    - name: dummy
      location: dummy/
      volumeName: volume_app_repo_us
    volumes:
    - name: volume_app_repo_us
      storageType: s3
      provider: aws
      path: test/cspl_1250_apps/
      endpoint: https://s3-us-west-2.amazonaws.com
      region: us-west-2
      secretRef: s3-secret

SHC CR status:

CODE 
bash# kubectl get shc -o yaml | grep -i appSrcDeployStatus -A 33
      appSrcDeployStatus:
        dummy:
          appDeploymentInfo:
          - appName: a.tgz
            appPackageTopFolder: "testapp"
            deployStatus: 1
            isUpdate: false
            objectHash: 67ab7....876
            phaseInfo:
              phase: install
              status: 303
            repoState: 1
          - appName: b.tgz
            appPackageTopFolder: "newapp"
            deployStatus: 1
            isUpdate: false
            objectHash: 876abc....987
            phaseInfo:
              phase: install
              status: 303
            repoState: 1
      appsRepoStatusPollIntervalSeconds: 100
      appsStatusMaxConcurrentAppDownloads: 5
      bundlePushStatus:
        bundlePushStage: 3
      isDeploymentInProgress: false
      lastAppInfoCheckTime: 1719281420
      version: 1
    captain: splunk-shc-search-head-0
    captainReady: true
    deployerPhase: Ready
    initialized: true
    maintenanceMode: true
    members:

App not updating after new file uploaded

The App Framework maintains a checksum via Etag. The Etag should update if the contents differ. If not updating, verify that the Etag differs. The splunk-operator-controller-manager pod logs contain Etag info. Look for the log:

PYTHON 
INFO	initAndCheckAppInfoStatus	Apps List retrieved from remote storage	{"controller"..., "App Source": "searchApps", "Content": [{"Etag":"etag","Key":"path/file.tgz","LastModified":"2025-06-06T15:15:01Z","Size":263096926,"StorageClass":"STANDARD"}]}

Based on appsRepoStatusPollIntervalSeconds, it might take some time. Wait the required time for the app framework to poll the latest versions from storage.

App not correctly copying to pod

If an app is packaged on a Mac, there might be issues extracting the contents on the running pod. If there are issues installing apps that were packaged manually, ensure that they can be extracted in a Linux environment before uploading to the remote storage.