Cisco Deep Time Series Model on- premises installation

The Cisco Deep Time Series Model (CDTSM) is a pretrained, generative AI model designed for forecasting metric time series data across the Splunk platform. For more information on CDTSM including use cases, syntax, and parameters, see Feature preview: Cisco Deep Time Series Model.

You can install the Cisco Deep Time Series Model on your own servers so that on-premises Splunk deployments can use CDTSM capabilities.The self-hosted CDTSM service runs outside Splunk and exposes the following HTTP inference endpoint:
CODE 
POST http://<hostname>:8080/cdtsm/v1/ai/infer
The Splunk AI Toolkit reads that endpoint from the mlspl.conf file. When | apply CDTSM runs, the AI Toolkit sends the time-series payload to the local model server and receives the forecast back.
Note: The CDTSM is a feature preview and participation in this preview is optional. You can provide feedback on this preview at https://voc.splunk.com/preview/ctsm_aitk.

Preview disclaimer

Beta features described in this document are provided by Splunk to you "as is" without any warranties, maintenance and support, or service-level commitments. Splunk makes this Beta feature available at its sole discretion and may discontinue it at any time. Use of Beta features is subject to the Splunk Pre-Release Agreement for Hosted Services.

Prerequisites

See the following table for the prerequisites to self-install the feature preview Cisco Deep Time Series Model (CDTSM):

Requirement Notes
Splunk Enterprise or Splunk Cloud with AI Toolkit The AI Toolkit app must be installed at $SPLUNK_HOME/etc/apps/Splunk_ML_Toolkit.
Docker and Docker Compose Docker Desktop is the easiest option. GPU is recommended for performance but is not required. The server runs on CPU as well.
Network access Required for the first model download.

Model weights (~1 GB) are fetched from Hugging Face (cisco-ai/cisco-time-series-model-1.0) when the service first loads. After the initial download the weights are cached in a Docker named volume (hf_cache) and no further internet access is needed.

If your server does not have internet access, see the Installing on air-gapped or restricted-network servers section before proceeding.

Installation steps

Complete these steps to connect a self-hosted CDTSM server to the Splunk AI Toolkit.

  1. Get the server files. Only clone the serve folder from the Cisco time-series model repository.
    CODE 
    git clone --depth 1 --sparse https://github.com/splunk/cisco-time-series-model.git
cd cisco-time-series-model
git sparse-checkout set serve
cd serve

  2. Set a user-defined bearer token. Create one token. Use this same value in both the model server and Splunk encrypted storage.

    CODE 
    export CDTSM_AUTH_TOKEN=$(openssl rand -hex 32)
  3. Put the token in the server .env. The FastAPI server reads the token from .env before Docker starts.
    CODE 
    printf "CDTSM_AUTH_TOKEN=%s\n" "$CDTSM_AUTH_TOKEN" > .env

  4. Start CDTSM with Docker.

    1. Build and launch the local FastAPI inference service on port 8080:
      CODE 
      docker compose up --build
    2. In a new terminal, wait until the model is ready:
      CODE 
      curl http://127.0.0.1:8080/ready
      Note: /ready returns 503 while the model weights are downloading and loading. This is normal on first start. Poll until it returns 200.
  5. Update the AI Toolkit mlspl.conf file. Edit $SPLUNK_HOME/etc/apps/Splunk_ML_Toolkit/local/mlspl.conf and add the following stanza:
    CODE 
    [CTSM]
self_hosted_cdtsm_endpoint = http://<hostname>:8080/cdtsm/v1/ai/infer
self_hosted_cdtsm_timeout = 300
self_hosted_cdtsm_model = CDTSM
    CAUTION: Do not edit the default/mlspl.conf file for any permanent changes. Always use local/mlspl.conf so any app upgrades do not overwrite your settings.
  6. Register the token with Splunk. The AI Toolkit reads the bearer token from Splunk encrypted storage passwords, not from mlspl.conf. Register the token as follows:
    CODE 
    $SPLUNK_HOME/bin/splunk _internal call \
  /servicesNS/nobody/Splunk_ML_Toolkit/storage/passwords \
  -method POST \
  -post:realm aitk_fm_tokens \
  -post:name CDTSM_AUTH_TOKEN \
  -post:password "$CDTSM_AUTH_TOKEN"

  7. Restart and test.

    1. Restart Splunk to pick up the new mlspl.conf and storage password:
      CODE 
      $SPLUNK_HOME/bin/splunk restart
    2. Once Splunk is back up, navigate to the AI Tooklit app and run a test from the Search bar. The following example uses the "internet_traffic.csv" file from the AI Toolkit Showcase:
      CODE 
      | inputlookup internet_traffic.csv 
| apply CDTSM bits_transferred
CAUTION: The token in serve/.env must exactly match the Splunk storage password registered as CDTSM_AUTH_TOKEN. A mismatch causes HTTP 401 on every inference request.

Installing on air-gapped or restricted-network servers

Many on-premises deployments run on servers with no direct internet access. In these environments, the model weights must be downloaded separately on an internet-connected machine and transferred to the target server before starting Docker.

Air-gapped servers without internet access

Complete the following steps on 2 machines: 1 with internet access for the download, and 1 for the air-gapped target server for the install.

  1. On an internet-connected machine, download the model weights:
    PYTHON 
    pip install huggingface_hub
python -c "from huggingface_hub import snapshot_download; snapshot_download('cisco-ai/cisco-time-series-model-1.0')"
    The weights are saved to ~/.cache/huggingface/hub/ by default.

  2. Transfer the cache to the air-gapped server via USB drive, internal file share, or similar:
    CODE 
    rsync -av ~/.cache/huggingface/hub/models--cisco-ai--cisco-time-series-model-1.0 \
  user@air-gapped-host:~/.cache/huggingface/hub/
  3. On the air-gapped server, add the following line to serve/.env before starting Docker. This tells the Hugging Face client to use the local cache and skip all network calls:
    CODE 
    HF_HUB_OFFLINE=1

  4. Continue from Step 4 of the regular, non air-gapped installation instructions in the previous section. Docker will load the weights from the local cache instead of downloading them.

Servers behind a corporate TLS-inspecting proxy

If the server has internet access but sits behind a proxy that performs Transform Layer Security (TLS) inspection, the Hugging Face client might fail to verify the proxy's certificate. Add one of the following to serve/.env:
CODE 
# Option A — provide your corporate CA bundle (recommended)
CDTSM_HF_SSL_CA_BUNDLE=/path/to/corporate-ca-bundle.pem

# Option B — disable TLS verification (dev/lab environments only)
CDTSM_HF_INSECURE_SSL=true
Note: For Docker deployments the CDTSM_HF_SSL_CA_BUNDLE must point to a path that exists inside the container, not on the host.
Mount your CA bundle into the container and set the variable to the in-container path. For example, in docker-compose.yml:
CODE 
services:
    cdtsm:
      volumes:
        - /path/to/corporate-ca-bundle.pem:/etc/cdtsm/corporate-ca-bundle.pem:ro

  then in serve/.env:

  CDTSM_HF_SSL_CA_BUNDLE=/etc/cdtsm/corporate-ca-bundle.pem

Troubleshooting

See the following table for issues you might encounter and how to resolve them.

Issue Resolution
Server does not start Check the container logs:
CODE 
docker compose logs
Model stuck loading The first run downloads approximately 1 GB of weights from Hugging Face. Check download progress:
CODE 
docker compose logs -f
Port 8080 already in use
Change the port in .env:
CODE 
CDTSM_PORT=8081
Update self_hosted_cdtsm_endpoint in mlspl.conf to match, then restart both the server and Splunk.
401 Unauthorized on inference Verify the token in .env matches what is stored in Splunk. Test the server directly:
CODE 
curl -H "Authorization: Bearer $CDTSM_AUTH_TOKEN" http://127.0.0.1:8080/health
Model weights not found after reinstall
Weights are cached at ~/.cache/huggingface/hub/. Force a re-download:
CODE 
docker compose down -v
docker compose up --build