Create an Azure Databricks connection

Create an Azure Databricks connection in the Data Management app to authenticate federated searches over Unity Catalog datasets using Delta Sharing.

Note: In the Controlled Availability release stage, Splunk products may have limitations on customer access, features, maturity, and regional availability. For additional information on Controlled Availability please contact your Splunk representative.

To set up Federated Search for Azure Databricks on your Splunk Cloud Platform deployment, you must first define an Azure Databricks connection in the Data Management app. This connection leverages the Azure Databricks Delta Sharing open sharing model to authenticate access to Unity Catalog schemas and tables, allowing you to create federated search datasets for those Unity Catalog data objects.

Establishing this connection requires a secure Delta Sharing credential file (config.share) that contains a unique bearer token and the sharing endpoint. To obtain this file, you must create a share (a securable object representing the specific Unity Catalog schemas, tables, and views you want to search) and a recipient representing users of your Splunk platform deployment. When you assign the recipient to the share, Databricks generates the credential file within your Azure Databricks workspace.

After you download the credential file from your Azure Databricks workspace, you upload it to your Azure Databricks connection in the Data Management app. This file serves as the primary authentication mechanism, allowing users of your Splunk platform deployment to securely access and search the Unity Catalog data objects included in your Azure Databricks share.

  • Your Splunk Cloud Platform deployment user account must have a role with the edit_datasets and edit_federated_providers capabilities. See Define roles on the Splunk platform with capabilities in Securing Splunk Cloud Platform.
  • You must have access to an Azure Databricks workspace with a runtime of 11.3 LTS or higher that contains the data you want to share and which is assigned to a Unity Catalog metastore. See Enable Unity Catalog for a workspace.
  • You must have appropriate privileges to access and manage the Azure Databricks data assets that you want your users to be able to search through this connection. You must be an admin for the Unity Catalog metastore where the data you want to share is registered, or you must have the following privileges:
    • CREATE SHARE and CREATE RECIPIENT for the Unity Catalog metastore that contains the data you want to share.
    • USE CATALOG and USE SCHEMA for the catalog and schemas that contain the data assets that you want to share. For details about privilege management in Unity Catalog, see Manage privileges in Unity Catalog.
  • You must obtain an Azure Databricks Delta Sharing credentials file that lets the Splunk platform access your Unity Catalog schemas, tables, and views through your connection. To obtain this file, complete the following tasks in your Azure Databricks workspace:
    Task Azure Databricks documentation
    Create a share object for Delta Sharing. When you create the share, select the data assets you want to add to the share object. Create a share object
    Create a recipient for your share that uses bearer tokens. Ensure the recipient type is Open and the authentication type is Token. Give the token a Token lifetime of up to 365 days. Create a recipient that uses bearer tokens
    Obtain the Delta Sharing credentials file. When you create a share recipient, you get an activation link to a page where you can download the credentials file (config.share).
    Note: You can access this activation link until it is used to download the credentials file. After the file is downloaded, Databricks does not display the activation link for the recipient.
    		Get an activation link
  1. On your Splunk Cloud Platform deployment, in Splunk Web, open the Data Management app.
  2. Open the Connections page in the Data Management app.
  3. Select Create connection.
  4. On the Select data store page, select Azure Databricks, and then Next.
  5. On the General page, configure the following settings, and then select Next:
    Setting Description
    Connection name Enter a unique name for the connection. The connection name must start with a lower-case letter and can contain only lower-case alphanumeric characters, underscores, and hyphens.
    Connection description (Optional) Provide a description of the connection.
  6. On the Delta Sharing authentication page, upload your Delta Sharing credential file.
  7. Take note of the Credential file expiration date that appears after you upload the file.

    When the file expires, this Azure Databricks connection and any datasets associated with it cease to function. You can update the lifetime for the recipient bearer token or rotate bearer tokens to get a new credential file with a new expiration date. See Manage recipient tokens in the Azure Databricks Documentation.

    Note: If you update the token lifetime for a recipient bearer token in Azure Databricks, you do not see the expiration date for the credential file change in your Data Management app UI.
  8. Select Next.
  9. Review your connection details and select Create to create the connection.
After you create a connection that authorizes access to a Delta Lake share containing one or more Unity Catalog tables or views, define a dataset that represents a specific table or view in that Unity Catalog share. See Define an Azure Databricks dataset.