Create a Databricks connection
Create a Databricks connection in Catalog that authenticates federated searches over Unity Catalog datasets using Delta Sharing.
To set up Federated Search for Databricks on your Splunk Cloud Platform deployment, you must first define an Databricks connection in your Splunk Cloud Platform deployment's Catalog. This connection leverages the Databricks OpenSharing model to authenticate access to Unity Catalog schemas and tables, allowing you to create federated search datasets for those Unity Catalog data objects.
Establishing this connection requires a secure OpenSharing credential file (config.share) that contains a unique bearer token and the sharing endpoint. To obtain this file, you must create a share (a securable object representing the specific Unity Catalog schemas, tables, and views you want to search) and a recipient representing users of your Splunk platform deployment. When you assign the recipient to the share, Databricks generates the credential file within your Databricks workspace.
After you download the credential file from your Databricks workspace, you upload it to your Databricks connection in the Data Management app. This file serves as the primary authentication mechanism, allowing users of your Splunk platform deployment to securely access and search the Unity Catalog data objects included in your Databricks share.
- Your Splunk Cloud Platform deployment user account must have a role with the
edit_connectionsandedit_datasetscapabilities. See Define roles on the Splunk platform with capabilities in Securing Splunk Cloud Platform. -
Use the Catalog to create and maintain your Databricks connection. See Discover Data Using the Catalog.
- You must have access to a Microsoft Azure or Amazon Web Services (AWS) Databricks workspace with a runtime of 11.3 LTS or higher that contains the data you want to share and which is assigned to a Unity Catalog metastore. See Enable a workspace for Unity Catalog.
-
Note: The Splunk platform currently only supports federated search of data in Azure and AWS Databricks tables.You must have appropriate privileges to access and manage the Databricks data assets that you want your users to be able to search through this connection. You must be an admin for the Unity Catalog metastore where the data you want to share is registered, or you must have the following privileges:
CREATE SHAREandCREATE RECIPIENTfor the Unity Catalog metastore that contains the data you want to share. See Metastore admins.USE CATALOGandUSE SCHEMAfor the catalog and schemas that contain the data assets that you want to share. See Manage privileges in Unity Catalog.
- You must obtain an Databricks OpenSharing credentials file that lets the Splunk platform access your Unity Catalog schemas, tables, and views through the Databricks connection. To obtain this file, complete the following tasks in your Databricks workspace:
Task Databricks documentation Create a share object for OpenSharing. When you create the share, select the data assets you want to add to the share object. Create a share object Create a recipient for your share that uses bearer tokens. Ensure the recipient type is Open and the authentication type is Token. Give the token a Token lifetime of up to 365 days. Create a recipient that uses bearer tokens Obtain the OpenSharing credentials file. When you create a share recipient, you get an activation link to a page where you can download the credentials file (config.share). Note: You can access this activation link until it is used to download the credentials file. After the file is downloaded, Databricks does not display the activation link for the recipient.Get an activation link