Connect Apache Spark to an Open Sharing analytics table

Use a downloaded external sharing profile and the Delta Sharing Spark connector to read a shared Machine Data Lake analytics table.

  • You downloaded an external sharing profile for the Machine Data Lake analytics table.

  • Your Apache Spark environment uses Spark, Scala, Java, and Delta Sharing Spark connector versions that Splunk has tested for Open Sharing.

  • Your Apache Spark environment can load packages from Maven Central or from an approved internal artifact repository.

  • You know the share name, schema name, and table name that the client must read.

The Delta Sharing Spark connector reads a shared table by using a table path. The table path is the local sharing profile path followed by # and the fully qualified table name: <profile-file-path>#<share-name>.<schema-name>.<table-name>.

This example shows a batch read of the latest snapshot. For the current Machine Data Lake sharing workflow, do not depend on time travel, Change Data Feed, streaming, Delta table versions, or commit history unless Splunk documents support for those features.

  1. Save the external sharing profile in a location that the Apache Spark driver can read.

    Treat the profile as a secret. Do not store the profile in source control, shared logs, notebooks, or chat messages.

  2. Start PySpark with the compatible Delta Sharing Spark connector package.
    CODE 
    pyspark --packages io.delta:delta-sharing-spark_2.12:<compatible_delta_sharing_spark_version>

    Replace <compatible_delta_sharing_spark_version> with the version that Splunk has tested for Open Sharing. If you run Spark in a managed cluster, add the same Maven coordinate through the cluster library or dependency configuration that your Spark platform supports.

  3. Build the table path from the profile path and the fully qualified table name.
    CODE 
    profile_file = r"C:\Users\<user>\.delta_sharing\profiles\mdl-open-sharing.share"
table_path = profile_file + "#<share_name>.<schema_name>.<table_name>"
  4. Read the shared table as a Spark DataFrame.
    CODE 
    df = spark.read.format("deltaSharing").load(table_path)

df.printSchema()
df.show(10, truncate=False)
  5. Confirm that the returned schema and sample rows match the authorized analytics workflow.

    If required fields are missing, review the analytics table schema. Create a new analytics table promotion with the required fields if the shared table does not contain the fields that the Spark workflow needs.

Spark reads the latest snapshot of the shared analytics table through the Delta Sharing profile until the profile expires or sharing is revoked.

For more information about Delta Sharing Spark reads, see Read Delta Sharing Tables and Apache Spark Connector in the Delta Sharing repository.