Spark (Scala)

Last updated Oct 1, 2026 Below is an example of how you can build an Apache Spark ↗︎ application (with Scala) which connects to Basin Catalog. This application is built to run locally, but it can be adapted to run on a cluster.

Prerequisites

Example usage

To start, create a new empty project directory somewhere on your machine.

Inside that directory, create the following file at src/main/scala/com/example/R2DataCatalogDemo.scala. This will serve as the main entry point for your Spark application.

package com.example

import org.apache.spark.sql.SparkSession

object R2DataCatalogDemo {
    def main(args: Array[String]): Unit = {

        val uri = sys.env("CATALOG_URI")
        val warehouse = sys.env("WAREHOUSE")
        val token = sys.env("TOKEN")

        val spark = SparkSession.builder()
            .appName("My Basin Catalog Demo")
            .master("local[*]")
            .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
            .config("spark.sql.catalog.mydemo", "org.apache.iceberg.spark.SparkCatalog")
            .config("spark.sql.catalog.mydemo.type", "rest")
            .config("spark.sql.catalog.mydemo.uri", uri)
            .config("spark.sql.catalog.mydemo.warehouse", warehouse)
            .config("spark.sql.catalog.mydemo.token", token)
            .getOrCreate()

        import spark.implicits._

        val data = Seq(
            (1, "Alice", 25),
            (2, "Bob", 30),
            (3, "Charlie", 35),
            (4, "Diana", 40)
        ).toDF("id", "name", "age")

        spark.sql("USE mydemo")

        spark.sql("CREATE NAMESPACE IF NOT EXISTS demoNamespace")

        data.writeTo("demoNamespace.demotable").createOrReplace()

        val readResult = spark.sql("SELECT * FROM demoNamespace.demotable WHERE age > 30")
        println("Records with age > 30:")
        readResult.show()
    }
}

For building this application and managing dependencies, we will use sbt (“simple build tool”) ↗︎. The following is an example build.sbt file to place at the root of your project. It is configured to produce a "fat JAR", bundling all required dependencies.

name := "R2DataCatalogDemo"

version := "1.0"

val sparkVersion = "3.5.3"
val icebergVersion = "1.8.1"

// You need to use binaries of Spark compiled with either 2.12 or 2.13; and 2.12 is more common.
// If you download Spark 3.5.3 with sdkman, then it comes with 2.12.18
scalaVersion := "2.12.18"

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % sparkVersion,
    "org.apache.spark" %% "spark-sql" % sparkVersion,
    "org.apache.iceberg" % "iceberg-core" % icebergVersion,
    "org.apache.iceberg" % "iceberg-spark-runtime-3.5_2.12" % icebergVersion,
    "org.apache.iceberg" % "iceberg-aws-bundle" % icebergVersion,
)

// build a fat JAR with all dependencies
assembly / assemblyMergeStrategy := {
    case PathList("META-INF", "services", xs @ _*) => MergeStrategy.concat
    case PathList("META-INF", xs @ _*) => MergeStrategy.discard
    case "reference.conf" => MergeStrategy.concat
    case "application.conf" => MergeStrategy.concat
    case x if x.endsWith(".properties") => MergeStrategy.first
    case x => MergeStrategy.first
}

// For Java  17 Compatibility
Compile / javacOptions ++= Seq("--release", "17")

To enable the sbt-assembly plugin ↗︎ (used to build fat JARs), add the following to a new file at project/assembly.sbt:

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "1.2.0")

Make sure Java, Spark, and sbt are installed and available in your shell. If you are using SDKMAN, you can install them as shown below:

sdk install java 17.0.14-amzn
sdk install spark 3.5.3
sdk install sbt 1.10.11

With everything installed, you can now build the project using sbt. This will generate a single bundled JAR file.

sbt clean assembly

After building, the output JAR should be located at target/scala-2.12/R2DataCatalogDemo-assembly-1.0.jar.

To run the application, you will use spark-submit. Below is an example shell script (submit.sh) that includes the necessary Java compatibility flags for Spark on Java 17:

# We need to set these "--add-opens" so that Spark can run on Java 17 (it needs access to
# parts of the JVM which have been modularized and made internal).
JAVA_17_COMPATIBILITY="--add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED"

spark-submit \
--conf "spark.driver.extraJavaOptions=$JAVA_17_COMPATIBILITY" \
--conf "spark.executor.extraJavaOptions=$JAVA_17_COMPATIBILITY" \
--class com.example.R2DataCatalogDemo target/scala-2.12/R2DataCatalogDemo-assembly-1.0.jar

Before running it, make sure the script is executable:

chmod +x submit.sh

At this point, your project directory should be structured like this:

  • Makefile
  • README.md
  • build.sbt
  • project
  • assembly.sbt
  • build.properties
  • project
  • spark-submit.sh
  • src
  • main
    • scala
    • com
      • example
      • R2DataCatalogDemo.scala

Before submitting the job, make sure you have the required environment variable set for your catalog URI, warehouse, and Cloudflare API token.

export CATALOG_URI=
export WAREHOUSE=
export TOKEN=

You are now ready to run the job:

./submit.sh

Was this helpful?

YesNo

On this page

Spark (Scala) - 图1Docs

{"@context":"https://schema.org","@type":"TechArticle","@id":"https://developers.cloudflare.com/basin-catalog/config-examples/spark-scala/#page","headline":"Spark (Scala)","description":"Build a Scala Spark application that connects to Basin Catalog for Iceberg table operations.","url":"https://developers.cloudflare.com/basin-catalog/config-examples/spark-scala/","inLanguage":"en","image":"https://developers.cloudflare.com/basin-catalog/config-examples/spark-scala/og.png?v=1a32dbc4664bde10","dateModified":"2026-10-01","publisher":{"@type":"Organization","name":"Cloudflare","description":"One platform for your apps, agents, and workforce. Build, secure, and scale without managing infrastructure","url":"https://www.cloudflare.com/","sameAs":["https://github.com/cloudflare","https://www.linkedin.com/company/cloudflare","https://x.com/cloudflare"],"logo":{"@type":"ImageObject","url":"https://developers.cloudflare.com/logo.svg"},"address":{"@type":"PostalAddress","streetAddress":"101 Townsend St","addressLocality":"San Francisco","addressRegion":"CA","postalCode":"94107","addressCountry":"US"},"contactPoint":[{"@type":"ContactPoint","contactType":"Customer Support","url":"https://support.cloudflare.com/","availableLanguage":["English"]},{"@type":"ContactPoint","contactType":"Sales","url":"https://www.cloudflare.com/contact/","availableLanguage":["English"]}]},"isPartOf":{"@type":"WebSite","@id":"https://developers.cloudflare.com/#website","name":"Cloudflare Docs","url":"https://developers.cloudflare.com/"}}

description: Connect StarRocks to Basin Catalog to query and modify Iceberg tables. title: StarRocks image: https://developers.cloudflare.com/basin-catalog/config-examples/starrocks/og.png?v=242389bf938f9820

Skip to content

Documentation Index
Fetch the complete documentation index at: https://developers.cloudflare.com/basin-catalog/llms.txt
Use this file to discover all available pages before exploring further.