Georg Heiler Georg Heiler - 16 days ago 5
Java Question

sbt assembly switch between provided dependencies

I am developing a spark application. To test it locally I want to run

sbt
run. This requires the dependencies to be available locally. But I also want to use
sbt assembly
to generate a jar which can be used for
spark-submit
that jar should only include some of the dependencies e.g. not include the spark dependencies.

When I mark a dependency in
build.sbt
as
% "provided"
it no longer is available via
sbt run

Answer

If the case to exclude jars in assembly use in main built.sbt

excludedJars in assembly := {
  val cp = (fullClasspath in assembly).value
  cp filter {f =>
     f.data.getName.contains("spark"),
     f.data.getName.startsWith("jar_name")
  }
}