Shadow Shadow - 2 months ago 18
Java Question

Apache Spark Maven Dependencies for release and develop an app

I have to follow this tutorial to create a uber jar for my Apache Spark application with maven.

I have set all Spark Dependencies in the pom with

<scope>provided</scope>
.
This works very well, but now when I run the application locally, I get an error for missing Spark dependencies.

At the moment I had to remove
provided
tag from the pom.

How can I make provided spark dependencies only when building app for release?

I use Intellij as IDE for developing applications.

Answer

You can create seperate Maven profiles.

Best option is to have dependencyManagment section in POM where you'll specify versions, then in profiles you will have only groupId + artifactId + scope

For example:

<profile>
    <id>dev</id>
    <activation>
        <activeByDefault>false</activeByDefault>
    </activation>
    <dependencies>
    <!-- Here Spark deps without provided -->
        <dependency>
            <groupId>...</groupId>
            <artifactId>...</artifactId>
            <scope>compile</scope>
        </dependency>
    </dependencies>
</profile>
<profile>
    <id>prod</id>
    <dependencies>
    <!-- Here Spark deps with provided -->
        <dependency>
            <groupId>...</groupId>
            <artifactId>...</artifactId>
            <scope>provided</scope>
        </dependency>
    </dependencies>
</profile>