Mohammad Mohammad - 1 month ago 13
Scala Question

Using Scala IDE and Apache Spark on Windows

I want to start working on a project that uses Spark with Scala on Windows 7.
I downloaded the Apache Spark pre-build for hadoop 2.4 (download page) and I can run it from command prompt (cmd). I can run all of the codes on the quick start of spark page before self-contains application section.

Then I downloaded Scala IDE 4.0.0 from its download page (Sorry it's not possible to post more than 2 links).

Now I created a new scala project and also import the spark assembly jar file into the project. When I want to run the example in the self-contains application section in quick start of spark page but I got the following errors:

15/03/26 11:59:55 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@myhost:7077/user/Master...

15/03/26 11:59:58 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@myhost:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@myhost:7077

15/03/26 11:59:58 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@myhost:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: myhost

15/03/26 12:00:15 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@myhost:7077/user/Master...

15/03/26 12:00:17 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@myhost:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@myhost:7077

15/03/26 12:00:17 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@myhost:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: myhost

15/03/26 12:00:35 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@myhost:7077/user/Master...

15/03/26 12:00:37 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@myhost:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@myhost:7077

15/03/26 12:00:37 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@myhost:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: myhost

15/03/26 12:00:55 ERROR SparkDeploySchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.

15/03/26 12:00:55 ERROR TaskSchedulerImpl: Exiting due to error from cluster scheduler: All masters are unresponsive! Giving up.

15/03/26 12:00:55 WARN SparkDeploySchedulerBackend: Application ID is not initialized yet.


The only line of code that I add to the example, is .setMaster("spark://myhost:7077") for SparkConf definition. I think I need to configure the Scala IDE to use the pre-build spark on my computer but actually I don't know how and I couldn't find anything by googling.

Could you help me to get Scala IDE works with the spark on windows 7?

Thanks in advance

Answer

I found the answer:

I should correct the master definition in my code as follow:

replace:

.setMaster("spark://myhost:7077")

with:

.setMaster("local[*]") 

Hope that it helps you as well.