I have been trying to install spark using the tutorial and everytime I run the command sbt/sbt assembly, I get the error "Error: Invalid or corrupt jarfile sbt/sbt-launch-0.13.5.jar"
I have tried everything: seperately adding the sbt file to the sbt folder in the spark folder, installing sbt individually, checking the download and reinstalling it over again, but in vain. Any advice about what I am doing wrong? Thanks.
Ok, After playing around for a while I finally got it and hopefully this will work for you aswell. That tutorial builds spark, where they do provide prebuilt binaries. I'm using Spark 1.2.0 just as a note (1.4.1 wouldn't work for me)
This is on Ubuntu 15.04 but should work on 14.04 the same
1) Remove the following lines from your bashrc
export SCALA_HOME=/usr/local/src/scala/scala-2.10.4 export PATH=$SCALA_HOME/bin:$PATH
2) Remove and reinstall scala
sudo rm -rf /usr/local/src/scala # The following line is only needed if you installed scala another way, if so remove the # # sudo apt-get remove scala-library scala wget http://www.scala-lang.org/files/archive/scala-2.11.7.deb sudo dpkg -i scala-2.11.7.deb sudo apt-get update sudo apt-get install scala
3) Download PreBuilt Spark and extract
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.2.0-bin-hadoop2.4.tgz tar -xzvf spark-1.2.0-bin-hadoop2.4.tgz
4) Run spark-shell
cd spark-1.2.0-bin-hadoop2.4/ ./bin/spark-shell