I am new to spark but am trying to do some development. I am following "Reducing Build Times" instructions from the spark developer page. After creating the normal assembly I have written some classes that are dependent on one specific jar. I test my package in the spark-shell in which I have been able to include my jar by using defining
Based on the answer in the comments above, it looks like you are trying to add your jar as a dependency to the the mllib project as you do development on mllib itself. You can accomplish this by modifying the pom.xml file in the mllib directory within the Spark distribution.
You can find instructions on how to add a local file as a dependency here - http://blog.valdaris.com/post/custom-jar/. I haven't used this approach myself to including local file as a dependency, but I think it should work.