Addison Addison - 1 year ago 158
Java Question

'Symbol lookup error' with netlib-java

Background & Problem

I am having a bit of trouble running the examples in Spark's MLLib on a machine running Fedora 23. I have built Spark 1.6.2 with the following options per Spark documentation:

build/mvn -Pnetlib-lgpl -Pyarn -Phadoop-2.4 \
-Dhadoop.version=2.4.0 -DskipTests clean package

and upon running the binary classification example:

bin/spark-submit --class org.apache.spark.examples.mllib.BinaryClassification \
examples/target/scala-*/spark-examples-*.jar \
--algorithm LR --regType L2 --regParam 1.0 \

I receive the following error:

/usr/lib/jvm/java-1.8.0-openjdk- symbol lookup error: /tmp/ undefined symbol: cblas_dscal

Errors of this form (symbol lookup error with netlib) are not limited to this particular example. On the other hand, the Elastic Net example (
./bin/run-example ml.LinearRegressionWithElasticNetExample
) runs without a problem.

Attempted Solutions

I have tried a number of solutions to no avail. For example, I went through some of the advice here, and while I can successfully import from
, the aforementioned symbol lookup error persists.

I have read through the netlib-java documentation at fommil/netlib-java, and have ensured my system has the
shared object files:

$ ls /usr/lib64 | grep libblas

$ ls /usr/lib64 | grep liblapack

The most promising advice I found was here, which suggests including

JAVA_OPTS="- Dcom.github.fommil.netlib.BLAS=com.github.fommil.netlib.NativeRefBLAS"

in the
script. So, I included appended those options to
in the
script, which also did not resolve the problem.

Finally, a last bit of advice I found online suggested passing the following flags to

sbt -Dcom.github.fommil.netlib.BLAS=com.github.fommil.netlib.F2jBLAS \
-Dcom.github.fommil.netlib.LAPACK=com.github.fommil.netlib.F2jLAPACK \

and again the issue persists. I am limited to two links in my post, but the advice can be found as the of lildata's 'scaladatascience' repo on github.

Has anybody suffered this issue and successfully resolved it? Any and all help or advice is deeply appreciated.

Answer Source

It's been a couple months, but I got back to this problem and was able to get a functioning workaround (posting here in case anybody else has the same issue).

It came down to library precedence; so, by calling:

export LD_PRELOAD=/path/to/

prior to launching Spark, everything works as expected.

I figured out the solution after reading:

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download