javadba javadba - 1 year ago 139
Scala Question

How to avoid requiring any native libraries for compression in Spark

We are doing a POC on a variety of server machines/architectures. We do not have the ability to rebuild native compression libraries for all of them.

Which codec is software only? The default snappy is giving the following error:

Caused by: java.lang.IllegalArgumentException

Answer Source

Is software only. That is working.

The way to configure: in $SPARK_HOME/conf/spark-defaults.conf: lz4
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download