DSpark DSpark - 1 year ago 95
Scala Question

Custom kryoSerialization flow not working in scala spark

I am new to scala.
I am trying to implement Custom kryo serialization.

In which I have two classes and one object :


package org.agg
object Operation {
def main(args: Array[String]) {
var SparkConf = new SparkConf()
.set("spark.io.compression.codec", "org.apache.spark.io.SnappyCompressionCodec")
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.set("spark.kryo.registrationRequired", "true")
.set("spark.kryo.registrator", "org.agg.KryoClass")

var sc = new SparkContext(SparkConf)
var sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)


package org.agg
class KryoClass extends KryoRegistrator {
def registerClasses(kryo: Kryo) {


package org.agg
class KryoSerializeCode {

I am considering that In Operation Class If I am writing set("spark.kryo.registrator", "org.agg.KryoClass")
So this should call KryoClass and It will print println("**********KryoClass***********") statement in log file.

Command to execute Operation object is below:

spark-submit --class org.agg.Operation --master yarn --deploy-mode
cluster --num-executors 40 --executor-cores 1 --executor-memory 3400m
--files /home/hive-site.xml --jars /usr/iop/,/usr/iop/,/usr/iop/

But After executing this it is printing print statement in Operation class only not in KryoClass or KryoSerializeCode Class.

Do anyone having idea why it is not calling print statement inside KryoClass or KryoSerializeCode Class.

Answer Source

Spark has the habit of doing things lazily - delaying unnecessary operations until they are actually needed. Seems like the creation of the Kryo serializer falls into that category - since you didn't actually do anything with your newly created SparkContext, Spark didn't bother creating the serializer.

If you add any Spark action to your Operation.main, e.g.:


You'll see the printouts you're expecting.

NOTE: you might have to register a few more classes, or disable registrationRequired

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download