marios marios - 1 year ago 179
Scala Question

Spark Kryo: Register a custom serializer

I have a class that implements a custom Kryo serializer by implementing the

methods from
(see example below). How can I register this custom serializer with Spark?

Here is a sudo-code example of what I have:

class A()

CustomASerializer extends com.esotericsoftware.kryo.Serializer[A]{
override def write(kryo: Kryo, output: Output, a: A): Unit = ???
override def read(kryo: Kryo, input: Input, t: Class[A]): A = ???

val Kryo kryo = ...
kryo.register(classOf[A], new CustomASerializer()); // I can register my serializer

Now in Spark:

val sparkConf = new SparkConf()

Unfortunately, Spark doesn't give me the option to register my custom serializer. Any idea if there is a way to do this?

Answer Source

Create your own KryoRegistrator with this custom serializer registered:

package com.acme

class MyRegistrator extends KryoRegistrator {
  override def registerClasses(kryo: Kryo) {
    kryo.register(classOf[A], new CustomASerializer())

Then, set spark.kryo.registrator to your registrator's fully-qualified name, e.g. com.acme.MyRegistrator:

val conf = new SparkConf()
conf.set("spark.kryo.registrator", "com.acme.KryoRegistrator")
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download