Milad Khajavi Milad Khajavi - 1 year ago 199
Java Question

How to convert DataFrame to Dataset in Apache Spark in Java?

I can convert DataFrame to Dataset in Scala very easy:

case class Person(name:String, age:Long)
val df ="/tmp/persons.json")
val ds =[Person]

but in Java version I don't know how to convert Dataframe to Dataset? Any Idea?

my effort is:

DataFrame df =;
Encoder<Person> encoder = new Encoder<>();
Dataset<Person> ds = new Dataset<Person>(ctx,df.logicalPlan(),encoder);

but the compiler say:

Error:(23, 27) java: org.apache.spark.sql.Encoder is abstract; cannot be instantiated


solution based on @Leet-Falcon answers:

DataFrame df =;
Encoder<Person> encoder = Encoders.bean(Person.class);
Dataset<Person> ds = new Dataset<Person>(ctx, df.logicalPlan(), encoder);

Answer Source

Official Spark docs suggest in Dataset API the following:

Java Encoders are specified by calling static methods on Encoders.

List<String> data = Arrays.asList("abc", "abc", "xyz");
Dataset<String> ds = context.createDataset(data, Encoders.STRING());

Encoders can be composed into tuples:

Encoder<Tuple2<Integer, String>> encoder2 = Encoders.tuple(Encoders.INT(), Encoders.STRING());
List<Tuple2<Integer, String>> data2 = Arrays.asList(new scala.Tuple2(1, "a");
Dataset<Tuple2<Integer, String>> ds2 = context.createDataset(data2, encoder2);

Or constructed from Java Beans by Encoders#bean:

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download