xuanyue xuanyue - 2 years ago 104
Scala Question

Does Apache spark 2.2 supports user-defined type (UDT)?

From this JIRA ticket Hide UserDefinedType in Spark 2.0
, seems that spark hide the UDT API from version 2.0.

Is there exists an alternative function or API we can use in version 2.2, so that we could define UserDefinedType? I wish to use a custom type in dataframe or structured streaming.

Answer Source

There is no alternative API and UDT remains private (https://issues.apache.org/jira/browse/SPARK-7768).

Generic Encoders (org.apache.spark.sql.Encoders.kryo and org.apache.spark.sql.Encoders.javaSerialization) serve similar purpose in Dataset, but there are not direct replacement:

How to store custom objects in Dataset?

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download