user2628641 user2628641 - 9 months ago 138
Scala Question

Spark convert list to dataframe error: Schema for type Any is not supported

I ma trying to make create dataframe by doing the following:

val df = Seq(
("tom", 25),
("andy", null)
).toDF("user", "rating")

but I get an error of "Schema for type Any is not supported".
I think the rating column is causing the problem.

I want the rating column to be

integer (nullable=true)

but i am not sure how to achieve that

Answer Source

You can use Option:

  ("tom", Some(25)),
  ("andy", None)
).toDF("user", "rating")

You could also be more specific about the types and use java.lang.Integer:

Seq[(String, java.lang.Integer)](
  ("tom", 25),
  ("andy", null)
).toDF("user", "rating")

but the first method should be preferred.