sag sag - 27 days ago 29
R Question

Change dataType of spark dataframe column in SparkR

I have a spark dataframe.

root
|-- col1: string (nullable = true)
|-- Id: string (nullable = true)
|-- col2: string (nullable = true)
|-- col3: string (nullable = true)
|-- col4: string (nullable = true)
|-- date1: string (nullable = true)
|-- col5: string (nullable = true)
|-- date2: string (nullable = true)


I just want to convert the date2 column to date. I used the below code to do that

to_date(myDF$date2)


But the dataframe remains the same. No change in date2 datatype.

How can I change the column date2 to date dataType?

Answer
myDF <- withColumn(myDF, "date2", cast(myDF$date2, "date"))

See these pages in the SparkR documentation for the latest version (2.0.1 at the time of writing this):

The cast will not change the existing dataframe, so you need to create a new dataframe or replace the existing dataframe with the newly casted column replacing the old column of the same name.