sag sag - 4 months ago 56
R Question

Change dataType of spark dataframe column in SparkR

I have a spark dataframe.

|-- col1: string (nullable = true)
|-- Id: string (nullable = true)
|-- col2: string (nullable = true)
|-- col3: string (nullable = true)
|-- col4: string (nullable = true)
|-- date1: string (nullable = true)
|-- col5: string (nullable = true)
|-- date2: string (nullable = true)

I just want to convert the date2 column to date. I used the below code to do that


But the dataframe remains the same. No change in date2 datatype.

How can I change the column date2 to date dataType?

myDF <- withColumn(myDF, "date2", cast(myDF$date2, "date"))

See these pages in the SparkR documentation for the latest version (2.0.1 at the time of writing this):

The cast will not change the existing dataframe, so you need to create a new dataframe or replace the existing dataframe with the newly casted column replacing the old column of the same name.