Ramesh Ramesh - 1 month ago 6
Scala Question

Convert Spark Dataframes each row as a String with a delimiter between each column value in scala

I want to convert Spark Dataframe each row as a String with a delimiter between each column value.

For example:
I have a input dataframe 'df' with 3 columns "firstname","lastname","age", with two records which look like below.

Row1: John Abhraham 21
Row2: Steve Austin 22

I want to create a new dataframe with just one column which has data like below.

Row1: John$Abhraham$21
Row2: Steve$Austin$22

Can anyone please help in doing this.

Answer

I don't have a Spark shell handy, but I think this one liner should do it:

def stringifyRows(df: DataFrame, sep: String): DataFrame = df.map(row => row.mkString(sep))

For your example, you would call this as stringifyRows(myDf, "$"). Let me know what the error message is if this doesn't work.