toofrellik toofrellik - 1 month ago 19
Scala Question

Scala - Save a data frame to text file without [ ]

When i save a Dataframe to a text file it gets saved with

[]

for example i have a data frame:

9206E599-2E07-439E-9C01-A6714E779D11
D607E87F-FB90-4F2B-8A99-335AE4F5C97F


when i do
myDataFrame.rdd.saveAsTextFile("My Path")
i get

[9206E599-2E07-439E-9C01-A6714E779D11]
[D607E87F-FB90-4F2B-8A99-335AE4F5C97F]


is there a way to remove those brackets.

Answer

Here's a way of doing it.

val df: DataFrame = ???
df.rdd.map(r => r.getString(0)).saveAsTextFile("My Path")

The issue is your RDD contains org.apache.spark.sql.Row objects. When these are 'toStringed' they have those square braces on either side. You want to extract the data out of the Row object into a String.