vivman vivman - 25 days ago 21
Scala Question

Save values in spark

i am trying to read and write data from the my local folder, but my data's not identical.

val data =sc.textFile("/user/cts367689/datagen.txt")

val a=data.map(line=>(line.split(",")(0).toInt+line.split(",")(4).toInt,line.split(",")(3),line.split(",")(2)))

a.saveAsTextFile("/user/cts367689/sparkoutput")


Output:

(526,female,avil)
(635,male,avil)
(983,male,paracetamol)
(342,female,paracetamol)
(158,female,avil)


how can i save output as below,need to remove brackets.

Expected Result:

526,female,avil
635,male,avil
983,male,paracetamol
342,female,paracetamol
158,female,avil

Answer
val a = data.map (
  line => 
    (line.split(",")(0).toInt + line.split(",")(4).toInt) + "," +
    line.split(",")(3) + "," + 
    line.split(",")(2)
)

Try doing this instead of returning it in (). That makes a tuple.

Comments