mongolol mongolol - 3 days ago 6
Scala Question

Writing an RDD to a CSV File

I have a relatively small RDD of the format

RDD[(Int, Double)]
which I was hoping to write to a csv file. Following the logic at Writing a RDD to a csv, I ended up with the below code:

val myRdd.map{case(a, b) =>
var line = a.toString + "," + b.toString
line
}.saveAsTextFile


However, I'm receiving the below error:

Main.scala:111: ambiguous reference to overloaded definition,
[error] both method saveAsTextFile in class RDD of type (path: String, codec: Class[_ <: org.apache.hadoop.io.compress.CompressionCodec])Unit
[error] and method saveAsTextFile in class RDD of type (path: String)Unit
[error] match expected type ?
[error] }.saveAsTextFile
[error] ^


Any suggestions?

Answer

You must provide path:

val myRdd.map{case(a, b) =>
  var line = a.toString + "," + b.toString
  line
}.saveAsTextFile("path");

Both local and HDFS paths are correct. Here is documentation

Comments