duckertito duckertito - 27 days ago 18
Scala Question

How to save RDD data into json files, not folders

I am receiving the streaming data

myDStream
(
DStream[String]
) that I want to save in S3 (basically, for this question, it doesn't matter where exactly do I want to save the outputs, but I am mentioning it just in case).

The following code works well, but it saves folders with the names like
jsonFile-19-45-46.json
, and then inside the folders it saves files
_SUCCESS
and
part-00000
.

Is it possible to save each
RDD[String]
(these are JSON strings) data into the JSON files, not the folders? I thought that
repartition(1)
had to make this trick, but it didn't.

myDStream.foreachRDD { rdd =>
// datetimeString = ....
rdd.repartition(1).saveAsTextFile("s3n://mybucket/keys/jsonFile-"+datetimeString+".json")
}

Answer

AFAIK there is no option to save it as a file. Because it's a distributed processing framework and it's not a good practice write on single file rather than each partition writes it's own files in the specified path.

We can pass only output directory where we wanted to save the data. OutputWriter will create file(s)(depends on partitions) inside specified path with part- file name prefix.