Tangle Tangle - 3 months ago 37
Scala Question

How to append keys to values for {Key,Value} pair RDD and How to convert it to an rdd?

Suppose i am having 2 files in file1,file2 in dataset directory:

val file = sc.wholeTextFiles("file:///root/data/dataset").map((x,y) => y + "," + x)


in the Above code i am trying to get an rdd having values:-> value,key as single value into rdd

suppose filename is file1 and say 2 records:

file1:

1,30,ssr

2,43,svr


And

file2:

1,30,psr

2,43,pvr


The desired rdd output is:

(1,30,ssr,file1),(2,43,svr,file1),(1,30,psr,file2),(2,43,pvr,file2)


Can we achieve this?
if possible Please Help me!

Answer
var files = sc.wholeTextFiles("file:///root/data/dataset")

var yourNeededRdd = files
  .flatMap({
    case (filePath, fileContent) => {
      val fileName = filePath.split('/).last
      fileContent.split("\n").map(line => line + "," + fileName)
    }
  })
Comments