anthurium anthurium - 1 year ago 54
Scala Question

scala creating key value pairs from textfile with multiple entries for values

How to create key value pairs in the following format?

Sample Input in a


X: a b c

Y: f g

I want the output to be key value pairs and stored in an



val sprk = new SparkContent(conf)
in = sprk.textFile("sample_input.txt")
val tuples = in.maps{s =>
val parts = s.split("\\s+")
(parts(0), parts(1))

Answer Source

First split using : and then using \\s+

val textFile = sc.textFile("hdfs://...")

textFile.flatMap { line => {
      val Array(label, rest) = line split ":"
      val items = rest.trim.split("\\s+") => (label.trim -> item))
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download