Darshan Darshan - 2 months ago 14
Scala Question

Appending Data to List or any other collection Dynamically in scala

I am new to scala.
Can we Add/Append data into List or any other Collection Dynamically in scala.

I mean can we add data in List or any collection using foreach (or any other loop).

I am trying to do something like below:

var propertyData = sc.textFile("hdfs://ip:8050/property.conf")

var propertyList = new ListBuffer[(String,String)]()

propertyData.foreach { line =>
var c = line.split("=")
propertyList.append((c(0), c(1)))
}


And suppose property.conf file contains:


"spark.shuffle.memoryFraction"="0.5"

"spark.yarn.executor.memoryOverhead"="712"


This is compiled fine But value is not added in ListBuffer.

Answer

I tried it using Darshan's code from his (updated) question:

val propertyData = List(""""spark.shuffle.memoryFraction"="0.5"""", """"spark.yarn.executor.memoryOverhead"="712" """)
val propertyList = new ListBuffer[(String,String)]()
propertyData.foreach { line =>
  val c = line.split("=")
  propertyList.append((c(0), c(1)))
}
println(propertyList)

It works as expected: it prints to the console:

ListBuffer(("spark.shuffle.memoryFraction","0.5"), ("spark.yarn.executor.memoryOverhead","712" ))

I didn't do it in a Spark Context, although I will try that in a few minutes. So, I provided the data in a list of Strings (shouldn't make a difference). I also changed the "var" keywords to "val" since none of them needs to be a mutable variable, but of course that makes no difference either. The code works whether they are val or var.

See my comment below. But here is idiomatic Spark/Scala code which does behave exactly as you would expect:

object ListTest extends App {
  val conf = new SparkConf().setAppName("listtest")
  val sc = new SparkContext(conf)
  val propertyData = sc.textFile("listproperty.conf")
  val propertyList = propertyData map { line =>
    val xs: Array[String] = line.split("""\=""")
    (xs(0),xs(1))
  }
  propertyList foreach ( println(_))
}