According to Spark source code comments.
/** Distribute a local Scala collection to form an RDD.
* @note Parallelize acts lazily. If `seq` is a mutable collection and is altered after the call
* to parallelize and before the first action on the RDD, the resultant RDD will reflect the
* modified collection. Pass a copy of the argument to avoid this.
* @note avoid using `parallelize(Seq())` to create an empty `RDD`. Consider `emptyRDD` for an
* RDD with no partitions, or `parallelize(Seq[T]())` for an RDD of `T` with empty partitions.
scala> var c = List("a0", "b0", "c0", "d0", "e0", "f0", "g0")
c: List[String] = List(a0, b0, c0, d0, e0, f0, g0)
scala> var crdd = sc.parallelize(c)
crdd: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD at parallelize at <console>:26
scala> c = List("x1", "y1")
c: List[String] = List(x1, y1)
[Stage 0:> (0 + 0) / 8]d0
You didn't modify
c at all. You re-assigned it to a new List.
Besides that point,
seqis a mutable collection
List is not a mutable collection
and is altered after the call to parallelize and before the first action on the RDD
Well, see, you didn't really alter the list.
Here's a proper example of the documented behavior.
scala> val c = scala.collection.mutable.ListBuffer(1, 2, 3) c: scala.collection.mutable.ListBuffer[Int] = ListBuffer(1, 2, 3) scala> val cRDD = sc.parallelize(c) cRDD: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD at parallelize at <console>:29 scala> c.append(4) scala> c res7: scala.collection.mutable.ListBuffer[Int] = ListBuffer(1, 2, 3, 4) scala> cRDD.collect() res8: Array[Int] = Array(1, 2, 3, 4)