mahboube boroumand mahboube boroumand - 1 month ago 7
Scala Question

How to create key-value pairs from textFile

I want to use a member in a RDD element as a key, how can i do this
this is my data:

2 1
4 1
1 2
6 3
7 3
7 6
6 7
3 7


I want to create key/value pairs such that key is a element, also value is next element;

I wrote this code:

def main(args: Array[String])
{
System.setProperty("hadoop.home.dir","C:\\spark-1.5.1-bin-hadoop2.6\\winutil")
val conf = new SparkConf().setAppName("test").setMaster("local[4]")
val sc = new SparkContext(conf)

val lines = sc.textFile("followers.txt")
.flatMap{x => (x.indexOfSlice(x1),x.indexOfSlice(x2))}

}


but it is not true and it wont determine the index of elements;
every two number is a line

Answer

Maybe I'm misunderstanding your question, but if you are simply looking to split your data into key-value pairs, you just need to do this:

val lines = sc.textFile("followers.txt").map(s => {
  val substrings = s.split(" ")
  (substrings(0), substrings(1))
})

Does this solve your problem?