mahboube boroumand mahboube boroumand - 1 year ago 77
Scala Question

How to create key-value pairs from textFile

I want to use a member in a RDD element as a key, how can i do this
this is my data:

2 1
4 1
1 2
6 3
7 3
7 6
6 7
3 7

I want to create key/value pairs such that key is a element, also value is next element;

I wrote this code:

def main(args: Array[String])
val conf = new SparkConf().setAppName("test").setMaster("local[4]")
val sc = new SparkContext(conf)

val lines = sc.textFile("followers.txt")
.flatMap{x => (x.indexOfSlice(x1),x.indexOfSlice(x2))}


but it is not true and it wont determine the index of elements;
every two number is a line

Answer Source

Maybe I'm misunderstanding your question, but if you are simply looking to split your data into key-value pairs, you just need to do this:

val lines = sc.textFile("followers.txt").map(s => {
  val substrings = s.split(" ")
  (substrings(0), substrings(1))

Does this solve your problem?

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download