I have the following code
file = spark.textFile("hdfs://...")
counts = file.flatMap(lambda line: line.split(" ")) \
.map(lambda word: (word, 1)) \
.reduceByKey(lambda a, b: a + b)
map is the easiest, it essentially says do the given operation on every element of the sequence and return the resulting sequence (very similar to foreach).
flatMap is the same thing but instead of returning just one element per element you are allowed to return a sequence (which can be empty). Here's an answer explaining the difference between
reduceByKey takes an aggregate function (meaning it takes two arguments of the same type and returns that type, should also be commutative and associative otherwise you will get inconsistent results) which is used to aggregate every
V for each
K in your sequence of
reduce (lambda a, b: a + b,[1,2,3,4])
This says aggregate the whole list with
+ so it will do
1 + 2 = 3 3 + 3 = 6 6 + 4 = 10 final result is 10
Reduce by key is the same thing except you do a reduce for each unique key.
So to explain it in your example
file = spark.textFile("hdfs://...") // open text file each element of the RDD is one line of the file counts = file.flatMap(lambda line: line.split(" ")) //flatMap is needed here to return every word (separated by a space) in the line as an Array .map(lambda word: (word, 1)) //map each word to a value of 1 so they can be summed .reduceByKey(lambda a, b: a + b) // get an RDD of the count of every unique word by aggregating (adding up) all the 1's you wrote in the last step counts.saveAsTextFile("hdfs://...") //Save the file onto HDFS
So, why count words this way, the reason is that the MapReduce paradigm of programming is highly parallelizable and thus scales to doing this computation on terabytes or even petabytes of data.
I don't use python much tell me if I made a mistake.