gsamaras - 1 year ago 57

Python Question

Based on Unbalanced factor of KMeans?, I am trying to compute the Unbalanced Factor, but I fail.

Every element of the RDD

`r2_10`

`In [1]: r2_10.collect()`

Out[1]:

[(0, ('438728517', '28138008')),

(13824, ('4647699097', '6553505321')),

(9216, ('2575712582', '1776542427')),

(1, ('8133836578', '4073591194')),

(9217, ('3112663913', '59443972', '8715330944', '56063461')),

(4609, ('6812455719',)),

(13825, ('5245073744', '3361024394')),

(4610, ('324470279',)),

(2, ('2412402108',)),

(3, ('4766885931', '3800674818', '4673186647', '350804823', '73118846'))]

In [2]: pdd = r2_10.map(lambda x: (x[0], 1)).reduceByKey(lambda a, b: a + b)

In [3]: pdd.collect()

Out[3]:

[(13824, 1),

(9216, 1),

(0, 1),

(13825, 1),

(1, 1),

(4609, 1),

(9217, 1),

(2, 1),

(4610, 1),

(3, 1)]

In [4]: n = pdd.count()

In [5]: n

Out[5]: 10

In [6]: total = pdd.map(lambda x: x[1]).sum()

In [7]: total

Out[7]: 10

and

`total`

What am I missing here?

Answer Source

The problem is because you missed to count the number of points grouped in each cluster, thus you have to change how `pdd`

was created.

```
pdd = r2_10.map(lambda x: (x[0], len(x[1]))).reduceByKey(lambda a, b: a + b)
```

However, You could obtain the same result in a single pass (without computing `pdd`

), by mapping the values of the `RDD`

and then reducing by using `sum`

.

```
total = r2_10.map(lambda x: len(x[1])).sum()
```