Nishant - 1 year ago 97
R Question

How to statistically identify outliers using Kmeans clustering

I have following data:

``````head(df.num1)
## 1                    723      178                       0
## 2                    302      169                     563
## 3                    602      148                       0
## 4                    813      164                   22000
## 5                    388      100                     131
## 6                    462      132                     475
## 1                    855                   1000 760505847          886204
## 2                   1000                  40000 309404152          471220
## 3                    161                  11000 200074175          275868
## 4                  23000                  27000 448130642         1144337
## 5                    365                    131  46975183               8
## 6                    530                    640  73058679          212204
## 1                      4834                    0                 3054
## 2                     48350                    0                 1238
## 3                     11700                    1                  994
## 4                    106759                    0                 2701
## 5                       143                    0                  450
## 6                      1873                    1                  738
##    budget title_year actor_2_facebook_likes imdb_score aspect_ratio
## 1 2.4e+08       2009                    936        7.9          1.8
## 2 3.0e+08       2007                   5000        7.1          2.4
## 3 2.4e+08       2015                    393        6.8          2.4
## 4 2.5e+08       2012                  23000        8.5          2.4
## 5 1.0e+07       2015                     12        7.1          2.4
## 6 2.6e+08       2012                    632        6.6          2.4
## 1                33000
## 2                    0
## 3                85000
## 4               164000
## 5                    0
## 6                24000
``````

Then I run kmeans as below:

``````set.seed(111)
km_out <- kmeans(df.num1,centers=3) #perform kmeans cluster with k=3
``````

we now calculate the distance between the objects and cluster centers to determine the outliers and identify say 5 largest distances which are outliers (arbitrary identification).

``````centers <- km_out\$centers[km_out\$cluster, ] # "centers" is a data frame of 3    centers but the length of dataset so we can calculate distance difference easily.
distances <- sqrt(rowSums((df.num1 - centers)^2))
(outliers <- order(distances, decreasing=T)[1:5])# these rows are 5 top outliers
## [1] 3860 3006 2324 2335 3424
``````

lets get the dataframe with distance appended:

``````df.num1\$distance<-distances
df.num1\$cluster<-km_out\$cluster
``````

print the details about the outliers (largest five distance values)

``````(df.num1[outliers,])

## 3860                    202      112                       0
## 3006                     73      134                      45
## 2324                    174      134                    6000
## 2335                    105      103                      78
## 3424                    150      124                      78
## 3860                     38                    717  211667           53508
## 3006                      0                      9  195888            5603
## 2324                    745                    893 2298191          221552
## 2335                    101                    488  410388           13727
## 3424                      4                      6  439162          106160
## 3860                       907                    0                  131
## 3006                        11                    0                   45
## 2324                      2710                    0                  570
## 2335                       991                    1                   79
## 3424                        28                    0                  430
##       budget title_year actor_2_facebook_likes imdb_score aspect_ratio
## 3860 4.2e+09       2005                    126        7.7          2.4
## 3006 2.5e+09       2005                      2        7.1          2.4
## 2324 2.4e+09       1997                    851        8.4          1.8
## 2335 2.1e+09       2004                    336        6.9          1.8
## 3424 1.1e+09       1988                      5        8.1          1.8
## 3860                 4000  4.1e+09       2
## 3006                  607  2.4e+09       2
## 2324                11000  2.3e+09       2
## 2335                  973  2.0e+09       2
## 3424                    0  9.8e+08       2
``````

But these are just data points selected on basis of largest distance from the cluster centers.....

What i wud like is something based on statistical measure such as extreme value based on z score (say > 2sd defined as outlier) instead of just taking few largest distance value obs(rows)........

final output ideas such as:

Or better still something like this:

Would be obliged if some help/pointers to get the kind of result as shown above ......

Regards

Edited to include global outlier

So my understanding is that you want to check each element's distance against distance of its cluster, by using z-score rather than just absolute value comparison.

I reproduced your codes with `iris` dataset. Albeit the code is QUITE messy, you can still see whether each element in each cluster is an outlier or not.

``````df.num1 = iris[,-5]

set.seed(111)

km_out = kmeans(df.num1, 3)
km_out_global = kmeans(df.num1, 1)

cluster_centers = km_out\$centers[km_out\$cluster,]

cluster_distances = sqrt(rowSums(df.num1 - cluster_centers)^2)
global_distances  = sqrt(rowSums(df.num1 - km_out_global\$centers)^2)

df.num1_v1 = data.frame(df.num1, cluster = km_out\$cluster, c_dist = cluster_distances)

CM = ave(df.num1_v1\$c_dist, df.num1_v1\$cluster, FUN = function(x) mean(x, na.rm=TRUE))
CSd = ave(df.num1_v1\$c_dist, df.num1_v1\$cluster, FUN = function(x) sd(x, na.rm=TRUE))
GM = mean(df.num1_v1\$c_dist)
GSd = sd(df.num1_v1\$c_dist)

cluster_z_score = (cluster_distances - CM)/CSd
global_z_score  = (global_distances  - GM)/GSd

df.num1_v2 = data.frame(df.num1_v1, CM, CSd, cluster_z_score,
cluster_outlier = ifelse(cluster_z_score > 2 | cluster_z_score < -2, T , F))

df.num1_v3 = data.frame(df.num1,
cluster_outlier = ifelse(cluster_z_score > 2 | cluster_z_score < -2, T , F),
global_outlier  = ifelse(global_z_score > 2 | global_z_score < -2, T , F)
)
#You can modify your threshold at ifelse

table(df.num1_v2\$cluster_outlier)
FALSE  TRUE
141     9

df.num1_v2[11:16,]
Sepal.Length Sepal.Width Petal.Length Petal.Width cluster distances      CM       CSd     z_score Outlier
11          5.4         3.7          1.5         0.2       3     0.658 0.63232 0.4551378  0.05642247   FALSE
12          4.8         3.4          1.6         0.2       3     0.142 0.63232 0.4551378 -1.07730008   FALSE
13          4.8         3.0          1.4         0.1       3     0.842 0.63232 0.4551378  0.46069563   FALSE
14          4.3         3.0          1.1         0.1       3     1.642 0.63232 0.4551378  2.21840501    TRUE
15          5.8         4.0          1.2         0.2       3     1.058 0.63232 0.4551378  0.93527716   FALSE
16          5.7         4.4          1.5         0.4       3     1.858 0.63232 0.4551378  2.69298655    TRUE
``````

Global Outliers

As I have commented, a data point that was NOT an outlier in cluster may become an outlier in global. This, however, is not an error nor a bug, but just a statistics.

``````table(df.num1_v3\$global_outlier)

FALSE  TRUE
31   119
``````

Note how cluster outliers became global outliers and vice versa.

``````df.num1_v3[11:16,]
Sepal.Length Sepal.Width Petal.Length Petal.Width cluster_outlier global_outlier
11          5.4         3.7          1.5         0.2           FALSE           TRUE
12          4.8         3.4          1.6         0.2           FALSE          FALSE
13          4.8         3.0          1.4         0.1           FALSE           TRUE
14          4.3         3.0          1.1         0.1            TRUE          FALSE
15          5.8         4.0          1.2         0.2           FALSE           TRUE
16          5.7         4.4          1.5         0.4            TRUE           TRUE
``````
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download