what what - 2 months ago 13
R Question

Find duplicate values in R

I have a table with 21638 unique* rows:

vocabulary <- read.table("http://socserv.socsci.mcmaster.ca/jfox/Books/Applied-Regression-2E/datasets/Vocabulary.txt", header=T)


This table has five columns, the first of which holds the respondent ID numbers. I want to check if any respondents appear twice, or if all respondents are unique.

To count unique IDs I can use

length(unique(vocabulary$id))


and to check if there are any duplicates I might do

length(unique(vocabulary$id)) == nrow(vocabulary)


which returns
TRUE
, if there are no duplicates (which there aren't).

My question:

Is there a direct way to return the values or line numbers of duplicates?

Some further explanation:

There is an interpretation problem with using the function
duplicated()
, because is only returns the duplicates in the strict sense, excluding the "originals". For example,
sum(duplicated(vocabulary$id))
or
dim(vocabulary[duplicated(vocabulary$id),])[1]
might return "5" as the number of duplicate rows. The problem is that if you only know the number of duplicates, you won't know how many rows they duplicate. Does "5" mean that there are five rows with one duplicate each, or that there is one row with five duplicates? And since you won't have the IDs or line numbers of the duplicates, you wouldn't have any means of finding the "originals".




*I know there are no duplicate IDs in this survey, but it is a good example, because using any of the answers given elsewhere to this question, like
duplicated(vocabulary$id)
or
table(vocabulary$id)
will output a haystack to your screen in which you'll be quite unable to find any possible rare duplicate needles.

Answer

You could use table, i.e.

n_occur <- data.frame(table(vocabulary$id))

gives you a data frame with a list of ids and the number of times they occurred.

n_occur[n_occur$Freq > 1,]

tells you which ids occurred more than once.

vocabulary[vocabulary$id %in% n_occur$Var1[n_occur$Freq > 1],]

returns the records with more than one occurrence.

Comments