Joe Joe - 2 months ago 7
Python Question

How to remove rows with duplicates in pandas dataframe?

Having a dataframe which contains duplicate values in two columns (A and B):

A B
1 2
2 3
4 5
7 6
5 8


I want to remove duplicates so that only unique values remain:

A B
1 2
4 5
7 6


This command does not provide what I want:

df.drop_duplicates(subset=['A','B'], keep='first')


Any idea how to do this?

Answer

You can use stack with unstack:

print (df.stack().drop_duplicates().unstack().dropna().astype(int))
   A  B
0  1  2
2  4  5
3  7  6

Solution with boolean indexing:

print (df[~df.stack().duplicated().unstack().any(1)])
   A  B
0  1  2
2  4  5
3  7  6