Juan Carlos Juan Carlos - 2 months ago 20
Python Question

Sort rows of a dataframe in descending order of NaN counts

I'm trying to sort the following Pandas DataFrame:

RHS age height shoe_size weight
0 weight NaN 0.0 0.0 1.0
1 shoe_size NaN 0.0 1.0 NaN
2 shoe_size 3.0 0.0 0.0 NaN
3 weight 3.0 0.0 0.0 1.0
4 age 3.0 0.0 0.0 1.0


in such a way that the rows with a greater number of NaNs columns are positioned first.
More precisely, in the above df, the row with index 1 (2 Nans) should come before ther row with index 0 (1 NaN).

What I do now is:

df.sort_values(by=['age', 'height', 'shoe_size', 'weight'], na_position="first")

Answer Source

Using df.sort_values and loc based accessing.

df = df.iloc[df.isnull().sum(1).sort_values(ascending=0).index]
print(df)

         RHS  age  height  shoe_size  weight
1  shoe_size  NaN     0.0        1.0     NaN
2  shoe_size  3.0     0.0        0.0     NaN
0     weight  NaN     0.0        0.0     1.0
4        age  3.0     0.0        0.0     1.0
3     weight  3.0     0.0        0.0     1.0

df.isnull().sum(1) counts the NaNs and the rows are accessed based on this sorted count.


@ayhan offered a nice little improvement to the solution above, involving pd.Series.argsort:

df = df.iloc[df.isnull().sum(axis=1).mul(-1).argsort()]
print(df)

         RHS  age  height  shoe_size  weight 
1  shoe_size  NaN     0.0        1.0     NaN           
0     weight  NaN     0.0        0.0     1.0           
2  shoe_size  3.0     0.0        0.0     NaN           
3     weight  3.0     0.0        0.0     1.0           
4        age  3.0     0.0        0.0     1.0