mlee_jordan mlee_jordan -4 years ago 131
Python Question

Cannot use fillna when a condition is introduced

I am very new to python. Trying to do some imputation in my data. however, I could not manage. Here is the simple code:

df['a'] = ""
df.loc[(df['c'] >= 0) & (df['c'] <= 43), 'a'] = 1
df.loc[(df['c'] >= 44) & (df['c'] <= 96), 'a'] = 2
df.loc[(df['c'] >= 97) & (df['c'] <= 151), 'a'] = 3
df.loc[(df['c'] >= 152) & (df['c'] <= 273), 'a'] = 4

print(df[df['a'] == 1]['b'].median())
print(df[df['a'] == 2]['b'].median())
print(df[df['a'] == 3]['b'].median())
print(df[df['a'] == 4]['b'].median())

print(df[df['a'] == 1]['b'].median())

df[df['a'] == 1]['b'].fillna(df[df['a'] == 1]['b'].median(), inplace=True)


When I tried this it threw a warning:

A value is trying to be set on a copy of a slice from a DataFrame


How can I apply fillna properly?

Answer Source

Use loc:

df = pd.DataFrame({'c':[10,50,100,200] * 3,
                   'b':[1,3,8,np.nan,5,8,np.nan,7, np.nan, 4,1,0]})
#print (df)
m1 = (df['c'] >= 0) & (df['c'] <= 43)
m2 = (df['c'] >= 44) & (df['c'] <= 96)
m3 = (df['c'] >= 97) & (df['c'] <= 151)
m4 = (df['c'] >= 152) & (df['c'] <= 273)

df.loc[m1,'b'] = df.loc[m1,'b'].fillna(df.loc[m1,'b'].median())
df.loc[m2,'b'] = df.loc[m2,'b'].fillna(df.loc[m2,'b'].median())
df.loc[m3,'b'] = df.loc[m3,'b'].fillna(df.loc[m3,'b'].median())
df.loc[m4,'b'] = df.loc[m4,'b'].fillna(df.loc[m4,'b'].median())

print (df)
      b    c
0   1.0   10
1   3.0   50
2   8.0  100
3   3.5  200
4   5.0   10
5   8.0   50
6   4.5  100
7   7.0  200
8   3.0   10
9   4.0   50
10  1.0  100
11  0.0  200

But better is use cut for category column and then groupby with custom function with fillna and median:

bins = [0,43,96,151,273]
labels=[1,2, 3, 4]
df['a'] = pd.cut(df['c'], bins=bins, labels=labels)
df['b'] = df.groupby('a')['b'].apply(lambda x: x.fillna(x.median()))
print (df)
      b    c  a
0   1.0   10  1
1   3.0   50  2
2   8.0  100  3
3   3.5  200  4
4   5.0   10  1
5   8.0   50  2
6   4.5  100  3
7   7.0  200  4
8   3.0   10  1
9   4.0   50  2
10  1.0  100  3
11  0.0  200  4
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download