I am trying to merge two columns into a third column based on the NaN values
df['code2'] = np.where(df['code']==np.nan, df['code'], df['code1'])
I think you need
isnull for comparing
df['code2'] = np.where(df['code'].isnull(), df['code'], df['code1'])
One has to be mindful that in python (and numpy), the nan's don’t compare equal, but None's do. Note that Pandas/numpy uses the fact that np.nan != np.nan, and treats None like np.nan.
In : None == None Out: True In : np.nan == np.nan Out: False
So as compared to above, a scalar equality comparison versus a None/np.nan doesn’t provide useful information.
In : df2['one'] == np.nan Out: a False b False c False d False e False f False g False h False Name: one, dtype: bool