Edamame Edamame - 2 months ago 30
Python Question

python pandas: filter out records with null or empty string for a given field

I am trying to filter out records whose field_A is null or empty string in the data frame like below:

my_df[my_df.editions is not None]
my_df.shape


This gives me error:

---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-40-e1969e0af259> in <module>()
1 my_df['editions'] = my['editions'].astype(str)
----> 2 my_df = my_df[my_df.editions is not None]
3 my_df.shape

/home/edamame/anaconda2/lib/python2.7/site-packages/pandas/core/frame.pyc in __getitem__(self, key)
1995 return self._getitem_multilevel(key)
1996 else:
-> 1997 return self._getitem_column(key)
1998
1999 def _getitem_column(self, key):

/home/edamame/anaconda2/lib/python2.7/site-packages/pandas/core/frame.pyc in _getitem_column(self, key)
2002 # get column
2003 if self.columns.is_unique:
-> 2004 return self._get_item_cache(key)
2005
2006 # duplicate columns & possible reduce dimensionality

/home/edamame/anaconda2/lib/python2.7/site-packages/pandas/core/generic.pyc in _get_item_cache(self, item)
1348 res = cache.get(item)
1349 if res is None:
-> 1350 values = self._data.get(item)
1351 res = self._box_item_values(item, values)
1352 cache[item] = res

/home/edamame/anaconda2/lib/python2.7/site-packages/pandas/core/internals.pyc in get(self, item, fastpath)
3288
3289 if not isnull(item):
-> 3290 loc = self.items.get_loc(item)
3291 else:
3292 indexer = np.arange(len(self.items))[isnull(self.items)]

/home/edamame/anaconda2/lib/python2.7/site-packages/pandas/indexes/base.pyc in get_loc(self, key, method, tolerance)
1945 return self._engine.get_loc(key)
1946 except KeyError:
-> 1947 return self._engine.get_loc(self._maybe_cast_indexer(key))
1948
1949 indexer = self.get_indexer([key], method=method, tolerance=tolerance)

pandas/index.pyx in pandas.index.IndexEngine.get_loc (pandas/index.c:4154)()

pandas/index.pyx in pandas.index.IndexEngine.get_loc (pandas/index.c:4018)()

pandas/hashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:12368)()

pandas/hashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:12322)()

KeyError: True


or

my_df[my_df.editions != None]
my_df.shape


This one gave no error but didn't filter out any None values.

I also tried:

my_df = my_df[my_df.editions.notnull()]


This one doesn't give error but doesn't filter out any None values either.

Could anyone please advise how to solve this problem? Thanks!

Answer

Can you create a new dataframe from the filtering?

Dataframe before:

a     b
1     9
2    10
3    11
4    12
5    13
6    14
7    15
8  null

Example:

import pandas

my_df = pandas.DataFrame({"a":[1,2,3,4,5,6,7,8],"b":[9,10,11,12,13,14,15,"null"]})

my_df2= my_df[(my_df['b']!="null")]
print(my_df2)

dataframe after:

a   b
1   9
2  10
3  11
4  12
5  13
6  14
7  15

What it is doing is looking for "null" and excluding it. You could do the same thing with empty strings.