Edamame Edamame - 27 days ago 7
Python Question

pandas filter date time: TypeError: can't compare offset-naive and offset-aware datetimes

I have a pandas dataframe:

name my_timestamp
------------------------------------------
0 a1 2016-07-28 09:27:07.536963-07:00
1 a2 2016-07-28 09:27:07.536963-07:00
2 a3 2016-08-15 13:05:54.924185-07:00
3 a4 2016-08-30 04:04:18.971667-07:00
4 a5 2016-03-22 14:36:22.999825-07:00
5 a6 2016-08-30 04:04:18.971667-07:00


I am trying to filter some rows in my pandas data frame like below:

import datetime
my_df[my_df.my_timestamp > datetime.datetime(2016, 7, 1)]


But get the following errors:

TypeErrorTraceback (most recent call last)
<ipython-input-21-35be746f191d> in <module>()
1 import datetime
----> 2 my_df[my_df.my_timestamp > datetime.datetime(2016, 7, 1)]

/usr/local/lib/python2.7/dist-packages/pandas/core/ops.pyc in wrapper(self, other, axis)
761 other = np.asarray(other)
762
--> 763 res = na_op(values, other)
764 if isscalar(res):
765 raise TypeError('Could not compare %s type with Series' %

/usr/local/lib/python2.7/dist-packages/pandas/core/ops.pyc in na_op(x, y)
681 result = lib.vec_compare(x, y, op)
682 else:
--> 683 result = lib.scalar_compare(x, y, op)
684 else:
685

pandas/lib.pyx in pandas.lib.scalar_compare (pandas/lib.c:14261)()

TypeError: can't compare offset-naive and offset-aware date times


It seems to be the timezone issue. What would be the best way to ignore time-zone here? Thanks!

Answer

Assuming that all timestamps in the dataframe are in the same timezone:

tz_info = my_df.iloc[0].my_timestamp.tzinfo
my_df[my_df.my_timestamp > datetime.datetime(2016, 7, 1, tzinfo=tz_info)]