Windstorm1981 Windstorm1981 - 5 months ago 21
Python Question

Pandas Replacing a Null Datetime Value in one df with a Timestamp Value from Another

Bug? If so, suggestions on a work around?

I have the following:

df1['DecisionDate'].head()
Out[238]:
ID
RED 2017-02-13 00:00:00
GREEN 2016-07-29 00:00:00
ORANGE 2017-01-26 00:00:00
PURPLE 2016-10-31 00:00:00
YELLOW NaT
Name: DecisionDate, dtype: datetime64[ns]


And this:

df2['DecisionDate']
Out[239]:
YELLOW 2014-04-05 00:00:00
Name: DecisionDate, dtype: datetime64[ns]


Now if I try to do this:

for ID in df2.index:

df1.ix[ID,'DecisionDate'] = df2.ix[ID,'DecisionDate']


I get this

TypeError: long() argument must be a string or a number, not 'Timestamp'


No matter what I try I cannot seem to replace the
pd.NaT
with a timestamp value.

All values in both dataframe are timestamps. df2 observations are a subset of df1 observations so all values in df2.index are in df1.index.

Am I overlooking something straightforwards? Or is this a bug?

Thanks in advance for your help.

EDIT

Here is the full traceback. I'm not good at reading this but maybe it will help in diagnosing:

File "C:\Users\Anaconda2\lib\site-packages\pandas\core\indexing.py", line 141, in __setitem__
def _slice(self, obj, axis=0, kind=None):

File "C:\Users\Anaconda2\lib\site-packages\pandas\core\indexing.py", line 533, in _setitem_with_indexer

File "C:\Users\Anaconda2\lib\site-packages\pandas\core\indexing.py", line 473, in setter
value = getattr(value, 'values', value).ravel()

File "C:\Users\Anaconda2\lib\site-packages\pandas\core\internals.py", line 3168, in setitem

File "C:\Users\Anaconda2\lib\site-packages\pandas\core\internals.py", line 3056, in apply
align_copy = False

File "C:\Users\Anaconda2\lib\site-packages\pandas\core\internals.py", line 668, in setitem
def _replace_single(self, *args, **kwargs):

File "C:\Users\Anaconda2\lib\site-packages\pandas\core\internals.py", line 2265, in _try_coerce_args
Parameters

TypeError: long() argument must be a string or a number, not 'Timestamp'

Answer Source

This appears to be a bug in Spyder.

for ID in df2.index:

        df1.ix[ID,'DecisionDate'] = df2.ix[ID,'DecisionDate']

works.

What I had after the line and commented out was the following:

for ID in df2.index:

        df1.ix[ID,'DecisionDate'] = df2.ix[ID,'DecisionDate']# some comment

Spyder didn't like the comment hash being right next to the active code. Instead of taking a single value, df2.ix[ID,'DecisionDate']# some comment was transformed into a tuple (myvalue,) and I was getting a type error. As soon as I removed the comment from the line it worked as it should.