doctorer doctorer - 1 month ago 22
Python Question

numpy datetime64 in pandas dataframe

I have a

pandas DataFrame
consisting of 3 columns, with the first column an hourly timeseries of dtype
np.datetime64


import numpy as np
import pandas as pd
timeseries = np.arange(np.datetime64('20/11/2009'), np.datetime64('19/11/2010), np.timedelta64(1, 'h'))
insolation =pd.DataFrame(columns=['datetime','GHI','DNI'])
insolation['datetime'] = timeseries


If I
print(insolation)
I get this output:

datetime GHI DNI
0 2009-11-20 00:00:00 NaN NaN
1 2009-11-20 01:00:00 NaN NaN
2 2009-11-20 02:00:00 NaN NaN
... ... ... ...
8734 2010-11-18 22:00:00 NaN NaN
8735 2010-11-18 23:00:00 NaN NaN


So far, so good. But if I use this datetime64 object:

datetime=np.datetime64('2009-11-20 00:00:00')


(which does seem to exist in the DataFrame) to reference the DataFrame:

insolation[datetime][GHI] =1.0


I get an error:

KeyError: numpy.datetime64('2009-11-20T00:00:00')


Full error message is below:

Traceback (most recent call last):
File "C:\Users\z5044992\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\indexes\base.py", line 1945, in get_loc
return self._engine.get_loc(key)
File "pandas\index.pyx", line 137, in pandas.index.IndexEngine.get_loc (pandas\index.c:4154)
File "pandas\index.pyx", line 159, in pandas.index.IndexEngine.get_loc (pandas\index.c:4018)
File "pandas\hashtable.pyx", line 675, in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:12368)
File "pandas\hashtable.pyx", line 683, in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:12322)
KeyError: numpy.datetime64('2009-11-20T00:00:00')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/PYTHONprojects/PY_EN_Models/Scripts/create_weatherfile.py", line 109, in <module>
main()
File "C:/PYTHONprojects/PY_EN_Models/Scripts/create_weatherfile.py", line 87, in main
insolation[datetime][GHI] = irradience_array[gridrow,gridcol]
File "C:\Users\z5044992\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\core\frame.py", line 1997, in __getitem__
return self._getitem_column(key)
File "C:\Users\z5044992\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\core\frame.py", line 2004, in _getitem_column
return self._get_item_cache(key)
File "C:\Users\z5044992\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\core\generic.py", line 1350, in _get_item_cache
values = self._data.get(item)
File "C:\Users\z5044992\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\core\internals.py", line 3290, in get
loc = self.items.get_loc(item)
File "C:\Users\z5044992\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\indexes\base.py", line 1947, in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File "pandas\index.pyx", line 137, in pandas.index.IndexEngine.get_loc (pandas\index.c:4154)
File "pandas\index.pyx", line 159, in pandas.index.IndexEngine.get_loc (pandas\index.c:4018)
File "pandas\hashtable.pyx", line 675, in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:12368)
File "pandas\hashtable.pyx", line 683, in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:12322)
KeyError: numpy.datetime64('2009-11-20T00:00:00')

Process finished with exit code 1

Answer

use loc

insolation.loc[insolation.datetime.eq(datetime), 'GHI'] = 1.0

what went wrong?

insolation[datetime]

Tries to get the column defined by the variable datetime.
Tip: don't use variable names that are the same as your column names.