Essi Shams Essi Shams - 22 days ago 8
Python Question

Performing Arithmetic operations on nested dataframe containing a list

I have a dataframe called dailyHistogram defined as follows:


dailyHistogram = pd.DataFrame({'NumVisits':[[0 for x in range(1440)]
for y in range (180)],
'DoW': [0]*ReportingDateRange.size
}
,columns=['NumVisits','DoF'],
index=ReportingDateRange)



Where NumVisits is a two-dimensional array (1440 by 180) and holds a histogram of some activity in 180 days. DoW is simply a column which holds the day of the week.
The index in this dataframe is the dates on which the activities occurred.

My problem is in performing any operations on dailyHistogram["NumVisits"].

Here's what dailyHistogram["NumVisits"] looks like:

> dailyHistogram["NumVisits"]

> Out[193]:
> 2016-01-01 [5, 0, 0, 0, 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
>
> 2016-01-02 [2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
>
> 2016-01-03 [6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
>
> 2016-01-04 [8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
>
> 2016-06-26 [3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
>
> 2016-06-27 [4, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
>
> 2016-06-28 [7, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
>
> 2016-06-29 [7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
>
> 2016-06-30 [4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
>
> Freq: D, Name: NumVisits, dtype: object


I'd like to sum up all vectors in "NumVisits" for a particular day of the week, but no arithmetic operations seem to be possible on dailyHistogram["NumVisits"]

Answer

That is because NumVisits is a list, and to perform arithmetic on the contents of a list, you need to explicitly apply your functions. For example:

df['NumVisits'].apply(sum)

For element by element sum in each row:

import numpy as np
df['NumVisits'].apply(np.cumsum)

For sum across all rows, for each element in row:

np.array(dailyHistogram['NumVisits'].tolist()).sum(axis=0)