MadProgrammer MadProgrammer - 2 months ago 31
Python Question

Pandas Groupby Return Average BUT! exclude NaN

So Im trying to make sense of the pandas groupby function and to reduce a large data frame I have. Here is an example:

A B
2016-09-23 19:36:08+00:00 NaN 34.0
2016-09-23 19:36:11+00:00 NaN 33.0
2016-09-23 19:36:12+00:00 24.1 NaN
2016-09-23 19:36:14+00:00 NaN 34.0
2016-09-23 19:36:17+00:00 NaN 34.0
2016-09-23 19:36:20+00:00 NaN 34.0
2016-09-23 19:36:22+00:00 24.2 NaN
2016-09-23 19:36:23+00:00 NaN 34.0
2016-09-23 19:36:26+00:00 NaN 34.0
2016-09-23 19:36:29+00:00 NaN 34.0
2016-09-23 19:36:32+00:00 24.1 NaN
2016-09-23 19:36:33+00:00 NaN 34.0
2016-09-23 19:37:00+00:00 NaN 34.0
2016-09-23 19:37:02+00:00 24.1 NaN


So I have 2 data series "A" and "B" that were sampled at different rates with their sampling time as the index of the original data frame.

I would like to now group the rows of the data frame by date/hour/minute and return the average of the data per minute. Here the average should ignore the missing values in the data frame.

So for example, I would return something like this:

A B
2016-09-23 19:36:00+00:00 24 34.0
2016-09-23 19:37:00+00:00 24.1 33.0


Is it possible to do this with a built in pandas function?

Answer

I think you need resample with Resampler.mean, which compute mean of groups, excluding missing values:

print (df.resample('1Min').mean())
                             A          B
2016-09-23 19:36:00  24.133333  33.888889
2016-09-23 19:37:00  24.100000  34.000000

Another solution with groupby:

print (df.groupby([pd.TimeGrouper('1Min')]).mean())
                             A          B
2016-09-23 19:36:00  24.133333  33.888889
2016-09-23 19:37:00  24.100000  34.000000