anthr anthr - 4 months ago 18
Python Question

Counting dates in a range set by pandas dataframe

I have a pandas dataframe that contains two date columns, a start date and an end date that defines a range. I'd like to be able to collect a total count for all dates across all rows in the dataframe, as defined by these columns.

For example, the table looks like:

index start_date end date
0 '2015-01-01' '2015-01-17'
1 '2015-01-03' '2015-01-12'


And the result would be a per date aggregate, like:

date count
'2015-01-01' 1
'2015-01-02' 1
'2015-01-03' 2


and so on.

My current approach works but is extremely slow on a big dataframe as I'm looping across the rows, calculating the range and then looping through this. I'm hoping to find a better approach.

Currently I'm doing :

date = pd.date_range (min (df.start_date), max (df.end_date))
df2 = pd.DataFrame (index =date)
df2 ['count'] = 0

for index, row in df.iterrows ():
dates = pd.date_range (row ['start_date'], row ['end_date'])
for date in dates:
df2.loc['date']['count'] += 1

Answer

After stacking the relevant columns as suggested by @Sam, just use value_counts.

df[['start_date', 'end date']].stack().value_counts()

EDIT:

Given that you also want to count the dates between the start and end dates:

start_dates = pd.to_datetime(df.start_date)
end_dates = pd.to_datetime(df.end_date)

>>> pd.Series(dt.date() for group in 
              [pd.date_range(start, end) for start, end in zip(start_dates, end_dates)]  
              for dt in group).value_counts()
Out[178]: 
2015-01-07    2
2015-01-06    2
2015-01-12    2
2015-01-05    2
2015-01-04    2
2015-01-10    2
2015-01-03    2
2015-01-09    2
2015-01-08    2
2015-01-11    2
2015-01-16    1
2015-01-17    1
2015-01-14    1
2015-01-15    1
2015-01-02    1
2015-01-01    1
2015-01-13    1
dtype: int64