Sitz Blogz Sitz Blogz - 3 months ago 22
Python Question

Use qcut pandas for multiple valuable categorizing

I am trying to use two values from two columns from a dataframe and perform

qcut
categorization.

single value categorizing it quite simple. But two variables as pairs and vs is something I am trying get.

Input:

date,startTime,endTime,day,c_count,u_count
2004-01-05,22:00:00,23:00:00,Mon,18944,790
2004-01-05,23:00:00,00:00:00,Mon,17534,750
2004-01-06,00:00:00,01:00:00,Tue,17262,747
2004-01-06,01:00:00,02:00:00,Tue,19072,777
2004-01-06,02:00:00,03:00:00,Tue,18275,785
2004-01-06,03:00:00,04:00:00,Tue,13589,757
2004-01-06,04:00:00,05:00:00,Tue,16053,735
2004-01-06,05:00:00,06:00:00,Tue,11440,636
2004-01-06,06:00:00,07:00:00,Tue,5972,513
2004-01-06,07:00:00,08:00:00,Tue,3424,382
2004-01-06,08:00:00,09:00:00,Tue,2696,303
2004-01-06,09:00:00,10:00:00,Tue,2350,262
2004-01-06,10:00:00,11:00:00,Tue,2309,254


Code with pure python but I am trying to do the same in pandas.

for row in csv.reader(inp):
if int(row[1])>(0.80*c_count) and int(row[2])>(0.80*u_count):
val='highly active'
elif int(row[1])>=(0.60*c_count) and int(row[2])<=(0.60*u_count):
val='active'
elif int(row[1])<=(0.40*c_count) and int(row[2])>=(0.40*u_count):
val='event based'
elif int(row[1])<(0.20*c_count) and int(row[2])<(0.20*u_count):
val ='situational'
else:
val= 'viewers'


What I am trying to find is ?


  1. c_count
    and
    u_count
    both

  2. Like in the above code
    c_count
    vs
    u_count


Answer

You can create a Series for each quantile group:

q = df[['c_count', 'u_count']].apply(lambda x: pd.qcut(x, np.linspace(0, 1, 6), 
                                                       labels=np.arange(5)))
q
Out: 
   c_count u_count
0        4       4
1        3       3
2        3       2
3        4       4
4        4       4
5        2       3
6        2       2
7        2       2
8        1       1
9        1       1
10       0       0
11       0       0
12       0       0

0 is for the first 20%, 1 is for 20%-40% and goes on.

Now the if logic works a little different here. For the else part, first populate the column:

df['val'] = 'viewers'

Anything we do afterwards will overwrite the values in this column if condition is satisfied. So the operation we do later precedes the previous one. From bottom to top:

df.ix[(q['c_count'] < 1) & (q['u_count'] < 1), 'val'] = 'situational'
df.ix[(q['c_count'] < 2) & (q['u_count'] > 1), 'val'] = 'event_based'
df.ix[(q['c_count'] > 2) & (q['u_count'] < 2), 'val'] = 'active'
df.ix[(q['c_count'] > 3) & (q['u_count'] > 3), 'val'] = 'highly active'

The first condition checks whether both c_count and u_count are in the first 20%. If so, changes the corresponding rows at 'val' column to situational. The remaining ones work in a similar manner. You might need to adjust comparison operators a little bit (greater vs greater than or equal to).