Flow Nuwen Flow Nuwen - 1 month ago 13
Python Question

Extract List from Pandas Cell and Use List Elements as new column

Trying to create a dataframe that has the name of the election race, result (Republican - Democratic Popular Vote, as a fraction), and polling difference per poll. My code so far:

def results_polls_diff(editinfo, polls):
rows = []
for i, election in enumerate(editinfo):
polls_key = election['slug']
this_election = polls[polls_key]

npolls = this_election.shape[0]

diff = (this_election[candidates['R'].ix[i]] - this_election[candidates['D or I'].ix[i]])/100


for c in election['estimates']:
if c['party'] == 'Rep' :
r1 = c['value']

for c in election['estimates']:
if c['party'] == 'Dem' or c['party'] == 'ind' :
r2 = c['value']

result = (r1-r2)/100

#init_rows = []
#for d in diff:
# init_rows.append((polls_key, result, d))
#return init_rows

rows.append((polls_key, result, [d for d in diff]))
return rows
result_df = pd.DataFrame(results_polls_diff(editinfo, polls), columns = ['race', 'result', 'diff_list'])
result_df.head()


output:

race result diff_list
0 2014-delaware-senate-wade-vs-coons -0.220 [-0.18, -0.16, -0.25, -0.15]
1 2014-massachusetts-senate-herr-vs-markey -0.207 [-0.2, -0.15, -0.16, -0.25, -0.22, -0.26, -0.2...
2 2014-rhode-island-senate-zaccaria-vs-reed -0.207 [-0.45, -0.42, -0.35]
3 2014-montana-senate-daines-vs-curtis 0.177 [0.14, 0.18, 0.16, 0.21, 0.13]
4 2014-hawaii-senate-cavasso-vs-schatz -0.477 [-0.52, -0.26, -0.51, -0.54, -0.37, -0.32]


What I'm aiming for is something more like this:

race result diff_list
0 2014-delaware-senate-wade-vs-coons -0.22 -0.18
1 2014-delaware-senate-wade-vs-coons -0.22 -0.16
2 2014-delaware-senate-wade-vs-coons -0.22 -0.25
3 2014-delaware-senate-wade-vs-coons -0.22 -0.15


If I used the hashed out part of my code and change the append to
rows.append((init_rows))
, I get that result, but it doesn't seem to iterate through all of
editinfo
anymore. So the solution I'm looking for is either a way to get the iteration to work, or extract a list from the
diff_list
column so that the element occupy a single cell in that column, and duplicate the rest of the row.

Answer

This is one strategy. Consider the df

df = pd.DataFrame(dict(A=list('ab'), B=[1, 2], C=[[1, 2, 3], [4, 5, 6]], ))
df

enter image description here

option 1
use set_index, apply, unstack

df.set_index(['A', 'B']).C.apply(pd.Series).stack().reset_index(['A', 'B'], name='C')

option 2
build new index and dataframe, then unstack

names = ['A', 'B']
idx = pd.MultiIndex.from_tuples(df[names].values.tolist(), names=names)
pd.DataFrame(df.C.tolist(), idx).stack().reset_index(names, name='C')

enter image description here