Al_Iskander Al_Iskander - 5 months ago 15
Python Question

concatenation large number of dataframes

I have a dictionary

D
that contains many dataframes.
I can access every dataframe with
D[0], D[1]...D[i]
, with the integers as keys/identifier of the respective dataframe.

I now want to concat all the dataframes in this fashion into a new dataframe:

new_df = pd.concat([D[0],D[1],...D[i]], axis= 1)


What would you suggest how I can solve this (concat needs still to be used)?

I tried with generating a list of D's and including this but received an error message.

Answer

I think the easiest thing to do is to use a list comprehension of the dict items:

In [14]:
d = {'a':pd.DataFrame(np.random.randn(5,3), columns=list('abc')), 'b':pd.DataFrame(np.random.randn(5,3), columns=list('def'))}
d

Out[14]:
{'a':           a         b         c
 0  0.030358  1.523752  1.040409
 1 -0.220019 -1.579467 -0.312059
 2  1.019489 -0.272261  1.182399
 3  0.580368  1.985362 -0.835338
 4  0.183974 -1.150667  1.571003, 'b':           d         e         f
 0 -0.911246  0.721034 -0.347018
 1  0.483298 -0.553996  0.374566
 2 -0.041415 -0.275874 -0.858687
 3  0.105171 -1.509721  0.265802
 4 -0.788434  0.648109  0.688839}

In [29]:
pd.concat([df for k,df in d.items()], axis=1)

Out[29]:
          a         b         c         d         e         f
0  0.030358  1.523752  1.040409 -0.911246  0.721034 -0.347018
1 -0.220019 -1.579467 -0.312059  0.483298 -0.553996  0.374566
2  1.019489 -0.272261  1.182399 -0.041415 -0.275874 -0.858687
3  0.580368  1.985362 -0.835338  0.105171 -1.509721  0.265802
4  0.183974 -1.150667  1.571003 -0.788434  0.648109  0.688839
Comments