durden2.0 durden2.0 - 1 month ago 12
Python Question

List of dictionaries from numpy array without for loop

Is there a way to vectorize an operation that takes several numpy arrays and puts them into a list of dictionaries?

Here's a simplified example. The real scenario might involve more arrays and more dictionary keys.

import numpy as np
x = np.arange(10)
y = np.arange(10, 20)
z = np.arange(100, 110)

print [dict(x=x[ii], y=y[ii], z=z[ii]) for ii in xrange(10)]


I might have thousands or hundreds of thousands of iterations in the
xrange
call. All the manipulation to create
x
,
y
, and
z
is vectorized (my example is not as simple as above). So, there's only 1 for loop left to get rid of, which I expect would result in huge speed ups.

I've tried using
map
with a function to create the dict and all sorts of other work arounds. It seems the Python
for
loop is the slow part (as usual). I'm sort of stuck to using dictionaries because of a pre-existing API requirement. However, solutions without dicts and record arrays or something would be interesting to see, but ultimately I don't think that will work with the existing API.

Answer

Here is one (Num)?Pythonic way:

In [18]: names = np.array(['x', 'y', 'z'])
In [38]: map(dict, np.dstack((np.repeat(names[None, :], 10, axis=0), np.column_stack((x, y, z)))))
Out[38]: 
[{'x': '0', 'y': '10', 'z': '100'},
 {'x': '1', 'y': '11', 'z': '101'},
 {'x': '2', 'y': '12', 'z': '102'},
 {'x': '3', 'y': '13', 'z': '103'},
 {'x': '4', 'y': '14', 'z': '104'},
 {'x': '5', 'y': '15', 'z': '105'},
 {'x': '6', 'y': '16', 'z': '106'},
 {'x': '7', 'y': '17', 'z': '107'},
 {'x': '8', 'y': '18', 'z': '108'},
 {'x': '9', 'y': '19', 'z': '109'}]

Also, note that if you don't need all of the dictionaries at once, you can simply create a generator and access to each item on demand.

(dict(x=x[ii], y=y[ii], z=z[ii]) for ii in xrange(10))

If you want a nested dictionary, I suggest a list comprehension:

In [88]: inner = np.dstack((np.repeat(names[None, :], 10, axis=0), np.column_stack((x, y))))

In [89]: [{'connection': d} for d in map(dict, inner)]
Out[89]: 
[{'connection': {'x': '0', 'y': '10'}},
 {'connection': {'x': '1', 'y': '11'}},
 {'connection': {'x': '2', 'y': '12'}},
 {'connection': {'x': '3', 'y': '13'}},
 {'connection': {'x': '4', 'y': '14'}},
 {'connection': {'x': '5', 'y': '15'}},
 {'connection': {'x': '6', 'y': '16'}},
 {'connection': {'x': '7', 'y': '17'}},
 {'connection': {'x': '8', 'y': '18'}},
 {'connection': {'x': '9', 'y': '19'}}]