durden2.0 - 1 year ago 67
Python Question

# List of dictionaries from numpy array without for loop

Is there a way to vectorize an operation that takes several numpy arrays and puts them into a list of dictionaries?

Here's a simplified example. The real scenario might involve more arrays and more dictionary keys.

``````import numpy as np
x = np.arange(10)
y = np.arange(10, 20)
z = np.arange(100, 110)

print [dict(x=x[ii], y=y[ii], z=z[ii]) for ii in xrange(10)]
``````

I might have thousands or hundreds of thousands of iterations in the
`xrange`
call. All the manipulation to create
`x`
,
`y`
, and
`z`
is vectorized (my example is not as simple as above). So, there's only 1 for loop left to get rid of, which I expect would result in huge speed ups.

I've tried using
`map`
with a function to create the dict and all sorts of other work arounds. It seems the Python
`for`
loop is the slow part (as usual). I'm sort of stuck to using dictionaries because of a pre-existing API requirement. However, solutions without dicts and record arrays or something would be interesting to see, but ultimately I don't think that will work with the existing API.

Here is one (Num)?Pythonic way:

``````In [18]: names = np.array(['x', 'y', 'z'])
In [38]: map(dict, np.dstack((np.repeat(names[None, :], 10, axis=0), np.column_stack((x, y, z)))))
Out[38]:
[{'x': '0', 'y': '10', 'z': '100'},
{'x': '1', 'y': '11', 'z': '101'},
{'x': '2', 'y': '12', 'z': '102'},
{'x': '3', 'y': '13', 'z': '103'},
{'x': '4', 'y': '14', 'z': '104'},
{'x': '5', 'y': '15', 'z': '105'},
{'x': '6', 'y': '16', 'z': '106'},
{'x': '7', 'y': '17', 'z': '107'},
{'x': '8', 'y': '18', 'z': '108'},
{'x': '9', 'y': '19', 'z': '109'}]
``````

Also, note that if you don't need all of the dictionaries at once, you can simply create a generator and access to each item on demand.

``````(dict(x=x[ii], y=y[ii], z=z[ii]) for ii in xrange(10))
``````

If you want a nested dictionary, I suggest a list comprehension:

``````In [88]: inner = np.dstack((np.repeat(names[None, :], 10, axis=0), np.column_stack((x, y))))

In [89]: [{'connection': d} for d in map(dict, inner)]
Out[89]:
[{'connection': {'x': '0', 'y': '10'}},
{'connection': {'x': '1', 'y': '11'}},
{'connection': {'x': '2', 'y': '12'}},
{'connection': {'x': '3', 'y': '13'}},
{'connection': {'x': '4', 'y': '14'}},
{'connection': {'x': '5', 'y': '15'}},
{'connection': {'x': '6', 'y': '16'}},
{'connection': {'x': '7', 'y': '17'}},
{'connection': {'x': '8', 'y': '18'}},
{'connection': {'x': '9', 'y': '19'}}]
``````
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download