b10hazard b10hazard - 1 year ago 88
Python Question

How do I stop numpy meshgrid from setting the default datatype to int64

I have to create a very large grid using numpy meshgrid. In order to save memory I use int8 as my dtype for the arrays I'm trying to mesh. However, meshgrid keeps changing the type to int64 which uses a ton of memory. Here is a simple example of the problem...

import numpy

grids = [numpy.arange(1, 4, dtype=numpy.int8), numpy.arange(1, 5, dtype=numpy.int8)]

print grids
print grids[0].dtype, grids[0].nbytes

x1, y1 = numpy.meshgrid(*grids)

print x1.dtype, x1.nbytes


This script prints

[array([1, 2, 3], dtype=int8), array([1, 2, 3, 4], dtype=int8)]

int8 3

int64 96


Why does meshgrid do this? Is there any way to stop it? I need to create a huge array so I cannot use meshgrid unless I can control the data type of the output. Is this intended behavior or is it a numpy bug? Every other function I've used in numpy preserves datatype of allows you to change it. The meshgrid function does not appear to allow this.

Answer Source

You can set the optional copy parameter of numpy.meshgrid() to False (note, however, that it has some constraints):

meshgrid(*xi, **kwargs)

...

copy : bool, optional

If False, a view into the original arrays are returned in order to conserve memory. Default is True. Please note that sparse=False, copy=False will likely return non-contiguous arrays. Furthermore, more than one element of a broadcast array may refer to a single memory location. If you need to write to the arrays, make copies first.

Proof that it works:

>>> import numpy
>>> 
>>> grids = [numpy.arange(1, 4, dtype=numpy.int8), numpy.arange(1, 5, dtype=numpy.int8)]
>>> 
>>> print grids
[array([1, 2, 3], dtype=int8), array([1, 2, 3, 4], dtype=int8)]
>>> print grids[0].dtype, grids[0].nbytes
int8 3
>>>
>>> x1, y1 = numpy.meshgrid(*grids, copy=False)
>>>                        #        ^^^^^^^^^^
>>> print x1.dtype, x1.nbytes
int8 12
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download