I have to create a very large grid using numpy meshgrid. In order to save memory I use int8 as my dtype for the arrays I'm trying to mesh. However, meshgrid keeps changing the type to int64 which uses a ton of memory. Here is a simple example of the problem...
import numpy
grids = [numpy.arange(1, 4, dtype=numpy.int8), numpy.arange(1, 5, dtype=numpy.int8)]
print grids
print grids[0].dtype, grids[0].nbytes
x1, y1 = numpy.meshgrid(*grids)
print x1.dtype, x1.nbytes
[array([1, 2, 3], dtype=int8), array([1, 2, 3, 4], dtype=int8)]
int8 3
int64 96
You can set the optional copy
parameter of numpy.meshgrid()
to False
(note, however, that it has some constraints):
meshgrid(*xi, **kwargs)
...
copy
:bool
, optionalIf
False
, a view into the original arrays are returned in order to conserve memory. Default isTrue
. Please note thatsparse=False
,copy=False
will likely return non-contiguous arrays. Furthermore, more than one element of a broadcast array may refer to a single memory location. If you need to write to the arrays, make copies first.
Proof that it works:
>>> import numpy
>>>
>>> grids = [numpy.arange(1, 4, dtype=numpy.int8), numpy.arange(1, 5, dtype=numpy.int8)]
>>>
>>> print grids
[array([1, 2, 3], dtype=int8), array([1, 2, 3, 4], dtype=int8)]
>>> print grids[0].dtype, grids[0].nbytes
int8 3
>>>
>>> x1, y1 = numpy.meshgrid(*grids, copy=False)
>>> # ^^^^^^^^^^
>>> print x1.dtype, x1.nbytes
int8 12