Mawg Mawg - 10 months ago 51
Python Question

Memory error on large Shapefile in Python

Since we like to see code, here is some:

import shapefile
data = shapefile.Reader("data_file.shp")
shapes = data.shapes()

it's not relevant, but someone will ask for it if I don't post it.

My problem is that getting the shapes from the Shapefile reader gives me an exception
when using Pyshp.

file is quite large, at 1.2 gB. But I am using ony 3% of my machine's 32gB, so I don't understand it.

Is there any other approach that I can take? Can process the file in chunks in Python? Or use some tool to spilt the file into chinks, then process each of them individually?

Perhaps I ought to have asked on ? @PolyGeo do you read S.O?


Quoting from this answer by thomas:

The MemoryError exception that you are seeing is the direct result of running out of available RAM. This could be caused by either the 2GB per program limit imposed by Windows (32bit programs), or lack of available RAM on your computer. (This link is to a previous question). You should be able to extend the 2GB by using 64bit copy of Python, provided you are using a 64bit copy of windows.

So try a 64bit copy of Python or provide more detail about your platform and Python versions.