eclark eclark - 2 years ago 153
Python Question

How to read a gzip netcdf file in python?

I have a working python program that reads in a number of large netCDF files using the Dataset command from the netCDF4 module. Here is a snippet of the relevant parts:

from netCDF4 import Dataset
import glob

infile_root = 'start_of_file_name_'

for infile in sorted(glob.iglob(infile_root + '*')):
ncin = Dataset(infile,'r')
ncin.close()


I want to modify this to read in netCDF files that are gzipped. The files themselves were gzipped after creation; they are not internally compressed (i.e., the files are *.nc.gz). If I were reading in gzipped text files, the command would be:

from netCDF4 import Dataset
import glob
import gzip

infile_root = 'start_of_file_name_'

for infile in sorted(glob.iglob(infile_root + '*.gz')):
f = gzip.open(infile, 'rb')
file_content = f.read()
f.close()


After googling around for maybe half an hour and reading through the netCDF4 documentation, the only way I can come up with to do this for netCDF files is:

from netCDF4 import Dataset
import glob
import os

infile_root = 'start_of_file_name_'

for infile in sorted(glob.iglob(infile_root + '*.gz')):
os.system('gzip -d ' + infile)
ncin = Dataset(infile[:-3],'r')
ncin.close()
os.system('gzip ' + infile[:-3])


Is it possible to read gzip files with the Dataset command directly? Or without otherwise calling gzip through os?

Answer Source

Because NetCDF4-Python wraps the C NetCDF4 library, you're out of luck as far as using the gzip module to pass in a file-like object. The only option is, as suggested by @tdelaney, to use the gzip to extract to a temporary file.

If you happen to have any control over the creation of these files, NetCDF version 4 files support zlib compression internally, so that using gzip is superfluous. It might also be worth converting the files from version 3 to version 4 if you need to repeatedly process these files.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download