I'm uploading potentially large files to a web server. Currently I'm doing this:
f = open('somelargefile.zip','rb')
request = urllib2.Request(url,f.read())
response = urllib2.urlopen(request)
Reading through the mailing list thread linked to by systempuntoout, I found a clue towards the solution.
mmap module allows you to open file that acts like a string. Parts of the file are loaded into memory on demand.
Here's the code I'm using now:
import urllib2 import mmap # Open the file as a memory mapped string. Looks like a string, but # actually accesses the file behind the scenes. f = open('somelargefile.zip','rb') mmapped_file_as_string = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) # Do the request request = urllib2.Request(url, mmapped_file_as_string) request.add_header("Content-Type", "application/zip") response = urllib2.urlopen(request) #close everything mmapped_file_as_string.close() f.close()