I use the following code to stream large files from the Internet into a local file:
fp = open(file, 'wb')
req = urllib2.urlopen(url)
for line in req:
No reason to work line by line (small chunks AND requires Python to find the line ends for you!-), just chunk it up in bigger chunks, e.g.:
# from urllib2 import urlopen # Python 2 from urllib.request import urlopen # Python 3 response = urlopen(url) CHUNK = 16 * 1024 with open(file, 'wb') as f: while True: chunk = response.read(CHUNK) if not chunk: break f.write(chunk)
Experiment a bit with various CHUNK sizes to find the "sweet spot" for your requirements.