joemastersemison joemastersemison - 10 months ago 63
Python Question

How can I use boto to stream a file out of Amazon S3 to Rackspace Cloudfiles?

I'm copying a file from S3 to Cloudfiles, and I would like to avoid writing the file to disk. The Python-Cloudfiles library has an call that looks to be what I need, but I can't find an equivalent call in boto. I'm hoping that I would be able to do something like:


Is this possible with boto (or I suppose any other s3 library)?

Answer Source

The Key object in boto, which represents on object in S3, can be used like an iterator so you should be able to do something like this:

>>> import boto
>>> c = boto.connect_s3()
>>> bucket = c.lookup('garnaat_pub')
>>> key = bucket.lookup('Scan1.jpg')
>>> for bytes in key:
...   write bytes to output stream

Or, as in the case of your example, you could do:

>>> shutil.copyfileobj(key,