kojiro kojiro - 1 year ago 60
Python Question

Can I still use StringIO when the containing Writer() closes it?

I am using the Python avro library. I want to send an avro file over http, but I don't particularly want to save that file to disk first, so I thought I'd use StringIO to house the file contents until I'm ready to send. But avro.datafile.DataFileWriter thoughtfully takes care of closing the file handle for me, which makes it difficult for me to get the data back out of the StringIO. Here's what I mean in code:

from StringIO import StringIO
from avro.datafile import DataFileWriter
from avro import schema, io
from httplib2 import Http

HTTP = Http()
# Write the message data to a StringIO
# @return StringIO
def write_data():
message = TESTDATA
schema = getSchema()
datum_writer = io.DatumWriter(schema)
data = StringIO()
with DataFileWriter(data, datum_writer, writers_schema=schema, codec='deflate') as datafile_writer:
# If I return data inside the with block, the DFW buffer isn't flushed
# and I may get an incomplete file
return data

# Make the POST and dump its response
def main():
headers = {
"Content-Type": "avro/binary",
"Authorization": "Bearer %s" % BEARER,
body = write_data().getvalue() # AttributeError: StringIO instance has no attribute 'buf'
# the StringIO instance returned by write_data() is already closed. :(
resp, content = HTTP.request(
print resp, content

I do have some workarounds I can use, but none of them are terribly elegant. Is there any way to get the data from the StringIO after it's closed?

Answer Source

Not really.

The docs are very clear on this:


Free the memory buffer. Attempting to do further operations with a closed StringIO object will raise a ValueError.

The cleanest way of doing it would be to inherit from StringIO and override the close method to do nothing:

class MyStringIO(StringIO):
   def close(self):
   def _close(self):
       super(MyStringIO, self).close()

And call _close() when you're ready.