Wayne Werner Wayne Werner - 2 months ago 9
Python Question

Is there a better way to handle writing csv in text mode and reading in binary mode?

I have code that looks something like this:

import csv
import os
import tempfile

from azure.storage import CloudStorageAccount


account = CloudStorageAccount(
account_name=os.environ['AZURE_ACCOUNT'],
account_key=os.environ['AZURE_ACCOUNT_KEY'],
)
service = account.create_block_blob_service()

with tempfile.NamedTemporaryFile(mode='w') as f:
writer = csv.DictWriter(f, fieldnames=['foo', 'bar'])
writer.writerow({'foo': 'just an example', 'bar': 'of what I do'})

with open(f.name, 'rb') as stream:
service.create_blob_from_stream(
container_name='test',
blob_name='nothing_secret.txt',
stream=stream,
)


Now, this is ugly. I don't like having to open the file twice. I know that the Azure API provides a way to upload text and binary, but my file has the potential to be several hundred MB large so I'm not too interested in sticking the whole thing in memory at a time (not that it would be the end of the world, but still).

Azure doesn't support uploading a file in text mode (that I can see), and csv doesn't seem to support writing to a binary file (at least not text data).

Is there a way that I can have two handles to the same file, one in binary and one in text mode? Of course I could write my own file wrapper, but I'd prefer to use something I don't have to maintain. Is there a better way to do this than what I've got?

Answer

Files opened in text mode have a buffer attribute. This object is the same one you would get by opening the file in binary mode, the text mode is just a wrapper on top of it.

Open your file in text mode, use it for read it, then seek the buffer back to the start and use it for uploading. Make sure you use + mode for reading and writing from the same handle.

with tempfile.NamedTemporaryFile(mode='w+') as f:
    ...

    f.seek(0)

    service.create_blob_from_stream(
        ...
        stream=f.buffer,
    )

You can go the other way too, by opening in binary mode then wrapping with io.TextIOWRapper(f).

Comments