Ahmad Farid Ahmad Farid - 1 month ago 10
Python Question

How to read (stream) files from Google Drive in memory in python?

I'm trying to read large CSV files that are dropped on Google Drive using the google-api-python-client https://google.github.io/google-api-python-client/docs/epy/googleapiclient.http.MediaIoBaseDownload-class.html

I was able to download the file on the hard drive doing this:

request = drive_service.files().get_media(fileId=file_id)
fh = io.FileIO('test.csv', mode='w')
downloader = MediaIoBaseDownload(fh, request)
done = False
while done is False:
status, done = downloader.next_chunk()


But I was wondering if there's a simple way to read it in chunks in memory.

Answer Source
    api_service_object = self.service
    request = api_service_object.files().get_media(fileId=file_id)
    stream = io.BytesIO()
    downloader = MediaIoBaseDownload(stream, request)
    done = False
    # Retry if we received HttpError
    for retry in range(0, 5):
        try:
            while done is False:
                status, done = downloader.next_chunk()
                print "Download %d%%." % int(status.progress() * 100)
            return stream.getvalue()
        except HTTPError as error:
            print 'There was an API error: {}. Try # {} failed.'.format(
                error.response,
                retry,
            )