Vasily Vasily - 5 months ago 17
Python Question

How to use progressbar module with urlretrieve

My pyhton3 script downloads a number of images over the internet using urlretrieve, and I'd like to add a progressbar with a completed percentage and download speed for each download.

The progressbar module seems like a good solution, but although I've looked through their examples, and example4 seems like the right thing, I still can't understand how to wrap it around the urlretrieve.

I guess I should add a third parameter:

urllib.request.urlretrieve('img_url', 'img_filename', some_progressbar_based_reporthook)


But how do I properly define it?

Answer

The hook is defined as:

urlretrieve(url[, filename[, reporthook[, data]]]) "The third argument, if present, is a hook function that will be called once on establishment of the network connection and once after each block read thereafter. The hook will be passed three arguments; a count of blocks transferred so far, a block size in bytes, and the total size of the file. The third argument may be -1 on older FTP servers which do not return a file size in response to a retrieval request. "

So, you can write a hook as follows:

# Global variables
pbar = None
downloaded = 0

def show_progress(count, block_size, total_size):
    if pbar is None:
        pbar = ProgressBar(maxval=total_size)

    downloaded += block_size
    pbar.update(block_size)
    if downloaded == total_size:
        pbar.finish()
        pbar = None
        downloaded = 0

As a side note I strongly recommend you to use requests library which is a lot easier to use and you can iterate over the response with the iter_content() method.