Vincent Vincent - 29 days ago 22
Python Question

Scrape password protected directory (with python)

I have a password protected directory with a lot of files which I am trying to download by using Python's BeautifulSoup.

enter image description here

The problem is that as it is just a password protected directory and there is no login form, I can't seem to use Mechanize like this example?

Anyone suggestions?

Answer

There is no need to do screen automation to log in. This is using HTTP Authentication which HTTP libraries like urllib2, urllib3 and request (a library built on top of urllib) all support.

Even simpler, you could just prefix your URL with the username and password, which is what the HTTP libraries do under the hood.

https://username:password@example.com