I have a password protected directory with a lot of files which I am trying to download by using Python's BeautifulSoup.
The problem is that as it is just a password protected directory and there is no login form, I can't seem to use Mechanize like this example?
There is no need to do screen automation to log in. This is using HTTP Authentication which HTTP libraries like urllib2, urllib3 and request (a library built on top of urllib) all support.
Even simpler, you could just prefix your URL with the username and password, which is what the HTTP libraries do under the hood.