josh josh - 4 years ago 105
Python Question

Mechanze form submission causes 'Assertion Error' in response when .read() is attempted

I am writing a web-crawl program with python and am unable to login using mechanize. The form on the site looks like:

<form method="post" action="PATLogon">
<h2 align="center"><img src="/myaladin/images/aladin_logo_rd.gif"></h2>
<!-- ALADIN Request parameters -->
<input type=hidden name=req value="db">
<input type=hidden name=key value="PROXYAUTH">
<input type=hidden name=url value="http://eebo.chadwyck.com/search">
<input type=hidden name=lib value="8">
<table>
<tr><td><b>Last Name:</b></td>
<td><input name=LN size=20 maxlength=26></td>
<tr><td><b>University ID or Library Barcode:</b></td>
<td><input type=password name=BC size=20 maxlength=21></td>
<tr><td><b>Institution:</b></td>
<td><select name="INST">
<option value="??">Select University ----</option>
<option value="AU">American</option>
<option value="CU">Catholic</option>
<option value="DC">District of Columbia</option>
<option value="GA">Gallaudet</option>
<option value="GM">George Mason</option>
<option value="GW">George Washington</option>
<option value="GT">Georgetown</option>
<option value="MU">Marymount</option>
<option value="TR">Trinity</option>
</select>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
<input type="submit" value="GO">
</td></tr></table></form>


So, I am able to set everything appropriately but on submitting the form and attempting to print the response I am left with an error. My code is as follows:

import mechanize
import time
br = mechanize.Browser()
br.set_handle_robots(False)

def connect():
# connection information
url = "https://www.aladin.wrlc.org/Z-WEB/Aladin?req=db&key=PROXYAUTH&lib=8&\url=http://eebo.chadwyck.com/search"
br.open(url)
time.sleep(0.5)
br.select_form(nr=0)
br["LN"] = "Reese"
br["BC"] = "myPassword"
br["INST"] = ["AU"]
response = br.submit()
print response.getheaders()


The error I get here is:

>>> eebolib.connect()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "eebolib.py", line 28, in connect
print response.read()
File "build/bdist.macosx-10.5-fat3/egg/mechanize/_response.py", line 190, in read
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 349, in read
data = self._sock.recv(rbufsize)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 553, in read
if self.length is not None:
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1282, in read
if amt is None or amt > self._line_left:
AssertionError


If anyone can provide some assistance on this I would be most appreciative.

Answer Source

This is the solution that I found:

import mechanize,urllib,ClientForm,cookielib,re,os,time
from BeautifulSoup import BeautifulSoup

cookies = mechanize.CookieJar()
opener = mechanize.build_opener(mechanize.HTTPCookieProcessor(cookies))
headers = [("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"),\
           ("Accept-Charset","ISO-8859-1,utf-8;q=0.7,*;q=0.7"),\
           ("Accept-Encoding","gzip, deflate"),\
           ("Accept-Language","en-us,en;q=0.5"),\
           ("Connection","keep-alive"),\
           ("Host","www.aladin.wrlc.org"),\
           ("Referer","https://www.aladin.wrlc.org/Z-WEB/Aladin?req=db&key=PROXYAUTHlib=8url=http://eebo.chadwyck.com/search"),\
           ("User-Agent","Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0) Gecko/20100101 Firefox/5.0")]
opener.addheaders = headers
mechanize.install_opener(opener)
params = urllib.urlencode({'LN':'myLN','BC':'myBC','INST':'myINST',\
                           'req':'db','key':'PROXYAUTH','lib':'8',\
                           'url':'http://eebo.chadwyck.com/search'})
mechanize.urlopen("https://www.aladin.wrlc.org/Z-WEB/PATLogon",params)

Hope this helps someone someday :)

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download