Jamie Bull Jamie Bull - 2 months ago 17
Python Question

Sending client-side authorisation certificate to HTTPS site - Python

I've had a long, long day of trying to get a Client Authorisation certificate working with either

mechanize
or
requests
and have finally resorted to
IEC
for automation which has been blessedly simple - up till now. I just have to click to say "yes, use this certificate" and then the automation fills in forms, finds the links and tries to download the files.

I've got to the stage of bringing up the
Open Save Cancel
dialog for the first file but can't see how to interact with that. Googling around I'm not sure it's possible without using mouse clicks which seems like overkill and I can't get
SendKeys.py
(link)
to work as I'm apparently misssing
vcvarsall.bat
.

Everything seems to be a dead end today!

So there are really two questions here.

1) Is there a way of passing
cert.pem
and
key.pem
files to either
requests
or
mechanize
? (my preferred route)

My current
mechanize
code here is:

br = mechanize.Browser()
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)
br.open(root_url + '/home.html')
# Hangs on the following line due to not passing Client Certificate
r = br.open(root_url + '/secure/secureTerms.html')


Traceback when it eventually times out is:

Traceback (most recent call last):
File "C:\Users\Jamie\Dropbox\workspace\XXXXX\XXXXX\myfile.py", line 29, in <module>
r = br.open('https://www.example.com/secure/authorisedUsers.html')
File "build\bdist.win32\egg\mechanize\_mechanize.py", line 203, in open
File "build\bdist.win32\egg\mechanize\_mechanize.py", line 230, in _mech_open
File "build\bdist.win32\egg\mechanize\_opener.py", line 193, in open
File "build\bdist.win32\egg\mechanize\_urllib2_fork.py", line 344, in _open
File "build\bdist.win32\egg\mechanize\_urllib2_fork.py", line 332, in _call_chain
File "build\bdist.win32\egg\mechanize\_urllib2_fork.py", line 1170, in https_open
File "build\bdist.win32\egg\mechanize\_urllib2_fork.py", line 1116, in do_open
File "C:\Python27\Lib\httplib.py", line 1025, in getresponse
response.begin()
File "C:\Python27\Lib\httplib.py", line 401, in begin
version, status, reason = self._read_status()
File "C:\Python27\Lib\httplib.py", line 365, in _read_status
raise BadStatusLine(line)
httplib.BadStatusLine: ''


And my
requests
code is:

agent = requests.session()
agent.get(root_url + '/home.html')
data = {'redirectPage':'reportSearchAddressByPostcode',
'accept':'Accept Terms'}
agent.post(URL, data=data, headers=hdr)
# Again, this hangs on the following line
response = agent.get(root_url + '/secure/secureTerms.html', headers=hdr)


And here's the Traceback I get from
requests
:

Traceback (most recent call last):
File "C:\Users\Jamie\Dropbox\workspace\XXXXX\XXXXX\myfile.py", line 27, in <module>
headers=hdr, cert=('cert.pem', 'key.pem'))
File "C:\Python27\lib\site-packages\requests\sessions.py", line 363, in get
return self.request('GET', url, **kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 347, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 460, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests\adapters.py", line 327, in send
raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.example.com', port=443): Max retries exceeded with url: /secure/authorisedUsers.html (Caused by <class 'httplib.BadStatusLine'>: '')


2) Or is there a way to bypass or click through the Open Save Cancel? (acceptable kludge)

EDIT

I've tried this in cURL and it works fine with the following parameters:

curl -k -v --key key.pem --cert cert.pem https://www.example.com/secure/authorisedUsers.html


The server responds with:

* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
* subject: OU=Go to https://www.name.com/repository/index.html; OU=Name
SSL123 certificate; OU=Domain Validated; CN=www.example.com
* start date: 2012-12-03 00:00:00 GMT
* expire date: 2013-12-03 23:59:59 GMT
* issuer: C=US; O=Name, Inc.; OU=Domain Validated SSL; CN=Name DV SSL
CA
* SSL certificate verify result: unable to get local issuer certificate (
20), continuing anyway.
> GET /secure/authorisedUsers.html HTTP/1.1
> User-Agent: curl/7.30.0
> Host: www.example.com
> Accept: */*
>
* SSLv3, TLS handshake, Hello request (0):
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Request CERT (13):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS handshake, CERT verify (15):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
< HTTP/1.1 302 Moved Temporarily
< Date: Fri, 21 Jun 2013 00:52:44 GMT
< Set-Cookie: site-text-size=medium; Domain=www.example.com; Expires=Sat,
21-Jun-2014 00:52:44 GMT; Path=/
< Set-Cookie: user=SITE_MEMBER_V2; Domain=www.example.com; Path=/
< Location: https://www.example.com/secure/addressSearch.html
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=UTF-8
<
<html><body><p>Redirecting to <a href="https://www.example.com/secure/addr
essSearch.html">https://www.example.com/secure/addressSearch.html</a></p><
/body></html>* Closing connection 0
* SSLv3, TLS alert, Client hello (1):


This is all what I was expecting to see, so is there anything in here that would suggest why it's failing in Python?

Answer

Here is how you specify the client certificates to use for that HTTPS session:

Mechanize

br = mechanize.Browser()
br.add_client_certificate(myurl, mykey, cert)
br.set_handle_robots( False )
#httplib.HTTPSConnection.connect = connect
resp = br.open(myurl)

requests

resp = requests.get('<https_url>', cert=(mycert, mykey), verify=False)

Update: I edited the code for mechanize. We need to use add_client_certificate API with the Browser() instance to specify the client certificate.