Turtle Turtle - 3 months ago 17
Javascript Question

Python3 requests library to submit form that disallows post request

I am trying to get the police district from a given location at the Philly Police webpage. I too many locations to do this by hand, so I am trying to automate the process using Python's requests library. The webpage's form that holds the location value is as follows:

<form id="search-form" method="post" action="districts/searchAddress">
<div class="clearfix">
<label for="search-address-box"><span>Enter Your Street Address</span></label>
<div class="input">
<input tabindex="1" class="district-street-address-input" id="search-address-box" name="name" type="text" value="">
<div class="actions" style="float: left;">
<button tabindex="3" type="submit" class="btn btn-success">Search</button>
<a id="use-location" href="https://www.phillypolice.com/districts/index.html?_ID=7&_ClassName=DistrictsHomePage#" style="float: left; margin: 7px 0 0 12px;"><i class="icon-location-arrow"></i>Use Current Location</a>
<div id="current-location-display" style="display: none;"><p>Where I am right now.</p></div>

However when I try to post or put to the webpage using the following:

r = requests.post('http://www.phillypolice.com/districts',data={'search-address-box':'425 E. Roosevelt Blvd'})

I get error 405, POST is not allowed. I then turned off Javascript and tried to find the district on the webpage, and when I hit submit I received the same 405 error message. Therefore the form is definitely not submitted and the district is found using JavaScript.

Is there a way to simulate 'clicking' the submit button to trigger the JavaScript using the requests library?


There are 2 major things happening when you hit "submit" - there is a request to the google geocode service and an XHR request to the "searchAddress.json" endpoint that uses the coordinates returned by the geocode service.

You can try to simulate the above requests carefully handling all the API keys and required parameters, or you can stay on a higher level and use browser automation via selenium.

Working example using PhantomJS headless browser:

In [2]: from selenium import webdriver

In [3]: driver = webdriver.PhantomJS()

In [4]: driver.get("https://www.phillypolice.com/districts/")

In [5]: address = "425 E. Roosevelt Blvd"

In [6]: search_box = driver.find_element_by_id("search-address-box")

In [7]: search_box.send_keys(address)

In [8]: search_box.submit()

In [9]: driver.find_element_by_css_selector("#district-menu h2").text
Out[9]: u'35th District'

In [10]: driver.find_element_by_css_selector("#district-menu h4").text
Out[10]: u'PSA 2'

And, you might need the Explicit Waits to handle the "timing" issues.