claudiaann1 claudiaann1 -5 years ago 119
Python Question

Scrape webpage after form fill

Im trying to scrape the response to this website using a pre-filled zip:
zip who (i.e. the zip code is already filled in.) I tried to do this using the scrapy shell as follows

scrapy shell

but the response does not contain the form-filled page, but only the content from the main page and without the details specific to that zip code. I tried filling in the form information using requests and lxml, but clearly I am doing something wrong.

import requests
import lxml.html as lh
url = ''

form_data = {
'zip': '77098'
response =, data=form_data)
tree = lh.document_fromstring(response.content)

and the table element for the data (td where class = 'keysplit') still does not exist. If you have ideas to get this working (hopefully with something simple in like requests & lxml) that would be best.

Answer Source

The reason that you can't find this data in the HTML is that it's generated dynamically with a script. If you look at the first script in the HTML, you'll see a function called getData that contains the data that you want. Another script later uses this function to build what you see in your browser.

So to scrape this data I'd just extract it directly from the script: get the string that the function returns, split it by , and so on.

Good luck!

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download