SaikiHanee SaikiHanee - 4 months ago 9
Python Question

Access and webscrape dynamic pages using python

I am trying to access a web page using it's form parameters.
I found the form parameters using Network headers in chrome's developer tab.
But it's not working, instead it's just opening the page before these parameters are used (i.e www.irishancestors.ie/search/townlands/ded_index.php)

import webbrowser

webbrowser.open('http://www.irishancestors.ie/search/townlands/ded_index.php?action=listp&parish=Aghagallon')


My intention is to retrieve all the tables of each District Electoral Division of all the counties.

enter image description here

Answer

webbrowser doesn't do what you think it does.

If you want to get/post data to a webpage, you should use requests

>>> import requests
>>> r = requests.get('https://api.github.com/events')
>>> r = requests.post('http://httpbin.org/post', data = {'key':'value'})

webbrowser is for launching your web browser.

Note that this won't work very well if there's a bunch of javascript in use (well, it might, but it requires a lot more work on your part). If you have a lot of Javascript, it may be easier to use selenium