MagicP MagicP - 4 years ago 109
PHP Question

Scraping php site with python for a beginner

I'm a complete newbie to scraping, and I'd like to get some data from here :
http://fse.vdkruijssen.eu/ferrylist.php

I'm using Python 2.7 as it is the only language I know something about.
I've tried and searched a lot, but there is a simple form to fill before accessing the data that I can't go through... No login required.

This is what I have so far:

import requests
from bs4 import BeautifulSoup

url='http://fse.vdkruijssen.eu/ferrylist.php'
params ={'selectplane':'Cessna 208 Caravan','submit':''}

response=requests.post(url, data=params)
soup = BeautifulSoup(response.text, 'html.parser')
print soup


Any help would be highly appreciated...
Thank you !

EDIT : thank you 宏杰李! Now beautifulsoup is raising a looping error in \bs4\element.py
and end with a max recursion depth exceed
Do you have any ideas about that?


File "C:\Python27\lib\site-packages\bs4\element.py", line 1235, in
decode_contents
formatter))

File "C:\Python27\lib\site-packages\bs4\element.py", line 1166, in decode
indent_contents, eventual_encoding, formatter)

File "C:\Python27\lib\site-packages\bs4\element.py", line 1235, in
decode_contents
formatter))

File "C:\Python27\lib\site-packages\bs4\element.py", line 1166, in decode
indent_contents, eventual_encoding, formatter)

File "C:\Python27\lib\site-packages\bs4\element.py", line 1235, in
decode_contents
formatter))

File "C:\Python27\lib\site-packages\bs4\element.py", line 1138, in decode
text = self.format_string(val, formatter)

File "C:\Python27\lib\site-packages\bs4\element.py", line 167, in
format_string
output = formatter(s) File "C:\Python27\lib\site-packages\bs4\element.py", line 124, in
substitute_xml
ns, EntitySubstitution.substitute_xml)

File "C:\Python27\lib\site-packages\bs4\element.py", line 108, in
_substitute_if_appropriate
if (isinstance(ns, NavigableString) RuntimeError: maximum recursion depth exceeded while calling a Python object

Answer Source
params ={'selectplane':'Cessna 208 Caravan',
        'submit':''}

You missed a param:

enter image description here

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download