Fabian Fabian - 1 year ago 203
Javascript Question

Web Scraping Python using Google Chrome extension

Hi I am a Python newbie and I am webscrapping a webpage.

I am using the Google Chrome Developer Extension to identify the class of the objects I want to scrape. However, my code returns an empty array of results whereas the screenshots clearly show that that those strings are in the HTML code.
Chrome Developer

import requests
from bs4 import BeautifulSoup

url = 'http://www.momondo.de/flightsearch/?Search=true&TripType=2&SegNo=2&SO0=BOS&SD0=LON&SDP0=07-09-2016&SO1=LON&SD1=BOS&SDP1=12-09-2016&AD=1&TK=ECO&DO=false&NA=false'
html = requests.get(url)
soup = BeautifulSoup(html.text,"lxml")
x = soup.find_all("span", {"class":"value"})
#pprint.pprint (soup.div)

I am very much appreciating your help!

Many thanks!

Answer Source

Converted my comment to an answer...

Make sure the data you are expecting is actually there. Use print(soup.prettify()) to see what was actually returned from the request. Depending on how the site works, the data you are looking for may only exist in the browser after the javascript is processed. You might also want to take a look at selenium

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download