Fabian Fabian - 3 months ago 20
Javascript Question

Web Scraping Python using Google Chrome extension

Hi I am a Python newbie and I am webscrapping a webpage.

I am using the Google Chrome Developer Extension to identify the class of the objects I want to scrape. However, my code returns an empty array of results whereas the screenshots clearly show that that those strings are in the HTML code.
Chrome Developer

import requests
from bs4 import BeautifulSoup

url = 'http://www.momondo.de/flightsearch/?Search=true&TripType=2&SegNo=2&SO0=BOS&SD0=LON&SDP0=07-09-2016&SO1=LON&SD1=BOS&SDP1=12-09-2016&AD=1&TK=ECO&DO=false&NA=false'
html = requests.get(url)
soup = BeautifulSoup(html.text,"lxml")
x = soup.find_all("span", {"class":"value"})
print(x)
#pprint.pprint (soup.div)


I am very much appreciating your help!

Many thanks!

Answer

Converted my comment to an answer...

Make sure the data you are expecting is actually there. Use print(soup.prettify()) to see what was actually returned from the request. Depending on how the site works, the data you are looking for may only exist in the browser after the javascript is processed. You might also want to take a look at selenium

Comments