Mic Mic - 7 months ago 43
Python Question

Access code in Firefox Inspector with Python

I'm learning scraping with Python. I use

requests
and
BeautifulSoup
to extract information from the source code. However, I see that not all the information that are available in
Firefox Inspector
are reported in the source code. Why is it the case?

jDo jDo
Answer

The site has a separate JSON API for auction bids.

import requests
import json
from bs4 import BeautifulSoup as soup


data = requests.get("http://www.charitystars.com/ajax/getbidshistory?oid=9315")
data_dict = json.loads(data.text)

auction_table = soup(data_dict["html"])

for td in auction_table.find_all("td"):
    print td.get_text()

"""
28/04/2016
 andy.shirt.5

€ 120

27/04/2016
 simone.vanin

€ 60

"""

You can simply increment the oid to get all auctions. Be nice to the site though! Add delays and such so you don't end up DOSing a charity!