MinimalMaximizer MinimalMaximizer - 2 months ago 13
Python Question

Scrape Data From a sports table using Python and Beautiful soup

I'm trying to scrape the data from a table - namely (http://stats.nba.com/leagueTeamGeneral.html?pageNo=1&rowsPerPage=30). I am having difficulty with using the right commands. Tried various parameters, none worked. Ideally having the data returned in the format,
example:
Atlanta Hawks,32, 48.8, 18, 14, .563, etc

I can get the data formatted no problem, just getting the required data is what is causing me grief.

import urllib2
from bs4 import BeautifulSoup

page = 'http://stats.nba.com/leagueTeamGeneral.html?pageNo=1&rowsPerPage=30'
page = urllib2.urlopen(page)
soup = BeautifulSoup(page)
for dS in soup.find_all(???):
print(dS.get(???))

Answer

use a tool like firefox firebug to track down the html call you need, looking at the link you shared in firebug 'net' tab shows that the data you are after is in a subsequent request call to http://www.nba.com/cmsinclude/desktopWrapperHeader_jsonp.html which actually contains json data, not sure BeautifulSoup will be handy here, try to load it using python json