edsheeran edsheeran - 3 months ago 9
HTML Question

Why isn't the HTML I get from BeautifulSoup the same as the one I see when I inspect element?

I am making a username scraper and I really can't understand why the HTML is 'disappearing' when I parse it. Let's take this site for example:
http://www.lolking.net/leaderboards#/eune/1

HTML output

See how there is a tbody and a bunch of tables in it?
Well when I parse it and output it to the shell the tbody is empty

<div style="background: #333; box-shadow: 0 0 2px #000; padding: 10px;">
<table class="lktable" id="leaderboard_table" width="100%">
<thead>
<tr>
<th style="width: 80px;">
Rank
</th>
<th style="width: 80px;">
Change
</th>
<th style="width: 100px;">
Tier
</th>
<th>
Summoner
</th>
<th style="width: 150px;">
Top Champions
</th>
</tr>
</thead>
<tbody>
</tbody>
</table>
</div>
</div>


Why is this happening and how can I fix it?

Answer

You can get all the data in json format, ll you need to do is parse a value from script tag inside the original page source and pass it to "http://www.lolking.net/leaderboards/some_value_here/eune/1.json":

from bs4 import BeautifulSoup
import requests
import re

patt = re.compile("\$\.get\('/leaderboards/(\w+)/")
js = "http://www.lolking.net/leaderboards/{}/eune/1.json"
soup = BeautifulSoup(requests.get("http://www.lolking.net/leaderboards#/eune/1").content)
script = soup.find("script", text=re.compile("\$\.get\('/leaderboards/"))

val = patt.search(script.text).group(1)
data = requests.get(js.format(val)).json()

data gives you json that contains all the player info like:

{'data': [{'division': '1',
           'global_ranking': '12',
           'league_points': '1217',
           'lks': '2961',
           'losses': '31',
           'most_played_champions': [{'assists': '238',
                                      'champion_id': '236',
                                      'creep_score': '7227',
                                      'deaths': '131',
                                      'kills': '288',
                                      'losses': '5',
                                      'played': '39',
                                      'wins': '34'},
                                     {'assists': '209',
                                      'champion_id': '429',
                                      'creep_score': '5454',
                                      'deaths': '111',
                                      'kills': '204',
                                      'losses': '3',
                                      'played': '27',
                                      'wins': '24'},
                                     {'assists': '155',
                                      'champion_id': '81',
                                      'creep_score': '4800',
                                      'deaths': '103',
                                      'kills': '168',
                                      'losses': '8',
                                      'played': '26',
                                      'wins': '18'}],
           'name': 'Sadastyczny',
           'previous_ranking': '2',
           'profile_icon_id': 7,
           'ranking': '1',
           'region': 'eune',
           'summoner_id': '42893043',
           'tier': '6',
           'tier_name': 'CHALLENGER',
           'wins': '128'},
          {'division': '1',
           'global_ranking': '30',
           'league_points': '1128',
           'lks': '2956',
           'losses': '180',
           'most_played_champions': [{'assists': '928',
                                      'champion_id': '24',
                                      'creep_score': '37601',
                                      'deaths': '1426',
                                      'kills': '1874',
                                      'losses': '64',
                                      'played': '210',
                                      'wins': '146'},
                                     {'assists': '501',
                                      'champion_id': '67',
                                      'creep_score': '16836',
                                      'deaths': '584',
                                      'kills': '662',
                                      'losses': '37',
                                      'played': '90',
                                      'wins': '53'},
                                     {'assists': '124',
                                      'champion_id': '157',
                                      'creep_score': '5058',
                                      'deaths': '205',
                                      'kills': '141',
                                      'losses': '14',
                                      'played': '28',
                                      'wins': '14'}],
           'name': 'Richor',
           'previous_ranking': '1',
           'profile_icon_id': 577,
           'ranking': '2',
           'region': 'eune',
           'summoner_id': '40385818',
           'tier': '6',
           'tier_name': 'CHALLENGER',
           'wins': '254'},
          {'division': '1',
           'global_ranking': '49',
           'league_points': '1051',
           'lks': '2953',
           'losses': '47',
           'most_played_champions': [{'assists': '638',
                                      'champion_id': '117',
                                      'creep_score': '11927',
                                      'deaths': '99',
                                      'kills': '199',
                                      'losses': '7',
                                      'played': '66',
                                      'wins': '59'},
                                     {'assists': '345',
                                      'champion_id': '48',
                                      'creep_score': '8061',
                                      'deaths': '99',
                                      'kills': '192',
                                      'losses': '11',
                                      'played': '43',
                                      'wins': '32'},
                                     {'assists': '161',
                                      'champion_id': '114',
                                      'creep_score': '5584',
                                      'deaths': '64',
                                      'kills': '165',
                                      'losses': '11',
                                      'played': '31',
                                      'wins': '20'}],
Comments