wwl wwl - 3 years ago 131
HTML Question

Finding where links are

When I go to this webpage, I see green buttons with the text "信息公开". My task is to download all links of this green button. So if there are ten buttons, I need all ten links.

However I cannot find the text "信息公开" when I download the page in Chrome. I suspect that some Javascript is executed to download information related to "信息公开". Indeed, when I use Chrome to inspect the green buttons, I find information that I cannot find in the HTML files which I download.

How can I find out where the links are?

Answer Source

You have two JavaScript-based options:

a) Use a headless browser like Phantom.js to scrape the site for the links, there should be no problem with the JavaScript-loaded content. This would be the solution if you want to automate the scraping (like running it daily and posting the links somewhere)

b) Much simpler, but not as automatic: Use the jQuery in the Chrome Console to build a selector to get all the links. For example this piece of code, will give you the links of the yellow community box on the right side of Stack Overflow: $('.community-bulletin a').each(function(){console.log($(this).attr('href'))})

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download