Phillip Phillip -4 years ago 110
Python Question

How can Selenium (or BeautifulSoup) be used to access these hidden elements?

Here is an example page with pagination controlling dynamically loaded results.

http://www.rehabs.com/local/jacksonville-fl/

All that I presently know to try is:

curButton = 1
driver.find_element_by_css_selector('ul[class="pagination"]').find_elements_by_tag_name('li')[curButton].click()


Nothing seems to happen (also when trying to access and click the
a
tag or driver.get() the href of the
a
element).

Is there another way to access the hidden elements? For instance, when reading the html of the entire page, the elements of different pagination are shown, but are apparently inaccessible with BeautifulSoup.

Answer Source

All articles on the page. Pagination was added for humans. Maybe you use wrong xpath or css. Check it.

I found 40 articles by Scrapy. So, BeautifulSoup also will find it.

Use this xpath:

//div[@id="listing-basic"]/article/div[@class="h3"]/a/@href
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download