Vinay K L Vinay K L - 1 month ago 7
Python Question

Selenium doesn't read the second page

By default page 1 will open. I am clicking on "next page" using

mores.click()
, which is opening properly in the browser. But when I try to read the html code, it is still the first page. How do I make sure that I read the second page.

This is my code:

driver = webdriver.Firefox()
driver.get('https://colleges.niche.com/stanford-university/reviews/')
mores = driver.find_element_by_class_name('icon-arrowright-thin--pagination')
mores.click()
vkl = driver.page_source
print vkl

Answer

You are probably doing it too quick. Add some wait after your click and make sure that the second page is actually appearing on the screen before you try to read the source html.

Keep in mind that Selenium will not automatically wait for the second page to load completely or at all. It will perform the next command: driver.page_source immediately.

Comments