RedVelvet RedVelvet - 2 days ago 5
Python Question

Infinite Scroll on Quora using Selenium in Python and Javascript

I try to handle the "infinite scrolling" in Quora website.
I use selenium lib with Python after trying to use the send_keys methods i try to run Javascript command in order to scroll down the page.

It doesn't working when i run the code, but if i try to run in the firefox console it's work.
How can i fix this problem? and it's possibile use PhantomJs?

def scrapying(self):
print platform.system()

browser = webdriver.Firefox()
#browser = webdriver.PhantomJS(executable_path='/usr/local/bin/node_modules/phantomjs/lib/phantom/bin/phantomjs')

browser.get("https://www.quora.com/C-programming-language")
#browser.get("https://answers.yahoo.com/dir/index?sid=396545660")
time.sleep(10)

#elem = browser.find_element_by_class_name("topic_page content contents main_content fixed_header ContentWrapper")
no_of_pagedowns = 500

while no_of_pagedowns:
#elem.send_keys(Keys.SPACE)
browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(0.5)
no_of_pagedowns -= 1

browser.quit()






myClassObject = getFrom()
myClassObject.scrapying()

Answer

One of the options would be to recursively scroll into view of the last loaded post on a page:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://www.quora.com/C-programming-language")

NUM_POSTS = 200
posts = driver.find_elements_by_css_selector("div.pagedlist_item")

while len(posts) < NUM_POSTS:
    driver.execute_script("arguments[0].scrollIntoView();", posts[-1])

    posts = driver.find_elements_by_css_selector("div.pagedlist_item")

    print(len(posts))

And it would scroll the page down until, at least, NUM_POSTS posts are loaded.

Comments