nevster nevster - 1 year ago 98
Python Question

Activating a button within a Python spider program

I am absolutely stuck on this one. I am scraping restaurant URLs from a webpage and there is a button at the bottom to reveal more restaurants. The website button code is below (i believe):

<div id="restsPages">
<a class="next" data-url="">Show more</a>
<a class="back">Back to top</a>

It is the "Show more" button i am trying to activate. The url within the "data-url" does not reveal more of the page.

It all seems a bit odd on what do do to activate the button from within the python spider?

The code i am trying to use to make this work is:

import scrapy

from hungryhouse.items import HungryhouseItem
from selenium import webdriver

class HungryhouseSpider(scrapy.Spider):
name = "hungryhouse"
allowed_domains = [""]
start_urls = ["",
def __init__(self):
self.driver = webdriver.Chrome()

def parse(self,response):

while True:
next =self.driver.find_element_by_xpath('//*[@id="restsPages"]/a[@class="next"]')

.... rest of the code follows

The error i get is: 'chromedriver' executable needs to be in PATH

Answer Source

This was resolved at Pressing a button within python code with reference to the answer at Error message: "'chromedriver' executable needs to be available in the path"

But specifically

self.driver = webdriver.Chrome()

needed to change to

self.driver = webdriver.Chrome("C:/Users/andrew/Downloads/chromedriver_win32/chromedriver.exe")

in my case.

i.e. I needed to add the path to chromedriver.exe.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download