nevster nevster - 14 days ago 8
Python Question

Activating a button within a Python spider program

I am absolutely stuck on this one. I am scraping restaurant URLs from a webpage and there is a button at the bottom to reveal more restaurants. The website button code is below (i believe):

<div id="restsPages">
<a class="next" data-url="https://hungryhouse.co.uk/takeaways/aberdeen-bridge-of-dee-ab10">Show more</a>
<a class="back">Back to top</a>
</div>


It is the "Show more" button i am trying to activate. The url within the "data-url" does not reveal more of the page.

It all seems a bit odd on what do do to activate the button from within the python spider?

The code i am trying to use to make this work is:

import scrapy

from hungryhouse.items import HungryhouseItem
from selenium import webdriver

class HungryhouseSpider(scrapy.Spider):
name = "hungryhouse"
allowed_domains = ["hungryhouse.co.uk"]
start_urls = ["https://hungryhouse.co.uk/takeaways/westhill-ab10",
]
def __init__(self):
self.driver = webdriver.Chrome()

def parse(self,response):
self.driver.get(response.url)

while True:
next =self.driver.find_element_by_xpath('//*[@id="restsPages"]/a[@class="next"]')
try:
next.click()
except:
break
self.driver.close()

.... rest of the code follows


The error i get is: 'chromedriver' executable needs to be in PATH

Answer

This was resolved at Pressing a button within python code with reference to the answer at Error message: "'chromedriver' executable needs to be available in the path"

But specifically

self.driver = webdriver.Chrome()

needed to change to

self.driver = webdriver.Chrome("C:/Users/andrew/Downloads/chromedriver_win32/chromedriver.exe")

in my case.

i.e. I needed to add the path to chromedriver.exe.

Comments