Benjamin James Benjamin James - 10 months ago 56
Python Question

Python selenium click element while displayed

I'm scraping a dynamic page that requires the user to click a "Load More Results" button several times in order to get all data. Is there a better way to approach the task of clicking on an element while displayed?

def clickon(xpath):
element = driver.find_element_by_xpath(xpath)
print "RETRYING CLICKON() for %s" % (xpath)

def click_element_while_displayed(xpath):
element = driver.find_element_by_xpath(xpath)
while element.is_displayed():

Answer Source

I suspect you are asking this question because the current solution is slow. This is mostly because you have these hardcoded time.sleep() delays which wait for more than they usually should. To tackle this problem I'd start using Explicit Waits - initialize an endless loop and break it once selenium stopped waiting for the button to be clickable:

from import By
from import WebDriverWait
from import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

wait = WebDriverWait(driver, 10)

while True:
       element = wait.until(EC.element_to_be_clickable((By.XPATH, xpath)))
   except TimeoutException:
       break  # cannot click the button anymore

   # TODO: wait for the results of the click action

Now the last TODO part is also important - here I suggest you wait for a specific condition that would indicate that click resulted into something on a page - for instance, more results were loaded. For example, you can use a custom Expected Condition similar to this one.