Vipul Vipul - 7 months ago 9
Python Question

Python, Selenium : 'Element is no longer attached to the DOM'

I am scraping a website, www.lipperleaders.com. I want to extract the funds detail of Singapore. I have successfully implemented drop-down selection and extracted the content of the first page appearing after submission of the options. But when I try to go to next pages (by making the code to click next button) I am getting error

'Element is no longer attached to the DOM'
.

My code is about 100 lines but I can give a general idea of the flow of execution of my code:

... # creating driver object and all the imports
def main():
...
result = find_elements_by_tag_name('span')
...
driver.find_element_by_id("ctl00_ContentPlaceHolder1_ucDataPager_btnNext").click()
main()
main()


This code works fine for the 1st page but when
main()
is called again after clicking of the next button. Before this recursive method, I also tried putting this inside a loop, then also same error.

And if I write the same code like:

# some code
result = find_elements_by_tag_name('span')
driver.find_element_by_id("ctl00_ContentPlaceHolder1_ucDataPager_btnNext").click()
# some code
driver.find_element_by_id("ctl00_ContentPlaceHolder1_ucDataPager_btnNext").click()
.
.


This code works fine w/o any error the next page loads and executes the code written after that. But I cannot write the same
driver.find_element_by_id().click()
for 500 pages, even I will have to repeat the rest of the code associated with each page. That's why I am trying for loop or recursion, but its not working for me.

Please let me know what is the problem with my approach.

Answer

The problem is that the element is being detached by some javascript. So you should make the driver wait for the element: This is done by setting implicitly_wait, see:

from selenium import webdriver

ff = webdriver.Firefox()
ff.implicitly_wait(10) # seconds
...
myDynamicElement = ff.find_element_by_id("myDynamicElement")

from http://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp#implicit-waits