kimpster kimpster - 19 days ago 4
Python Question

Selenium timeout exception even after loop is executed

I encountered a problem which I do not understand why my code is working this way. Essentially I am attempting to run a for loop for x amount of times but my code keeps saying TimeoutException

Traceback (most recent call last):
File "/Users/Ryan/Desktop/selftest1.py", line 33, in <module>
EC.presence_of_element_located((By.ID, "ctl00_lblStockname"))
File "/Library/Python/2.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
TimeoutException: Message:


And the body of my code is :

for x in range(1,10):

baseurl = 'http://www.hkexnews.hk'
url = 'http://www.hkexnews.hk/listedco/listconews/advancedsearch/search_active_main.aspx'

driver = webdriver.Firefox()
driver.get(url)
driver.find_element_by_id("ctl00_txt_stock_code").clear()
driver.find_element_by_id("ctl00_txt_stock_code").send_keys(x)
driver.find_element_by_id("ctl00_rbAfter2006").click()
Select(driver.find_element_by_id("ctl00_sel_DateOfReleaseFrom_y")).select_by_visible_text("1999")
Select(driver.find_element_by_id("ctl00_sel_tier_1")).select_by_visible_text("Financial Statements/ESG Information")
Select(driver.find_element_by_id("ctl00_sel_tier_2")).select_by_visible_text("Annual Report")
driver.find_element_by_css_selector("label > a > img").click()

match = re.compile('\.(html|pdf)')
try:
element = WebDriverWait(driver, 1).until(
EC.presence_of_element_located((By.ID, "ctl00_lblStockname"))
)
finally:
f = driver.page_source
soup = BeautifulSoup(f,'html.parser')
for link in soup.findAll('a'):
try:
href = link['href']
if re.search(match, href):
file = open("newfile.txt", "a")
file.write(baseurl+href+'\n')
file.close
print ('finished write')
print baseurl+href
except KeyError:
pass
driver.quit()


To my understanding the timeout exception is being thrown because of the first try. But shouldn't the loop stop when it hits finally? Also I attempted to add an exception after 'try' and before 'finally' for timeout error and it gave me the error

error: [Errno 61] Connection refused


I am honestly lost at how to fix this issue or what is causing the issue in the first place.

EDIT:

I added back a exception block after resetting everything and it seems to be working fine now. As in:

try:
...
except TimeoutException:
driver.quit
finally:
...


Just for future references if anyone wants to know the solution.

Answer

I have updated the code,please try this

from selenium import webdriver
from selenium.webdriver.support.ui import Select
import re
from bs4 import BeautifulSoup
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By


for x in range(1,10):

    baseurl = 'http://www.hkexnews.hk'
    url = 'http://www.hkexnews.hk/listedco/listconews/advancedsearch/search_active_main.aspx'

    driver = webdriver.Chrome()
    driver.maximize_window()
    driver.get(url)
    driver.find_element_by_id("ctl00_txt_stock_code").clear()
    driver.find_element_by_id("ctl00_txt_stock_code").send_keys(x)
    driver.find_element_by_id("ctl00_rbAfter2006").click()
    Select(driver.find_element_by_id("ctl00_sel_DateOfReleaseFrom_y")).select_by_visible_text("1999")
    Select(driver.find_element_by_id("ctl00_sel_tier_1")).select_by_visible_text("Financial Statements/ESG Information")
    Select(driver.find_element_by_id("ctl00_sel_tier_2")).select_by_visible_text("Annual Report")
    driver.find_element_by_css_selector("label > a > img").click()

    match = re.compile('\.(html|pdf)')

    wait = WebDriverWait(driver, 10)
    wait.until(EC.presence_of_element_located(
            (By.XPATH, '//*[@id="ctl00_lblStockName"]')))

    f = driver.page_source
    soup = BeautifulSoup(f,'html.parser')
    for link in soup.findAll('a'):
        try:
            href = link['href']
            if re.search(match, href):
                file = open("newfile.txt", "a")
                file.write(baseurl+href+'\n')
                file.close()
                print ('finished write')
                print baseurl+href
        except KeyError:
            pass
    driver.quit()

It is giving this output:

C:\Python27\python.exe 
C:/XXXX/kimpster.py
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2016/0412/LTN20160412398.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2015/0429/LTN201504291354.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2014/0407/LTN20140407336.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2013/0408/LTN20130408921.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2012/0410/LTN20120410623.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2011/0411/LTN20110411707.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2010/0422/LTN20100422489.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2008/0423/LTN20080423279.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0319/LTN20090319103.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0319/LTN20090319097.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2016/0421/LTN20160421233.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2015/0422/LTN20150422417.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2014/0423/LTN20140423340.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2013/0422/LTN20130422293.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2012/0425/LTN20120425287.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2011/0421/LTN20110421583.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2010/0423/LTN20100423265.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0420/LTN20090420355.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2008/0423/LTN20080423322.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2016/0407/LTN20160407581.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2015/0413/LTN20150413273.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2014/0428/LTN20140428711.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2013/0429/LTN20130429395.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2012/0426/LTN20120426622.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2011/0426/LTN20110426450.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2010/0423/LTN20100423393.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0423/LTN20090423238.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2008/0425/LTN20080425250.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2015/0319/LTN20150319329.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2014/0324/LTN20140324959.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2013/0402/LTN201304021122.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2012/0326/LTN20120326263.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2012/0326/LTN20120326253.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0330/LTN20090330188.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0320/LTN20090320083.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2016/0428/LTN201604281016.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2015/0429/LTN20150429233.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2014/0429/LTN20140429945.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2013/0429/LTN201304291031.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2012/0426/LTN20120426229.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2011/0421/LTN20110421266.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2010/0429/LTN20100429830.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0428/LTN200904281430.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2008/0429/LTN20080429728.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2016/0323/LTN20160323343.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2015/0313/LTN20150313356.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2014/0327/LTN20140327637.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2013/0326/LTN20130326368.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2012/0326/LTN20120326620.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2011/0426/LTN20110426261.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2010/0408/LTN20100408709.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0429/LTN20090429932.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2008/0416/LTN20080416269.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2016/0425/LTN20160425745.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2015/0423/LTN20150423635.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2014/0422/LTN20140422239.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2013/0417/LTN20130417330.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2012/0423/LTN20120423313.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2011/0406/LTN20110406041.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2010/0426/LTN20100426737.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0428/LTN20090428495.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2008/0429/LTN20080429825.pdf

Process finished with exit code 0

Is this not what you expect.As its writing this in file also.

Comments