Sergey Bakotin Sergey Bakotin - 22 days ago 10
Javascript Question

How can i execute JavaScript in selenium and grab data?

Hello and thank you for your time.

My purpose is to make parser, that follow the hundreds of sites and check if site has module (special plugin) or not. The main problem, that the only way is to make sure, it a a lunch JavaScript for site.

enter image description here

So, i don't know how to lunch a javascript and grap data

For example:

driver = webdriver.Chrome('D:\chromedriver_win32\chromedriver')
driver.get("https://lieman.com/")
tag = driver.execute_script("document.getElementsByClassName('arsenal_title')")
driver.close()
print(tag)


tag prints None.

Can someone helps me to receive data or tell me to other way to check javascipt

===

def main():
driver = webdriver.Chrome('D:\chromedriver_win32\chromedriver')
driver.get("http://redhelper.ru/")
morn = driver.execute_script("return redhlpSettings()")
driver.close()
print(morn)

if __name__ == '__main__':
main()

Answer

You don't need to execute javascript, use find_elements_by_class_name(class_name) or find_element_by_class_name(class_name) instead:

tags = driver.find_elements_by_class_name('arsenal_title')

OR

tag = driver.find_element_by_class_name('arsenal_title')

If you need to use execute_script for other reason, and want to get the value of the expression, you need to return in javascript side:

tags = driver.execute_script("return document.getElementsByClassName('arsenal_title')")

UPDATE

Replace the following line:

morn = driver.execute_script("return redhlpSettings()")

with:

morn = driver.execute_script("return redhlpSettings")

because redhlpSettings is not a function. You will see an object in javascript is convertd to a dictionary object in python.

Comments