Shantanu Bedajna Shantanu Bedajna - 2 months ago 23
Python Question

Python Selenium not working inside scrapy

I have a script that scrapes ebay products but in the product page i need to select a dropdown input a zipcode and click get details button so i used selenium

but for some reason selenium is not working inside scrapy selenium worked fine when i tested the code in python interpreter

here is the example
enter image description here
enter image description here

the code works fine in practice
but when running scrapy

it does not select the drop down and skips over to the next request

enter image description here

SCRAPY OUTPUT send keys not working because dropdown is not selected when running scrapy AND UNTIL DROPDOWN FIELD IS SELECTED THE ZIPCODE FIELD WILL NOT APPEAR



enter image description here

here is the code

# -*- coding: utf-8 -*-
import scrapy, time
from urlparse import urljoin
from selenium import webdriver


class EbayspiderSpider(scrapy.Spider):
name = "ebayspider"
#allowed_domains = ["www.ebay.com"]
start_urls = ['http://stores.ebay.com/Complete-Tractor/Other-/_i.html?_fsub=1&_sid=1159087983&_trksid=p4634.c0.m322']

def parse(self, response):
#INITIALLY STARTING THE WEB BROWSER
self.driver = webdriver.Chrome()
SET_SELECTOR = '.li.nol'
for attr in response.css(SET_SELECTOR):
linkse = '.v4lnk ::attr(href)'
link = attr.css(linkse).extract_first()
yield scrapy.Request(urljoin(response.url, link), callback=self.parse_link)

next_page = 'td.next a ::attr(href)'
next_page = response.css(next_page).extract_first()
if next_page:
yield scrapy.Request(urljoin(response.url, next_page), callback=self.parse)

def parse_link(self, response):

#GETTING THE URL
self.driver.get(response.url)
#CLICKING on the show button to show the form
self.driver.find_element_by_id('viTabs_1').click()
#SELECTING THE DROPDOWN
el = self.driver.find_element_by_id('shCountry')
for option in el.find_elements_by_tag_name('option'):
if option.text == 'United States':
option.click()
break
#FILLING IN THE ZIPCODE FIELD
el = self.driver.find_element_by_id('shZipCode')
el.send_keys("4883")
#CLICKING THE GET BUTTON
self.driver.find_element_by_name("getRates").click()
price1 = self.driver.find_element_by_xpath('//*[@id="shippingSection"]/table/tbody/tr[2]/td[1]/div').text
price2 = self.driver.find_element_by_xpath('//*[@id="shippingSection"]/table/tbody/tr[2]/td[2]/div').text

try:
print '\n\t', price1, '\t', price2
except:
pass


SO what is happening here why is it working in interpreter but not inside scrapy ?

i do not see what is wrong, please point me to the right direction

EDIT:

HYML CODE FOR THE DROPDOWN



enter image description here
and
enter image description here

code

<tr>
<td width="53%" nowrap="nowrap">
<div id="shQuantityDiv" class="sh-InlCnt">
<div id="shQuantity-errIcn" class="sh-err-icon sh-err-hide"></div>
<label class="sh-ShipDtl" for="shQuantity">Quantity:</label>
<input type="text" class="sh-TxtCnt" value="1" size="5" name="quantity" id="shQuantity">
</div>
<div id="shCountryDiv" class="sh-InlCnt sh-dropDownPadLeft active sel">
<div id="shCountry-errIcn" class="sh-err-icon sh-err-hide"></div>
<label class="sh-ShipDtl" for="shCountry">Change country:</label>
<select name="country" id="shCountry" class="sh-TxtCnt sh-InlCnt">
<option value="-99" selected="">-Select-</option>
<option value="7">American Samoa</option>
<option value="10">Anguilla</option>
<option value="11">Antigua and Barbuda</option>
<option value="14">Aruba</option>
<option value="15">Australia</option>
<option value="16">Austria</option>
<option value="18">Bahamas</option>
<option value="21">Barbados</option>
<option value="23">Belgium</option>
<option value="24">Belize</option>
<option value="32">British Virgin Islands</option>
<option value="2">Canada</option>
<option value="41">Cayman Islands</option>
<option value="50">Cook Islands</option>
<option value="51">Costa Rica</option>
<option value="55">Cyprus</option>
<option value="57">Denmark</option>
<option value="60">Dominican Republic</option>
<option value="69">Fiji</option>
<option value="70">Finland</option>
<option value="71">France</option>
<option value="73">French Polynesia</option>
<option value="77">Germany</option>
<option value="79">Gibraltar</option>
<option value="82">Grenada</option>
<option value="84">Guam</option>
<option value="94">Iceland</option>
<option value="99">Ireland</option>
<option value="101">Italy</option>
<option value="102">Jamaica</option>
<option value="104">Japan</option>
<option value="105">Jersey</option>
<option value="109">Kiribati</option>
<option value="130">Malta</option>
<option value="131">Marshall Islands</option>
<option value="132">Martinique</option>
<option value="226">Micronesia</option>
<option value="144">Nauru</option>
<option value="146">Netherlands</option>
<option value="147">Netherlands Antilles</option>
<option value="148">New Caledonia</option>
<option value="149">New Zealand</option>
<option value="153">Niue</option>
<option value="154">Norway</option>
<option value="157">Palau</option>
<option value="158">Panama</option>
<option value="159">Papua New Guinea</option>
<option value="164">Portugal</option>
<option value="165">Puerto Rico</option>
<option value="183">Solomon Islands</option>
<option value="186">Spain</option>
<option value="192">Sweden</option>
<option value="193">Switzerland</option>
<option value="201">Tonga</option>
<option value="206">Turks and Caicos Islands</option>
<option value="207">Tuvalu</option>
<option value="3">United Kingdom</option>
<option value="1" selected="">United States</option>
<option value="213">Vanuatu</option>
<option value="217">Virgin Islands (U.S.)</option>
<option value="218">Wallis and Futuna</option>
<option value="220">Western Samoa</option>
</select>
</div>
<div aria-live="assertive" role="alert" id="shQuantity-errTxt" class="sh-err-text sh-err-hide">There are 15 items available. Please enter a number less than or equal to 15.</div>
<div aria-live="assertive" role="alert" id="shCountry-errTxt" class="sh-err-text sh-err-hide">Select a valid country.</div>
</td>
<td width="47%">
<div aria-live="assertive" role="alert" class="sh-InlCnt sh-float-l" id="shZipCodeDiv" style="display: block;">
<div id="shZipCode-errIcn" class="sh-err-icon sh-err-hide"></div>
<label class="sh-ShipDtl" for="shZipCode" id="shZipCodeTextDiv">ZIP Code:</label>
<div class="sh-ZipAln sh-InlCnt">
<input type="text" class="sh-TxtCnt" name="zipCode" size="12" id="shZipCode" value="4883">
</div>
<div id="shZipCode-errTxt" class="sh-err-text sh-err-hide">Please enter a valid ZIP Code.</div>
<div id="shZipCode-errTxt2" class="sh-err-text sh-err-hide">Please enter 5 or 9 numbers for the ZIP Code.</div>
</div>
<div aria-live="assertive" role="alert" id="shGetRatesDiv" class="sh-InlCnt active sel" style="display: inline;">
<input type="button" class="sh-BtnTxt btn btn-s btn-ter" name="getRates" id="shGetRates" value="Get Rates">
</div>
</td>
</tr>

Answer

It seems that you're trying to send value to input field that is not currently visible. When you're using Python shell to execute commands, you works with already rendered page, so required input field is visible and everything works fine, but your script works differently... You can try to wait some time until it become visible as below:

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as wait


element = wait(self.driver, 10).until(EC.presence_of_element_located((By.ID, "shCountry")))
select = Select(element)
select.select_by_visible_text("United States")
wait(self.driver, 10).until(EC.visibility_of_element_located((By.ID, "shZipCode"))).send_keys("4883")