Ineedsteamhelp Ineedsteamhelp - 1 year ago 82
Python Question

Why does my Scrapy code return an empty array?

I am building a web scraper for, but I my code returns the value of "[]" for inches_rain and humidity. Could anyone see why this is happening?

# -*- coding: utf-8 -*-
import scrapy
from scrapy.selector import Selector
import time

from wunderground_scraper.items import WundergroundScraperItem

class WundergroundComSpider(scrapy.Spider):
name = "wunderground"
allowed_domains = [""]
start_urls = (

def parse(self, response):
info_set = Selector(response).xpath('//div[@id="current"]')
list = []
for i in info_set:
item = WundergroundScraperItem()
item['description'] = i.xpath('div/div/div/div/span/text()').extract()
item['description'] = item['description'][0]
item['humidity'] = i.xpath('div/table/tbody/tr/td/span/span/text()').extract()
item['inches_rain'] = i.xpath('div/table/tbody/tr/td/span/span/text()').extract()
return list

I also know that the humidity and inches_rain items are set to the same xpath, but that should be correct because once the information is in an array I just set them to certain values from the array.

Answer Source

Let me suggest a more reliable and readable XPath to locate, for the sake of an example, "Humidity" value where the base is that "Humidity" column label:


Outputs 45% now.

FYI, your XPath had at least one problem - the tbody tag - remove it from the XPath expression.