Mace Mace - 2 months ago 13
Python Question

Following hyperlink and "Filtered offsite request"

I know that there are several related threads out there, and they have helped me a lot, but I still can't get all the way. I am at the point where running the code doesn't result in errors, but I get nothing in my

csv
file. I have the following
Scrapy
spider that starts on one webpage, then follows a hyperlink, and scrapes the linked page:

from scrapy.http import Request
from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.item import Item, Field

class bbrItem(Item):
Year = Field()
AppraisalDate = Field()
PropertyValue = Field()
LandValue = Field()
Usage = Field()
LandSize = Field()
Address = Field()

class spiderBBRTest(BaseSpider):
name = 'spiderBBRTest'
allowed_domains = ["http://boliga.dk"]
start_urls = ['http://www.boliga.dk/bbr/resultater?sort=hus_nr_sort-a,etage-a,side-a&gade=Septembervej&hus_nr=29&ipostnr=2730']

def parse2(self, response):
hxs = HtmlXPathSelector(response)
bbrs2 = hxs.select("id('evaluationControl')/div[2]/div")
bbrs = iter(bbrs2)
next(bbrs)
for bbr in bbrs:
item = bbrItem()
item['Year'] = bbr.select("table/tbody/tr[1]/td[2]/text()").extract()
item['AppraisalDate'] = bbr.select("table/tbody/tr[2]/td[2]/text()").extract()
item['PropertyValue'] = bbr.select("table/tbody/tr[3]/td[2]/text()").extract()
item['LandValue'] = bbr.select("table/tbody/tr[4]/td[2]/text()").extract()
item['Usage'] = bbr.select("table/tbody/tr[5]/td[2]/text()").extract()
item['LandSize'] = bbr.select("table/tbody/tr[6]/td[2]/text()").extract()
item['Address'] = response.meta['address']
yield item

def parse(self, response):
hxs = HtmlXPathSelector(response)
PartUrl = ''.join(hxs.select("id('searchresult')/tr/td[1]/a/@href").extract())
url2 = ''.join(["http://www.boliga.dk", PartUrl])
yield Request(url=url2, meta={'address': hxs.select("id('searchresult')/tr/td[1]/a[@href]/text()").extract()}, callback=self.parse2)


I am trying to export the results to a csv file, but I get nothing the file. Running the code, however, doesn't result in any errors. I know it's a simplyfied example with only one URL, but it illustrates my problem.

I think my problem could be that I am not telling
Scrapy
that I want to save the data in the
Parse2
method.

BTW, I run the spider as
scrapy crawl spiderBBR -o scraped_data.csv -t csv

Answer

You need to modify your yielded Request in parse to use parse2 as its callback.

EDIT: allowed_domains shouldn't include the http prefix eg:

allowed_domains = ["boliga.dk"]

Try that and see if your spider still runs correctly instead of leaving allowed_domains blank

Comments