khpeek khpeek - 1 year ago 145
Python Question

In scrapy, 'start_urls' not defined when passed as an input argument

The following spider with fixed


import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class NumberOfPagesSpider(CrawlSpider):
name = "number_of_pages"
allowed_domains = [""]

# def __init__(self, place='amsterdam'):
# self.start_urls = ["" % place]

start_urls = [""]

le_maxpage = LinkExtractor(allow=r'%s+p\d+' % start_urls[0])

rules = (Rule(le_maxpage, callback='get_max_page_number'),)

def get_max_page_number(self, response):
links = self.le_maxpage.extract_links(response)
max_page_number = 0 # Initialize the maximum page number
for link in links:
if link.url.count('/') == 6 and link.url.endswith('/'): # Select only pages with a link depth of 3
page_number = int(link.url.split("/")[-2].strip('p')) # For example, get the number 10 out of the string ''
if page_number > max_page_number:
max_page_number = page_number # Update the maximum page number if the current value is larger than its previous value
filename = "max_pages.txt" # File name with as prefix the place name
with open(filename,'wb') as f:
f.write('max_page_number = %s' % max_page_number) # Write the maximum page number to a text file

If I run it by
scrapy crawl number_of_pages
, it writes a .txt file as expected. However, if I modify it by commenting in the
def __init__
lines and commenting out the
start_urls =
line, and try to run it with a user-defined input argument,

scrapy crawl number_of_pages -a place=amsterdam

I get the following error:

le_maxpage = LinkExtractor(allow=r'%s+p\d+' % start_urls[0])
NameError: name 'start_urls' is not defined

So according to the spider,
is not defined, even though in the code it is fully determined in the initialization. How can I get this spider to work with
defined by an input argument?

Answer Source

Your le_maxpage is a class level variable. When you pass the argument to __init__, you're creating an instance level variable start_urls.

You used start_urls in le_maxpage, so for the le_maxpage variable to work, there needs to be a class level variable named start_urls.

To fix this issue, you need to move your class level variables to instance level, that is to define them inside the __init__ block.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download