Adam F Adam F - 1 year ago 139
Python Question

How do I use the Python Scrapy module to list all the URLs from my website?

I want to use the Python Scrapy module to scrape all the URLs from my website and write the list to a file. I looked in the examples but didn't see any simple example to do this.

Answer Source

Here's the python program that worked for me:

from scrapy.selector import HtmlXPathSelector
from scrapy.spider import BaseSpider
from scrapy.http import Request

URL = 'http://%s' % DOMAIN

class MySpider(BaseSpider):
    name = DOMAIN
    allowed_domains = [DOMAIN]
    start_urls = [

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        for url in'//a/@href').extract():
            if not ( url.startswith('http://') or url.startswith('https://') ):
                url= URL + url 
            print url
            yield Request(url, callback=self.parse)

Save this in a file called

You can then use a shell pipeline to post process this text:

bash$ scrapy runspider > urls.out
bash$ cat urls.out| grep '' |sort |uniq |grep -v '#' |grep -v 'mailto' > example.urls

This gives me a list of all the unique urls in my site.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download