Luis Ramon Ramirez Rodriguez Luis Ramon Ramirez Rodriguez - 3 months ago 33x
Python Question

Download a full page with scrapy

I want to download the content a whole page using scrapy.

With selenium this is quite easy:

import os,sys
from selenium import webdriver

url = ''

driver = webdriver.Firefox()
content = driver.page_source
with open('source','w') as output:

But selenium is much slower than scrapy.

Is it an simple way to do in scrapy?

I want to save the code of each page in a different file text, not as a csv or json file. And if posible without creating a project, that seems a bit of overkill for such a simple task


Code will download this page and save it in file download-a-full-page-with-scrapy.html

import scrapy
class TestSpider(scrapy.Spider):
    name = "test"

    start_urls = [

    def parse(self, response):
        filename = response.url.split("/")[-1] + '.html'
        with open(filename, 'wb') as f:

run scrapy by this command

scrapy runspider