Nikhil Parmar Nikhil Parmar - 1 month ago 12
Python Question

How to enable overwriting a file everytime in scrapy item export?

I am scraping a website which returns in a list of

urls
.
Example -
scrapy crawl xyz_spider -o urls.csv


It is working absolutely fine now I want is to make new
urls.csv
not append
data
into the file. Is there any parameter passing I can do to make it enable?

Answer

Unfortunately scrapy can't do this at the moment.
There is a proposed enhancement on github though: https://github.com/scrapy/scrapy/issues/547

However you can easily do redirect the output to stdout and redirect that to a file:

scrapy crawl myspider -t json --nolog -o - > output.json

-o - means output to minus and minus in this case means stdout.
You can also make some aliases to delete the file before running scrapy, something like:

alias sc='-rm output.csv && scrapy crawl myspider'
Comments