Nikhil Parmar Nikhil Parmar - 1 year ago 93
Python Question

How to enable overwriting a file everytime in scrapy item export?

I am scraping a website which returns in a list of

Example -
scrapy crawl xyz_spider -o urls.csv

It is working absolutely fine now I want is to make new
not append
into the file. Is there any parameter passing I can do to make it enable?

Answer Source

Unfortunately scrapy can't do this at the moment.
There is a proposed enhancement on github though:

However you can easily do redirect the output to stdout and redirect that to a file:

scrapy crawl myspider -t json --nolog -o - > output.json

-o - means output to minus and minus in this case means stdout.
You can also make some aliases to delete the file before running scrapy, something like:

alias sc='-rm output.csv && scrapy crawl myspider'
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download