shiva krishna shiva krishna - 2 years ago 320
Python Question

error in deploying a project using scrapyd

I had multiple spiders in my project folder and want to run all the spiders at once, so i decided to run them using scrapyd service.
I have started doing this by seeing here

First of all i am in current project folder

  1. I had opened the
    file and uncommented the url line after


  2. I had run
    scrapy server
    command, that works fine and scrapyd server runs

  3. I tried this command
    scrapy deploy -l

    Result :
    default http://localhost:6800/

  4. when i tried this command
    scrapy deploy -L scrapyd
    i got following output


scrapy deploy [options] [ [target] | -l | -L <target> ]

deploy: error: Unknown target: scrapyd

when i tried to deploy the project with this command
scrapy deploy scrapyd -p default
got following error

scrapy deploy [options] [ [target] | -l | -L <target> ]

deploy: error: Unknown target: scrapyd

I am really unable to identify whey scrapyd is showing the above errors, can lead me in to a correct way of how to deploy a project in to scrapyd

Thanks in advance..........

Edited Code:

After seeing the answer of Peter Kirby,i named target in scrapy.cfg and tried the following command in my project folder,


scrapy deploy ebsite -p ebsite

then i got the below error

Building egg of ebsite-1341808241
'build/lib' does not exist -- can't clean it
'build/bdist.linux-x86_64' does not exist -- can't clean it
'build/scripts-2.7' does not exist -- can't clean it
zip_safe flag not set; analyzing archive contents...
Deploying ebsite-1341808241 to http://localhost:6800/addversion.json
Deploy failed: <urlopen error [Errno 111] Connection refused>

How to solve this.....

Answer Source

From scrapyd service documentation: (

You can define targets by adding them to your project’s scrapy.cfg file... Here’s an example of defining a new target scrapyd2 with restricted access through HTTP basic authentication:

url =
username = john
password = secret

Essentially what your error means is that your "target" name is not correct. If I remember correctly, the scrapy.cfg file sets the initial target name as "default". What you should be typing is something like:

scrapy deploy default -p project_name
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download