Solaris_9 Solaris_9 - 1 month ago 13
Bash Question

Scrapy shell doesn't work

I am new to scrapy, then I want to try scrapy shell to debug and learn, but it's strange the shell command doesn't work at all.


  • it seems the website was successfully crawled, but nothing has been printed more. THe program is pending, seems dead, I must use ctrl-c to end it.



can you help to figure out what's wrong?

I'm using Anaconda + scrapy 1.0.3

$ ping 135.251.157.2

Pinging 135.251.157.2 with 32 bytes of data:
Reply from 135.251.157.2: bytes=32 time=13ms TTL=56
Reply from 135.251.157.2: bytes=32 time=14ms TTL=56
Reply from 135.251.157.2: bytes=32 time=14ms TTL=56
Reply from 135.251.157.2: bytes=32 time=14ms TTL=56

Ping statistics for 135.251.157.2:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 13ms, Maximum = 14ms, Average = 13ms


$ scrapy shell "http://135.251.157.2/"
2016-01-28 21:35:18 [scrapy] INFO: Scrapy 1.0.3 started (bot: demo)
2016-01-28 21:35:18 [scrapy] INFO: Optional features available: ssl, http11, boto
2016-01-28 21:35:18 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'demo.spiders', 'SPIDER_MODULES': ['demo.spiders'], 'LOGSTATS_INTERVAL': 0, 'BOT_NAME': 'demo'}
2016-01-28 21:35:18 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
2016-01-28 21:35:19 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, HttpProxyMiddleware, ChunkedTransferMiddleware, DownloaderStats
2016-01-28 21:35:19 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-01-28 21:35:19 [scrapy] INFO: Enabled item pipelines:
2016-01-28 21:35:19 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-01-28 21:35:19 [scrapy] INFO: Spider opened
2016-01-28 21:35:24 [scrapy] DEBUG: Crawled (200) <GET http://135.251.157.2/> (referer: None)
2016-01-28 21:35:24 [root] DEBUG: Using default logger
2016-01-28 21:35:24 [root] DEBUG: Using default logger
ctrl-c

$

Answer

I'd like to close this thread, as I find out the root cause is related with different terminal. When I use Git Bash, it doesn't work, but if I use Anaconda Prompt, it works quite well.

[Anaconda2] D:\SVN\tools\Spider\demo>scrapy shell "http://135.251.157.2/"
2016-01-29 13:40:33 [scrapy] INFO: Scrapy 1.0.3 started (bot: demo)
2016-01-29 13:40:33 [scrapy] INFO: Optional features available: ssl, http11, boto
2016-01-29 13:40:33 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'demo.spiders', 'SPIDER_MODULES': ['demo.spiders'], 'LOGSTATS_INTERVAL': 0, 'BOT_NAME': 'demo'}
2016-01-29 13:40:33 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
2016-01-29 13:40:33 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, HttpProxyMiddleware, ChunkedTrans
ferMiddleware, DownloaderStats
2016-01-29 13:40:33 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-01-29 13:40:33 [scrapy] INFO: Enabled item pipelines:
2016-01-29 13:40:33 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-01-29 13:40:33 [scrapy] INFO: Spider opened
2016-01-29 13:40:41 [scrapy] DEBUG: Crawled (200) <GET http://135.251.157.2/> (referer: None)
[s] Available Scrapy objects:
[s]   crawler    <scrapy.crawler.Crawler object at 0x0136B290>
[s]   item       {}
[s]   request    <GET http://135.251.157.2/>
[s]   response   <200 http://135.251.157.2/>
[s]   settings   <scrapy.settings.Settings object at 0x034204B0>
[s]   spider     <DefaultSpider 'default' at 0x3e3c6d0>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects
[s]   view(response)    View response in a browser
2016-01-29 13:40:41 [root] DEBUG: Using default logger
2016-01-29 13:40:41 [root] DEBUG: Using default logger

In [1]: