I have a script that runs multiple instances of Python Scrapy crawlers, Crawlers are int
from scrapy.settings import Settings
from scrapy.utils.project import get_project_settings
from amazon_crawler.spiders.amazon_scraper import MySpider
process = CrawlerProcess(get_project_settings())
alias batchscript="cd /root/crawler/"
When I runinside
But when I run it from outside of this directory usingthen it does not run as intended,
(Settings are not imported correctly),are
using bash, you could always do:
cd /root/crawler && python batchscript.py
it's always good policy to use absolute paths to programs/executables referenced in cron jobs.