Umair Umair - 1 year ago 96
Python Question

Cookies not showing in Scrapy output even I have enabled it

By default Cookies are enabled in Python Scrapy

I have this in


It works in all other projects and shows cookies in terminal when I run code.

But it is not showing received cookies in terminal for a specific project.

I have searched internet but I am not sure what to do.


The website I am scraping of course sets cookies, I can see cookies when I visit that site from browser

What I can be missing?

Answer Source

From the dicsussions with OP, it appears that this website does not send Set-Cookie headers when using scrapy's default User-Agent string.

Changing the User-Agent string to something like this (in for example):

USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.100 Safari/537.36)'

fixes the issue.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download