Umair Umair - 21 days ago 7
Python Question

Cookies not showing in Scrapy output even I have enabled it

By default Cookies are enabled in Python Scrapy

I have this in settings.py

COOKIES_DEBUG = True


It works in all other projects and shows cookies in terminal when I run code.

But it is not showing received cookies in terminal for a specific project.

I have searched internet but I am not sure what to do.

PS:

The website I am scraping of course sets cookies, I can see cookies when I visit that site from browser

What I can be missing?

Answer

From the dicsussions with OP, it appears that this website does not send Set-Cookie headers when using scrapy's default User-Agent string.

Changing the User-Agent string to something like this (in settings.py for example):

USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.100 Safari/537.36)'

fixes the issue.