abhishake abhishake - 8 days ago 8
Python Question

Scrapy and github login

I am new to scrapy, and I've searched a lot, but I can't find any working example that can show how to do login with scrapy.

I've prepared this github login code to understand the flow pattern, but seems like it is not working ...

Can anyone show me how can to do this?

Thanks in advance.

import scrapy
from scrapy.spider import BaseSpider
from scrapy.http import FormRequest

class SpiderOne(BaseSpider):
name = "spider"
allowed_domains = ["github.com"]
start_urls = ["https://github.com/login"]

def parse(self, response):
print "in parse function"
return [FormRequest.from_response(
response,
url='https://github.com/session',
method="POST",
formdata={
'login':'valid_email',
'password':'valid_password'
},
callback=self.after_login()
)]

def after_login(self, response):

print "in after_login function"
print response

Answer

You shall try like this

def parse(self, response):
  print "in parse function"
  yield FormRequest.from_response(
     response,
     url='https://github.com/session',
     method="POST",
     formdata={
        'login':'valid_email',
        'password':'valid_password'
     },
     callback=self.after_login)
Comments