Lord G. Lord G. - 9 months ago 39
Javascript Question

Scrapy splash log-in

I used to use splash form request to login in one of the site. However, developers changed it, added more javascript, and I can't figure out what I'm doing wrong. I added javascript, which beeing used in that site as well.

class MySpider(scrapy.Spider):
name = "lost"
start_urls = ["mysite",] ###########changed main loggin form

def start_requests(self):

for url in self.start_urls:
yield SplashRequest(
url,
self.parse,
args={'wait': 1},
)

def parse(self, response):
return SplashFormRequest.from_response(
response,
formdata={'mail': 'mymail', 'pass': 'mypasswd'},
callback=self.after_login
)


def after_login(self,response):
print('This is body '+response.body+' The end of body')
### Going to film list ######
if "Username" in response.body:
self.logger.error("##Success##")


Javascript:

$(document).ready(function(){
$('input[name="mail"],input[name="pass"]').keydown(function (e)
{
if(e.keyCode == 13)
{
login();
}
});
});
function login()
{
mail = $('input[name="mail"]').val();
pass = $('input[name="pass"]').val();
if($('input[name="rem"]:checked').length)
rem = 1;
else
rem = 0;
if(mail.length && pass.length > 5)
{
metrikaEvents('LOGIN');
console.log('OK!');
$.ajax({
type: "POST",
url: "/ajaxik.php",
dataType : "json",
data:
{
act:'users',
type:'login',
mail:encodeURIComponent(mail),
pass:encodeURIComponent(pass),
rem:encodeURIComponent(rem)
},
success: function(msg)
{
if(msg.result == 'ok')
{
if(msg.error)
{
switch(msg.error)
{
default:
text = lf_config.errors.user.login_error;
break;
}
ntfctn(text,'error');
}
else if(msg.success)
{
ntfctn(msg.name+lf_config.notifications['user_login'],'information');
setTimeout('goTo("/",false)',1000);
// goTo('/',false);
}
}
},
});
}
}
function loginTogglePass(t)
{
if($('input[name="'+t+'"]').attr('type') == 'password')
{
$('input[name="'+t+'"]').attr('type','text');
$('input[name="'+t+'"]').prev('div.eye- icon').removeClass('closed').addClass('opened');
}
else
{
$('input[name="'+t+'"]').attr('type','password');
$('input[name="'+t+'"]').prev('div.eye-icon').removeClass('opened').addClass('closed');
}
}


I see that javascript looking for "enter" key down. But clicking button should work as well. Can anyone put me in the right direction? Thanks

Answer Source

Borrowed idea from this topic

enter Scrapy + splash: can't select element

Instead using formdata to login, use splash and detect page's elements one by one

class MySpider(scrapy.Spider):
name = "lost"
allowed_domains = ["mydomain"]
start_urls = ['myurl']
req = 10
series = {}





def start_requests(self):                               
    script = """
    function main(splash)
        local url = splash.args.url
        assert(splash:go(url))
        assert(splash:wait(10))

        splash:set_viewport_full()

        local search_input = splash:select('input[name=mail]')   
        search_input:send_text("email")
        local search_input = splash:select('input[name=pass]')
        search_input:send_text("password")
        assert(splash:wait(5))
        local submit_button = splash:select('input[class^=primary-btn]')
        submit_button:click()

        assert(splash:wait(10))

        return {
            html = splash:html(),
            png = splash:png(),
        }
      end
    """
    yield SplashRequest(
        'myurl',
        callback = self.after_login,      ###inserting callabck
        endpoint='execute',
        args={
        'lua_source': script,
        'ua': "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36"
        }
                        )

def parse(self, response):
    script = response.body