Eric Bellamy Eric Bellamy - 1 year ago 71
Javascript Question

Javascript scraper logging in

I seem to be doing something wrong.

I have a student website that I want to scrape, but first I need to log in. Currently I have a python scraper that does it. The website logs in with a post request to a url containing a sid and PIN.

var login_url = '';

var formData = {
sid: 'username',
PIN: 'password'

How would I go about creating the same scraper but with javascript? I have seen the request library, which seems like what I want to use but cannot get it to work.

Answer Source

You need to use the request module to POST the form data to your endpoint. The response from the server will be in the call back to the .post() method.

const request = require('request');
// do not reassign "request", if you need to set properties us a different variable

// use the action= value from the form for the URL
const url = '';
const data = {
    sid: 'username',
    PIN: 'password',
};{ url: url, formData: data }, (err, response, body) => {
    if (err) {
        console.log('failed', err);
    } else {
        console.log('the response', body);

If you are interesting in parsing the resulting HTML I recommend using CheerioJS - much like jQuery but server side.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download