nikonan nikonan - 3 months ago 17
Perl Question

Create a session with necessary cookies using Perl

I will try to give as many details as possible.

Using Perl, I am trying to download the list of Impact Factors of all Journals for the JCR (Journal Citation Reports) from the Web of Knowledge (WoS) (url: It should be noted that the link is only accessible using IPs from most academic institutions, like mine, or from paying users.

Whenever I visit the link (using the browser) and sometime has passed, the session has expired so I need to click on a link ( to establish a new session. Using my rudimentary knowledge of Firebug I found that when I am authenticated I get a cookie named jcrsid with a random string value.

My question is the following:
Using Perl's LWP module, what steps should I take in order to establish a session and download the web pages I need. (If I use LWP to download the page, i download exactly the same page that appears when I am asked to establish a new session using the browser).


You need to set a cookie_jar on your LWP::UserAgent object. LWP::UA will then store all the cookies it encounters in a HTTP::Cookies object and handle them for you. They'll be sent with every request.

You can use an empty hash reference {} to store your cookie in memory. There is no need to use a file for this kind of job.

use strict;
use warnings;
use LWP::UserAgent;

my $ua = LWP::UserAgent->new(
    cookie_jar => {},

That's it. Then you would to get the link that gives you the login cookie first.

my $res_login = $ua->get('');
die $res_login->status_line unless $res_login->is_success;

Afterwards you can go about loading your pages. The cookie will be sent automatically.