nanocv nanocv - 1 month ago 16
Perl Question

Perl large download direct to file (to avoid RAM overload)

I'm trying to download a file with 1 GB of aproximated size to a server with 1 GB of RAM memory, so if I try to download it into a variable (code below), the OS kills the process due RAM overload.

require LWP::UserAgent;

my $ua = LWP::UserAgent->new;
$ua->timeout(3600);
$ua->env_proxy;

my $response = $ua->get('http://example.com/largefile.xml');

if ($response->is_success) {
print "File downloaded\n";
}
else {
die $response->status_line;
}


The only way I think could work is using
system("wget ...")
(or curl or something similar) but I'm sure there's a right way to do it directly with Perl.

Do you know any method or option to download it directly to a file without using a system call?

Answer

Yep. Look at LWP::UserAgent's get method:

$ua->get( $url , $field_name => $value, ... )

Fields names that start with ":" are special. These will not initialize headers of the request but will determine how the response content is treated. The following special field names are recognized:

:content_file   => $filename

If a $filename is provided with the :content_file option, then the response content will be saved here instead of in the response object.

Note - it probably won't handle filesystem errors nicely, so check you can write to that location. (permissions, directory exists, etc.)