Eduardo Perez Eduardo Perez - 7 months ago 29
PHP Question

Batch download URLs in PHP?

So, I have a PHP script that is supposed to download images that the user inputs. However, if the user uploads a TXT file and it contains direct links to images, it should download the images from all the URLs in the file. My script seems to be working, although it seems that only the last file is downloaded while the others are stored as files containing no data.

Here's the portion of my script where it parses the TXT

$contents = file($file_tmp);
$parts = new SplFileObject($file_tmp);
foreach($parts as $line) {
$url = $line;
$dir = "{$save_loc}".basename($url);
$fp = fopen ($destination, 'w+');
$raw = file_get_contents($url);
file_put_contents($dir, $raw);
}


How do I make it download every URL from the TXT file?

Answer

When you iterate over an SplFileObject, you get the whole line, including whitespace. Your URL will thus be something like

http://example.com/_

(php seems to mangle the newline to an underscore) and thus you'll get an error for many URLs (some URLs will still work fine, since they contain the important information prior. For instance, Batch download URLs in PHP? works, but http://stackoverflow.com/_ does not). If an error occurs, file_get_contents will return false, and file_put_contents will interpret that like an empty string.

Also, the line $fp = fopen ($destination, 'w+'); is really strange. For one, since $destination is not defined, it would error anyways. Even if $destination is defined, you'll end up with lots of file handles and overwrite that poor file multiple times. You can just remove it.

To summarize, your code should look like

<?php
$file_tmp = "urls.txt";
$save_loc = "sav/";

$parts = new SplFileObject($file_tmp);
foreach($parts as $line) {
    $url = trim($line);
    if (!$url) {
        continue;
    }
    $dir = "{$save_loc}".basename($url);
    $raw = file_get_contents($url);
    if ($raw === false) {
        echo 'failed to donwload ' . $url . "\n";
        continue;
    }
    file_put_contents($dir, $raw);
}
Comments