Gesh Gesh - 1 month ago 14
PHP Question

Azure SDK for php blob download causes out of memory

I'm using the Azure blob storage to store huge pdf and zip files in the cloud.
I'm accessing the files through the azure sdk for php and pushing the files directly to the user (of course the user should not see where the file is coming from, so I don't redirect him to a Microsoft url).
My code looks like this:

$blobRestProxy = ServicesBuilder::getInstance()->createBlobService($this->azureConfig['connectionString']);

$blob = $blobRestProxy->getBlob($container, $blobName);
$properties = $blobRestProxy->getBlobProperties($container, $blobName);

$size = $properties->getProperties()->getContentLength();
$mime = $properties->getProperties()->getContentType();
$stream = $blob->getContentStream();

header("Pragma:no-cache");
header("Cache-Control: no-cache, must-revalidate");
header("Content-type: $mime");
header("Content-length: $size");

fpassthru($stream);


For small files, no problem at all, for bigger files, I get this error:

Fatal error: Out of memory (allocated 93323264) (tried to allocate 254826985 bytes) in \vendors\azure-sdk-for-php\WindowsAzure\Common\Internal\Utilities.php on line 450


Is there a better way to provide cloud stored files to users with php without them recognizing it?

I already found this discussion https://github.com/Azure/azure-sdk-for-php/issues/729, but the curl solution just don't work.

Thank you!

Best
Gesh

Answer

Per my understanding, due to download the bigger file, the program consumes more memory. In this scenario, we can take those actions to overcome memory exhausted error:

  1. Set the memory_limit value.

There is a PHP env configuration named memory_limit to limit the memory which can be allocated. We can set the memory_limit value in php page using the code:

ini_set("memory_limit","200M");

If you would not like to set the file size, you can set the memory_limit value as "-1", just like:

ini_set("memory_limit","-1");

Another approach, we can enlarge this in configuration file (like php.ini). This official guide tells you how to configure PHP environment.

  1. Download file in chucks

We also can download large blob in chucks to reduce the memory overhead.

To see SDK source code in BlobRestProxy.php, there is a function used to get Blob public function getBlob($container, $blob, $options = null), we can set additional param into $options to get the Blob in pieces every time. Here is my code snippet:

$properties = $blobRestProxy->getBlobProperties($container, $blobName);
$size = $properties->getProperties()->getContentLength();
$mime = $properties->getProperties()->getContentType();
$chunk_size = 1024 * 1024;
$index = 0;
//$stream = "";

header("Pragma: public");
header('Content-Disposition: attachment; filename="' . $blobName . '"');
header('Expires: 0');
header("Cache-Control: must-revalidate, post-check=0, pre-check=0");
header("Content-Transfer-Encoding: binary");
header("Content-type: $mime");
header("Content-length: $size");
ob_clean();

while ($index < $size) {
       $option = new GetBlobOptions();
       $option->setRangeStart($index);
       $option->setRangeEnd($index + $chunk_size - 1);
       $blob = $blobRestProxy->getBlob($container, $blobName, $option);
       $stream_t = $blob->getContentStream();
       $length = $blob->getProperties()->getContentLength();
       $index += $length;
       flush();
       fpassthru($stream_t);
}

Any concerns, please feel free to let me know.