Norbert Norbert - 3 months ago 35
PHP Question

Any way to offset results with AWS SDK for PHP?

I'm currently working on scanning a folder in my S3 bucket and removing files that are no longer in my database. The problem is that I have millions of files, so no way of scanning this in one go.

// get files
$files = $s3->getIterator('ListObjects', array(
"Bucket" => $S3Bucket,
"Prefix" => 'collections/items/',
"Delimiter" => '/'
), array(
'return_prefixes' => true,
'names_only' => true,
'limit' => 10

The documentation included something about limiting results, but I can't find anything about offsetting. I want to be able to start from 0, scan 500 items, remove them, stop, save the last scanned index and then run the script again, start from the saved index (501), scan 500 items, and so on.

Does the SDK offer some sort of offset option? Is it called something else? Or can you recommend a different method of scanning such a large folder?


Remember the last key you processed and use it as the Marker parameter.

$files = $s3->getIterator('ListObjects', array(
  "Bucket" => "mybucket",
  "Marker" => "last/key"

BTW, dont set Limit, its slowing down. Limit 10 will cause a request to the API every 10 objects, the API can return up to 1000 objects per request.