I'm currently working on scanning a folder in my S3 bucket and removing files that are no longer in my database. The problem is that I have millions of files, so no way of scanning this in one go.
// get files
$files = $s3->getIterator('ListObjects', array(
"Bucket" => $S3Bucket,
"Prefix" => 'collections/items/',
"Delimiter" => '/'
'return_prefixes' => true,
'names_only' => true,
'limit' => 10
Remember the last key you processed and use it as the Marker parameter.
$files = $s3->getIterator('ListObjects', array( "Bucket" => "mybucket", "Marker" => "last/key" ));
BTW, dont set Limit, its slowing down. Limit 10 will cause a request to the API every 10 objects, the API can return up to 1000 objects per request.