micmic micmic - 1 month ago 17
Python Question

Python simples3 listdir limit the output to 1000 files?

I am using simples3 version 1.0. When calling listdir on s3 key that has a lot more than 1000 files, the generator return only the first 1000 files.

The method has an optional limit argument which is None by default. Setting the limit return an empty list. And there is no additional limit to listdir mentioned in the documentation.

Here is a code example, where bucket is simples3 S3Bucket and s3_key is a key in s3 that include over 5000 files under it:

count = 0
for file_info in bucket.listdir(prefix=s3_key):
count += 1

>>> count
1000 ## Expected to be over 5000


Is there a limitation that is not documented?
Is there a way with simples3 lib to get all the files under the s3 key?

Answer

The simples3 library is using aws's rest api, which by default limits the max keys to 1000.

It's not documented, but based on the code can use set the limit using the limit key:

for file_info in bucket.listdir(prefix=s3_key, limit=10000):

I didn't find any way to set it to unlimited.

Comments