Lingbo Tang Lingbo Tang - 1 year ago 241
Node.js Question

AWS Lambda nodejs: Iteratively invoke functions when files in S3 bucket changed

I created a micro service on lambda using nodejs to generate thumbnails of my images in S3 bucket. However, it didn't get triggered after I uploaded new images to the S3 bucket. I set the trigger event type as S3 object created. And I also configured my test event as:

"Records": [
"s3": {
"configurationId": "testConfigRule",
"object": {
"eTag": "0123456789abcdef0123456789abcdef",
"sequencer": "0A1B2C3D4E5F678901",
"key": "HappyFace.jpg",
"size": 1024
"bucket": {
"arn": "arn:aws:s3:::myS3bucket",
"name": "myS3bucket",
"ownerIdentity": {
"principalId": "EXAMPLE"
"s3SchemaVersion": "1.0"
"awsRegion": "us-east-1",
"eventName": "ObjectCreated:Put",
"eventSource": "aws:s3"

which means when some files are put(uploaded) in the bucket, it should trigger the test event and invoke this lambda function. I also set up the same notification configuration on the bucket side. It worked for me the first time I created this lambda function from this example: Create a deployment package

The function only worked for the exact file "HappyFace.jpg", but failed for all other images. And I got "Access Denied" error sometimes. I'm using the following code:

function download(next) {
// Download the image from S3 into a buffer.
Bucket: srcBucket,
Key: srcKey

and has configured the type-match before downloading. I tried to use s3.ListObjects, but it didn't make sense to me logically. Since lambda can be triggered by the upload event, every time I upload a new image it should be invoked for that image, so I don't want to list the objects every time.


I got rid of the access denied problem after I got admin access. It inspired me to inspect the node packages I installed. We might troubleshoot it through this way.

Another issue
I noticed that the async function has been called thousands of times.
enter image description here

So what I did is to set the time interval for the async function by:

setTimeout(function() {
function function1() {
}, 5000);

But it didn't seem to work for me, should I change it to
async.setTimeout(function(Blabla) {});

Answer Source

The access denied error might not be the IAM/S3 bucket/lambda permission issue. If your service can't find the given key in your S3 bucket, it will also return an access denied error to requesters. Because returning NoSuchKey would leak information about the nonexistence of the requested key. For Reference, please check this link: Causes of Access Denied Error

As for how to iteratively invoking the lambda function, you definitely don't need to invoke s3.ListObject() in your code, because that will slow down your performance. But this link might help you to customize your function: Listing Large S3 Buckets with the AWS SDK for Node.js. In the given example of this question, notice that they included util package by:

var util = require('util');

But how they installed with npm is through this command line:

npm install async gm

If you want to make the function be invoked iteratively, you would also like to install "utils" through npm by npm install utils. When it works iteratively through your bucket, you might get access denied error for some files, because you might not have the key configured in your event. You can ignore that.


I also managed to put original images and thumbnails in the same bucket, what you need to do is two things:

  1. Skip the thumbnails by checking prefix or suffix.
  2. Set Timeout Interval. Since we are using 'async', then we don't need to setTimeout for the waterfall function, we can set it outside the waterfall but inside the handler.