Akash Sharma Akash Sharma - 27 days ago 9
Node.js Question

"event" argument in exports.handler of nodejs lambda function triggered by cloudwatch log

I have an AWS Lambda function written in NodeJS which is triggered by a CloudWatch Subscription filter. I was told that the "event" argument of exports.handler in the Lambda function is what I use to get the Log data.

When I print that I see the following which looks nothing like the human readable data I see in the actual CloudWatch console. What does this mean?!? Here's what I get:

{ awslogs: { data: 'H4sIAAAAAAAAAKWS207jMBRFf8XyMwFfYztvURpQpFIqEmYeUIV8S4nUJlXizggh/p3D7Qt4Xfvs47O3/IqPcVnsPnYvp4gLvCq78um2btvypsYXePo/xhmwNlQSKZk0QgE+TPubeTqfQCn/ts2U1tN++eJtmqM9guBD0JzF2CuleiWJ9rLvuZfWyyAAM0u4NULaGIxw1hjgOvggeueYYBHWLWe3+Hk4pWEar4dDivOCi0e8tkcX7NdDTzaEahrH6D+Gumnl7BLx7vOU+l8c04fjFQ8BLuJcaqm1ZkTo3EAUKSUEy3NNieQiJ1wzRSEpBQa61oRSQjRckgaoKdkjJIYiciI4p4Lk/OKnPljPCFUZMRljiNCC0UKKSxhB3X1Z1c2qyF2vvPU00yr6zFCjMglhM+cFM8oCpRRt75tN1WzLNRh+2yB6bDbXdzuE6j/1piu2Z3cYlufPVlB3t22qTXlbF80YhnhFOKnkSvCrdkgRfX+AH0szojbZdF4K1D5UFYj4bff2DvK/Lnc7AgAA' } }


According to the following link, this data is not wrong:
http://docs.aws.amazon.com/lambda/latest/dg/eventsources.html#eventsources-cloudwatch-logs

And according to this answer in StackOverflow, this is data is not wrong either:
KeyError: 'awslogs' ... outEvent = str(event['awslogs']['data']) - Python

Thanks a lot in advance for any help!

Answer Source

You accurately found a Amazon CloudWatch Logs Sample Event. So based on this, your event object would be structured something like:

{
  "awslogs": {
    "data": "H4sIAAAAAAAAAHWPwQqCQBCGX0Xm7EFtK+smZBEUgXoLCdMhFtKV3akI8d0bLYmibvPPN3wz00CJxmQnTO41whwWQRIctmEcB6sQbFC3CjW3XW8kxpOpP+OC22d1Wml1qZkQGtoMsScxaczKN3plG8zlaHIta5KqWsozoTYw3/djzwhpLwivWFGHGpAFe7DL68JlBUk+l7KSN7tCOEJ4M3/qOI49vMHj+zCKdlFqLaU2ZHV2a4Ct/an0/ivdX8oYc1UVX860fQDQiMdxRQEAAA=="
  }
}

What is it?

Lets unpack this to understand better what is going on here. In this case event.awslogs.data would have the value:

"H4sIAAAAAAAAAHWPwQqCQBCGX0Xm7EFtK+smZBEUgXoLCdMhFtKV3akI8d0bLYmibvPPN3wz00CJxmQnTO41whwWQRIctmEcB6sQbFC3CjW3XW8kxpOpP+OC22d1Wml1qZkQGtoMsScxaczKN3plG8zlaHIta5KqWsozoTYw3/djzwhpLwivWFGHGpAFe7DL68JlBUk+l7KSN7tCOEJ4M3/qOI49vMHj+zCKdlFqLaU2ZHV2a4Ct/an0/ivdX8oYc1UVX860fQDQiMdxRQEAAA=="

This value is actually a Base64 encoded string of a GZIPPED payload. This is done to compress the payload, and keep it small, while allowing it to be transported in JSON.

If we Base64 decode this, then unzip it, we end up with the following value:

{
  "messageType": "DATA_MESSAGE",
  "owner": "123456789123",
  "logGroup": "testLogGroup",
  "logStream": "testLogStream",
  "subscriptionFilters": [
    "testFilter"
  ],
  "logEvents": [
    {
      "id": "eventId1",
      "timestamp": 1440442987000,
      "message": "[ERROR] First test message"
    },
    {
      "id": "eventId2",
      "timestamp": 1440442987001,
      "message": "[ERROR] Second test message"
    }
  ]
}

The elements of this are the following:

  • messageType: Either be DATA_MESSAGE or CONTROL_MESSAGE, where if my understanding is right, CONTROL_MESSAGE is used for checking for the reachability of your lambda
  • owner: The AWS Account ID of the originating log
  • logGroup: The Log Group of the originating log
  • logStream: The Log Stream of the originating log
  • subscriptionFilter: The matching Subscription Filter Names
  • logEvents: The actual logs related to what you see in the Console

How do I use it?

You can easily achieve something similar in your Node.js Lambda using the following example from CloudWatches Subscription Filters with AWS Lambda:

var zlib = require('zlib');
exports.handler = function(input, context) {
    var payload = new Buffer(input.awslogs.data, 'base64');
    zlib.gunzip(payload, function(e, result) {
        if (e) { 
            context.fail(e);
        } else {
            result = JSON.parse(result.toString('ascii'));
            console.log("Event Data:", JSON.stringify(result, null, 2));
            context.succeed();
        }
    });
};

Hope that helps.