How to trigger an aws lambda function to process data from Kinesis stream.

What is aws lambda?

3 min readMay 10, 2023

If you not working with aws lambda functions yet, you probably heard of it at least, it’s at the core of the serverless architectures if you building it on aws.

From the official documentation by aws:

AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume.
With Lambda, you can run code for virtually any type of application or backend service — all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.

Isn’t that something cool?! Please read through the documentation to learn more about the benefits and the possible scenarios where lambda can be a good fit.

What are kinesis streams?

Also from the official documentation:

Amazon Kinesis Data Streams (KDS) is a massively scalable and durable real-time data streaming service. KDS can continuously capture gigabytes of data per second from hundreds of thousands of sources such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events. The data collected is available in milliseconds to enable real-time analytics use cases such as real-time dashboards, real-time anomaly detection, dynamic pricing, and more.

Let’s get started

In this example, I’ll explain how to attach a lambda function to a kinesis data stream and act on it. For this example you will need a valid non-free aws account, since the kinesis resource is not under the aws free tier resources unfortunately.

1. Create the Kinesis stream

Login to the aws console
In the services search box for Kinesis and click on it
Click the “Create data stream” button.
For the name set it as “kinesis-stream” or any other name you prefer.
Set the “Number of Shards” to 1.

With this quick setup the kinesis data stream is ready.

2. Create the Execution role

Your aws lambda function will need permissions to access the kinesis resource, so let’s create the execution role with the following steps.

From the Services select IAM
Click Roles and the “Create Role”
Select “Lambda” and click “Next: Permissions”
Search for “AWSLambdaKinesisExecutionRole” and click “Next: Tags”
You can skip the tags step for now and click “Next: Review”
Enter role name as “lambda-exec-role-kinesis” and click create.

3. Create the lambda function

In the services search box for Lambda and click on it
Click “Create function” button.
We are going to use the Blueprint option for this example.
In the Blueprints searchbox look for “kinesis-process-record” select it and click “Configure”, it’s a predefined lambda function built with nodejs.
For the function name you can use “kinesis-data-processor-example”
Select “Use Existing Role” and search for “lambda-exec-role-kinesis” the one we created in the previous step.
In the Kinesis trigger section for the stream select “kinesis-stream” this is the stream that we created earlier.
At the bottom of the section check “Enable trigger”
Click “Create function”

And now as you can see we have completed the steps need to attach the lambda to the kinesis stream so every record dropped in the stream it will trigger the execution of the code within the lambda function.

The lambda function code itself is basic where it just loops over the stream data and print it out to the console. Of course in a real world scenario you would be doing different handling read the data and pushing it to a database where it gets.

Go ahead and test the lambda function, you can find detailed steps in this article on how to: https://docs.aws.amazon.com/lambda/latest/dg/getting-started-create-function.html#get-started-invoke-manually