Executing Asynchronous Operations during the AWS Lambda NodeJs Init Phase
Previously an impossibility, AWS Lambda NodeJs async operations can finally complete during the init phase before the handler executes — with a little help. AWS Lambda Extensions provide the necessary addition to AWS Lambda to support asynchronous initialization before the function handler is started.
With a little help, it is finally possible to fully initialize NodeJs code during the function init phase, and leverage the value of provisioned concurrency including the benefits of warming through asynchronous operations — before a user connects.
Asynchronous operations can now fully complete before the first user connects with provisioned concurrency. This list is a sample of the potential operations.
- Pre-caching data in-memory data
- Pre-loading configuration such as secrets from AWS Secrets Manager
- Initializing connections to AWS DynamoDb
- Initializing connections to AWS RDS (Relational Database Service)
- Connecting to backend services
These are just a few examples of operations that a developer might want to complete during initializing before a user connects. These types of initialization operations minimize delays when responding to user requests.
It was exciting to see AWS announce provisioned concurrency, but the AWS Lambda NodeJs initialization phase only completed synchronous code. Asynchronous calls during the init phase did not complete until subsequent requests. They started but did not complete until the function handler started. This was a surprise and limited the full benefits of a separate init phase until now.
Github examples are included at the bottom of the article to demonstrate this process. The important parts are discussed in this article.
The NodeJs Async Init Problem
When AWS announced provisioned concurrency, I was excited, and expected most cold start issues to be resolved. Provisioned concurrency resolved most cold start issues except in the case of NodeJs Lambda initialization that needed to make asynchronous calls. In this case, the initialization phase does not complete until a future request. The async operation can be started with an async IIFE; however, the operation does not complete during initialization. This is because the lambda container is paused as soon as all synchronous tasks complete during the initialization phase.
Async operations during the NodeJs initialization phase are placed on the event loop with any other code that needs to be executed until the next iteration. Once all synchronous operations complete, the container is paused while the asynchronous operations wait in the background. Nothing else happens until the function is actually started on the first request. This means that any asynchronous initialization code will not complete.
On the first request, the asynchronous operation will continue in the background while other operations execute. Since the asynchronous operation executes in the background, it may complete during the first request or any subsequent request. The completion of the init phase is indeterminate.
Note, the init phase can be forced to complete on the first request by awaiting a promise during that request. This is occurs after the initialization phase during the function execution phase. This provides no more benefits for provisioned concurrency than old school AWS Lambda warming techniques.
This problem is unique to AWS Lambda NodeJs runtimes, and is partially due to how the event loop works in JavaScript. The initialization phase for other types of runtimes will complete successfully during the initialization phase.
The Solution
I considered multiple options to resolve this problem without success until discovering that this could be accomplish by leveraging an AWS Lambda Extension to solve the problem. AWS Lambda Extensions keep the container in an active state until the Lambda Extensions API is notified that the extension layer is ready for the next request. Once the lambda function returns and all extension layers notify the Lambda Extensions API that they are ready for another request, the lambda container will pause until the next request.
The rest of this article walks through the solution demonstrated in the below Github repository. It demonstrates how a simple extension layer can be used to allow asynchronous functions to complete during the AWS Lambda function’s init phase.
Solution Overview
This extension layer works by preventing the initialization phase from completing until the layer receives a socket message to indicate that the asynchronous initialization is complete. The below code is used to warm the function during the initialization phase. The 2 second timeout promise demonstrates a long running asynchronous initialization. Once complete, a socket message is sent to notify the extension layer that initialization is complete. For this demonstration, the extension layer does not care about the content of the message.
The extension layer waits to receive a notice from the function or a timeout, whichever occurs first. The timeout is specified through an environment variable called WARMING_TIMEOUT. If this environment variable is not set, then the AWS Lambda Extension will immediately notify the Lambda Extensions API that it is ready for the next request. This prevents functions from having unintentionally long startup times.
If the WARMING_TIMEOUT variable is not set, the extension allows the function to work as if the extension was not installed. If WARMING_TIMEOUT is set, the extension will wait for the specified number of milliseconds to receive a notification from the function. If a notification is received or if the timeout completes (whichever happens first), the extension notifies the Lambda Extensions API that it is ready for the next request. At this point, the container is paused.
Log Output Demonstration and Review
This is observable by adjusting the WARMING_TIMEOUT. The demonstrations leverage on-demand concurrency. If provisioned concurrency is used, the “START RequestId” lines will output after the provisioned concurrency initialization. Note, on-demand requests can still occur when provisioned concurrency is enabled. This occurs when there are more concurrent requests than provisioned lambda instances.
WARMING_TIMEOUT=0 effectively disables the extension layer functionality.
If a WARMING_TIMEOUT environment variable is set to 0 milliseconds, the function will show in the logs that warming is starting. Then during a subsequent request, it will show that warming was completed. This demonstrates that the warming functionality does not complete during the initialization phase. These log entries will be something like the below image.
Specific items to note in this output are below.
1. The init duration is only 226.87 milliseconds rather than waiting the full 2 seconds in the async sleep init. This is because the timeout occurs immediately (0 milliseconds) before the initialization is complete.
2. The “starting warming” occurs during initialization after the request starts at 58:38.771.
3. The “starting function” at 58:38.827 means that the function init phase has completed and the function has started. In NodeJs, the async init IIFE continues in the background but does not prevent the function from starting because the extension layer timeout indicates the extension layer is ready.
3. The function completes. The container is paused. Then the “warming complete” occurs after the lambda is unpaused, and the next request has started at 58:46.567. In this case, the next request occurs about 8 seconds after the first request, and the initialization completes during this request (the sleep timeout completes).
A WARMING_TIMEOUT of 0 causes the extension layer to immediately allow the next request. This allows the function to execute as if the extension layer was not there. This prevents unexpectedly delayed initializations for lambda functions that do not contain the socket code to notify the extension.
WARMING_TIMEOUT=3000 provides the init phase up to 3000 milliseconds for async initialization before the handler executes.
If the WARMING_TIMEOUT is increased to 3000 milliseconds, the log entries will look similar to the below entries. The fake async warming task of sleeping for 2 seconds will complete during the initialization phase, and then the function will initialize. WARMING_TIMEOUT should be set to the maximum desired time to allow for function initialization.
Specific items to note when the WARMING_TIMEOUT is 3000 are mentioned below.
1. The initialization time is 2251.00 milliseconds. This demonstrates that the init phase waits for the sleep promise within the function initialization to complete.
2. The “starting warming” output from the function is output at 9:37.174 although the write occurs at 9:35.168. This demonstrates that the log output can sometimes have delays. Warming starts a 9:35.168 even though the log is not written until 2 seconds later.
3. The “warming complete” output happens at 9:37.175. This occurs just over 2 seconds after the sleep promise is started, and shows that the initialization waits for the sleep promise to complete before the function handler executes.
4. The “starting function” from inside the handler body occurs at 9:37.179. The function handler does not execute until after the function initialization is complete.
This review walks through a clean implementation of an extension layer that waits for asynchronous initialization during AWS Lambda NodeJs functions. With a little help, it is finally possible to fully initialize NodeJs code during the function init phase, and leverage the value of provisioned concurrency including the benefits of warming through asynchronous operations — before a user connects.
The code referenced in this repo can be deployed with the AWS CDK. The output will provide a URL to hit the function AWS API Gateway endpoint. The “npm run deploy-sample” command will deploy code to the default configured AWS cli credentials (aws configure).
npm ci
npm run deploy-sample
The code can be un-deployed by running “npm run destroy-sample”. It can also be run locally if the sam cli is installed with “npm run deploy-local”. Running the code locally does not display the same timestamp information as the CloudWatch log output.
The full source code for this sample is currently available at the below Github repository.
Conclusion
Asynchronous operations can finally be initialized in AWS Lambda NodeJs runtimes.
I appreciate any comments or feedback. If you have questions about implementing AWS Lambda Extensions, let me know via comments or messages. AWS Lambda Extensions and Lambda Layers provide amazing value to the overall AWS Lambda experience.
