fix(custom-resources): inactive lambda functions fail on invoke #22612

TheRealAmazonKendra · 2022-10-23T05:48:14Z

closes #20123

All lambda functions can become inactive eventually. This will result in invocations failing. This PR adds logic to wait for functions to become active on a failed invocation.

All Submissions:

Have you followed the guidelines in our Contributing guide?

Adding new Unconventional Dependencies:

This PR adds new unconventional dependencies following the process described here

New Features

Have you added the new feature to an integration test?
- Did you use yarn integ to deploy the infrastructure and generate the snapshot (i.e. yarn integ without --dry-run)?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

closes #20123 All lambda functions can become inactive eventually. This will result in invocations failing. This PR adds logic to wait for functions to become active on a failed invocation.

gitpod-io · 2022-10-23T05:48:21Z

aws-cdk-automation

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

TheRealAmazonKendra · 2022-10-23T05:51:42Z

We can't really integ test this as it would require a lambda function that was already inactive and we don't have that resource setup.

✅ Updated pull request passes all PRLinter validations. Dissmissing previous PRLinter review.

aws-cdk-automation

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

TheRealAmazonKendra · 2022-12-08T00:49:59Z

@Mergifyio update

mergify · 2022-12-08T00:50:34Z

update

✅ Branch has been successfully updated

✅ Updated pull request passes all PRLinter validations. Dissmissing previous PRLinter review.

ryparker · 2022-12-08T23:20:37Z

packages/@aws-cdk/custom-resources/test/provider-framework/outbound.test.ts

+    expectedFunctionStates.push('Active');
+    expectedFunctionStates.push('Pending');
+
+    expect(await invokeFunction(req)).toEqual({ Payload: req.Payload });


Clever use of await inside & outside the expect()

mergify · 2022-12-08T23:23:24Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

aws-cdk-automation · 2022-12-09T00:01:06Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: b211a04
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2022-12-09T00:01:15Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

…22612) closes aws#20123 All lambda functions can become inactive eventually. This will result in invocations failing. This PR adds logic to wait for functions to become active on a failed invocation. ---- ### All Submissions: * [ ] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

…ermission (#32904) ### Issue # (if applicable) Closes #26838. ### Reason for this change In the Provider Framework lambda code, there is a logic to catch error arise from invoking the User Defined handler lambda. Upon error, it polls the state of the User Defined handler until it is in ACTIVE state (#22612 added this logic): https://github.com/aws/aws-cdk/blob/64b865ba7697f454a1f091a67bf54a6d4ad0e76e/packages/aws-cdk-lib/custom-resources/lib/provider-framework/runtime/outbound.ts#L66-L80 The polling uses the AWS SDK `waitUntilFunctionActiveV2` function, which calls the Lambda GetFunction API: https://github.com/aws/aws-sdk-js-v3/blob/6858c7e04730a2b524b06355969e4076c28ae863/clients/client-lambda/src/waiters/waitForFunctionActiveV2.ts#L57 However, the Provider Framework lambda does not have the `lambda:GetFunction` permission. ##### Why is the issue saying the `lambda:GetFunctionConfiguration` is needed instead of `lambda:GetFunction`? At some point in time, the retry logic used `waitUntilFunctionActive` for polling, which use the `GetFunctionConfiguration`. But this is no longer the case after c3a4b7b#diff-85920270c638d83b082246e0026f1a316dd39aaa3cd8720fdaeb3d526e438f7fR66 ### Description of changes Added the `lambda:GetFunction` permission on the role used by the Provider Framework lambda. ### Describe any new or updated permissions being added The `lambda:GetFunction` permission is added. ### Description of how you validated changes There isn't a straight forward way to test the INACTIVE lambda scenario as one need to wait 14 days for a Lambda function to become INACTIVE. Therefore, I am not able to create an integ test. What I did was locally changing the Provider Framework lambda code to throw an error such that it executes the catch block. Then I verified in CloudTrail that the Provider Framework lambda called `GetFunction` successfully and then it was also able to invoke the User Defined Handler lambda. ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

…ermission (#32904) ### Issue # (if applicable) Closes #26838. ### Reason for this change In the Provider Framework lambda code, there is a logic to catch error arise from invoking the User Defined handler lambda. Upon error, it polls the state of the User Defined handler until it is in ACTIVE state (#22612 added this logic): https://github.com/aws/aws-cdk/blob/64b865ba7697f454a1f091a67bf54a6d4ad0e76e/packages/aws-cdk-lib/custom-resources/lib/provider-framework/runtime/outbound.ts#L66-L80 The polling uses the AWS SDK `waitUntilFunctionActiveV2` function, which calls the Lambda GetFunction API: https://github.com/aws/aws-sdk-js-v3/blob/6858c7e04730a2b524b06355969e4076c28ae863/clients/client-lambda/src/waiters/waitForFunctionActiveV2.ts#L57 However, the Provider Framework lambda does not have the `lambda:GetFunction` permission. ##### Why is the issue saying the `lambda:GetFunctionConfiguration` is needed instead of `lambda:GetFunction`? At some point in time, the retry logic used `waitUntilFunctionActive` for polling, which use the `GetFunctionConfiguration`. But this is no longer the case after c3a4b7b#diff-85920270c638d83b082246e0026f1a316dd39aaa3cd8720fdaeb3d526e438f7fR66 ### Description of changes Added the `lambda:GetFunction` permission on the role used by the Provider Framework lambda. ### Describe any new or updated permissions being added The `lambda:GetFunction` permission is added. ### Description of how you validated changes There isn't a straight forward way to test the INACTIVE lambda scenario as one need to wait 14 days for a Lambda function to become INACTIVE. Therefore, I am not able to create an integ test. What I did was locally changing the Provider Framework lambda code to throw an error such that it executes the catch block. Then I verified in CloudTrail that the Provider Framework lambda called `GetFunction` successfully and then it was also able to invoke the User Defined Handler lambda. ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license* (cherry picked from commit 035d17d)

…ermission (#33315) ### Issue # (if applicable) Closes #26838. ### Reason for this change In the Provider Framework lambda code, there is a logic to catch error arise from invoking the User Defined handler lambda. Upon error, it polls the state of the User Defined handler until it is in ACTIVE state (#22612 added this logic): https://github.com/aws/aws-cdk/blob/64b865ba7697f454a1f091a67bf54a6d4ad0e76e/packages/aws-cdk-lib/custom-resources/lib/provider-framework/runtime/outbound.ts#L66-L80 The polling uses the AWS SDK `waitUntilFunctionActiveV2` function, which calls the Lambda GetFunction API: https://github.com/aws/aws-sdk-js-v3/blob/6858c7e04730a2b524b06355969e4076c28ae863/clients/client-lambda/src/waiters/waitForFunctionActiveV2.ts#L57 However, the Provider Framework lambda does not have the `lambda:GetFunction` permission. ##### Why is the issue saying the `lambda:GetFunctionConfiguration` is needed instead of `lambda:GetFunction`? At some point in time, the retry logic used `waitUntilFunctionActive` for polling, which use the `GetFunctionConfiguration`. But this is no longer the case after c3a4b7b#diff-85920270c638d83b082246e0026f1a316dd39aaa3cd8720fdaeb3d526e438f7fR66 ### Description of changes Added the `lambda:GetFunction` permission on the role used by the Provider Framework lambda. ### Describe any new or updated permissions being added The `lambda:GetFunction` permission is added. ### Description of how you validated changes There isn't a straight forward way to test the INACTIVE lambda scenario as one need to wait 14 days for a Lambda function to become INACTIVE. Therefore, I am not able to create an integ test. What I did was locally changing the Provider Framework lambda code to throw an error such that it executes the catch block. Then I verified in CloudTrail that the Provider Framework lambda called `GetFunction` successfully and then it was also able to invoke the User Defined Handler lambda. ### Checklist - [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md) ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

fix(custom-resources): inactive lambda funtions fail on invoke

5a0a8e9

closes #20123 All lambda functions can become inactive eventually. This will result in invocations failing. This PR adds logic to wait for functions to become active on a failed invocation.

github-actions bot added the bug This issue is a bug. label Oct 23, 2022

aws-cdk-automation requested a review from a team October 23, 2022 05:48

github-actions bot added effort/small Small work item – less than a day of effort p1 labels Oct 23, 2022

mergify bot added the contribution/core This is a PR that came from AWS. label Oct 23, 2022

aws-cdk-automation previously requested changes Oct 23, 2022

View reviewed changes

TheRealAmazonKendra added the pr-linter/exempt-integ-test The PR linter will not require integ test changes label Oct 23, 2022

TheRealAmazonKendra removed the pr-linter/exempt-integ-test The PR linter will not require integ test changes label Oct 26, 2022

aws-cdk-automation previously requested changes Oct 26, 2022

View reviewed changes

mergify bot and others added 4 commits December 8, 2022 00:50

Merge branch 'main' into TheRealAmazonKendra/inactive-custom-resource

ce3fa3a

update awscli-layer integ test

e0e1468

update lambda-layer-kubectl integ test

8226b6d

update lambda-layer-node-proxy-agent integ test

6081005

TheRealAmazonKendra added the pr-linter/exempt-integ-test The PR linter will not require integ test changes label Dec 8, 2022

ryparker reviewed Dec 8, 2022

View reviewed changes

ryparker approved these changes Dec 8, 2022

View reviewed changes

Merge branch 'main' into TheRealAmazonKendra/inactive-custom-resource

b211a04

mergify bot merged commit def2971 into main Dec 9, 2022

mergify bot deleted the TheRealAmazonKendra/inactive-custom-resource branch December 9, 2022 00:01

samson-keung mentioned this pull request Jan 13, 2025

fix(custom-resource): provider framework lambda missing GetFunction permission #32904

Merged

1 task

samson-keung mentioned this pull request Feb 5, 2025

fix(custom-resource): provider framework lambda missing GetFunction permission #33301

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(custom-resources): inactive lambda functions fail on invoke #22612

fix(custom-resources): inactive lambda functions fail on invoke #22612

TheRealAmazonKendra commented Oct 23, 2022

gitpod-io bot commented Oct 23, 2022

aws-cdk-automation left a comment

TheRealAmazonKendra commented Oct 23, 2022

aws-cdk-automation left a comment

TheRealAmazonKendra commented Dec 8, 2022

mergify bot commented Dec 8, 2022

ryparker Dec 8, 2022

mergify bot commented Dec 8, 2022

aws-cdk-automation commented Dec 9, 2022

mergify bot commented Dec 9, 2022

fix(custom-resources): inactive lambda functions fail on invoke #22612

fix(custom-resources): inactive lambda functions fail on invoke #22612

Conversation

TheRealAmazonKendra commented Oct 23, 2022

All Submissions:

Adding new Unconventional Dependencies:

New Features

gitpod-io bot commented Oct 23, 2022

aws-cdk-automation left a comment

Choose a reason for hiding this comment

TheRealAmazonKendra commented Oct 23, 2022

aws-cdk-automation left a comment

Choose a reason for hiding this comment

TheRealAmazonKendra commented Dec 8, 2022

mergify bot commented Dec 8, 2022

✅ Branch has been successfully updated

ryparker Dec 8, 2022

Choose a reason for hiding this comment

mergify bot commented Dec 8, 2022

aws-cdk-automation commented Dec 9, 2022

AWS CodeBuild CI Report

mergify bot commented Dec 9, 2022