GO Serverless! Part 2 - Terraform and AWS Lambda External CI

Hamza Adami
September 29th, 2021 · 8 min read

If you didn’t read the previous part, we highly advise you to do that!


In the previous part we have discussed the pros and cons of serverless architectures, different serverless tools and technologies, integration patterns between serverless components and more.

In this part, we are going to discuss how we can decouple Lambda code and dependencies continuous-integration from Terraform and delegate it to external CI services like Codebuild and Codepipeline.

Why Lambda CI using Terraform is a bad and difficult approach?

Difficult

There are many drawbacks when using Terraform to continuously integrate new lambda code/dependencies changes:

From a structural perspective: it is considered a bad practice to mix business logic (code) and packages (dependencies) with Terraform HCL files in the same codebase, because the engineering team building lambda code should not worry about cloning the infrastructure repository and committing to it. On the other hand, the DevOps team should have a clean Terraform codebase without any business logic.

From an infrastructure CI perspective: if your DevOps team is using Terraform Cloud for infrastructure CI, each time a backend engineer pushes a new commit to Lambda code, Terraform Cloud will have to start plan/apply runs, and if the lambda code is actively updated, for example the Lambda is an API and the engineering team is updating it many times during the day, this will slow down Terraform Cloud runs queue as terraform can only have one active run per workspace, and, eventually, the DevOps team will be frustrated because they will have to wait for the runs to finish.

No way to support releases: you can’t have a QA Lambda which is updated on merges to master, and a Production Lambda which is updated if a tag/release is created, because the lambda is not version controlled on its own git repository, and it is part of the infrastructure repository.

No proper way to build lambda application dependencies: if the package’s descriptor changed, there is no way to build and publish the dependencies again and the process should be done manually each time there is a change.

For compatibility: Lambda application dependencies should be built using a runtime similar to the actual Lambda runtime, and even worse than that, if you decide to build dependencies using Terraform Cloud, you can’t. because you don’t have control over terraform instances, and you don’t know if a specific nodejs or python version is already installed.

What are we trying to achieve?

Bring Balance

Before we start implementing the solution we should agree on the set of capabilities the solution must adhere to:

Support famous runtimes: The CI process should be able to support at least the two famous lambda runtimes python and nodejs.

Support custom packages: installing custom packages that does not exist in Lambda runtime passed to CI process as a package’s descriptor file path in git repository. packages.json or requirements.txt.

Caching: the integration/deployment process should be fast thanks to code and dependencies caching.

Resiliency: during deployment, the process should not break the currently published version and traffic shifting between the current and new deployment should be seamless.

Compatibility: Lambda dependencies packages should be built in a sandboxed local environment that replicates the live AWS Lambda environment almost identically – including installed software and libraries.

Loose coupling: Lambda code should be decoupled entirely from Terraform.

Interoperability: The CI process should be able to deploy Lambdas versioned in github repositories.

Support mono-repos: The solution should be able to support mono-repos and dedicated repos, so all lambdas can be placed in a single git repositories, or it can be versioned in a dedicated repository.

Alerts: The CI pipeline should notify the failure or success of the process through slack.

Prepare terraform to delegate CI to codepipeline/codebuild

are_we_realy_doing_this

In order to delegate Lambda CI to codepipeline and codebuild, we should decouple lambda code and dependencies from terraform resources.

Lambda dummy code package

The first thing we need to do is providing a dummy code package to Terraform Lambda resource and instructing Terraform to ignore any changes affecting s3_key, s3_bucket, layers and filename as they will be changed by the CI process and we don’t want Terraform to revert those changes on every deployment.

In other words, the external CI process will say to terraform: Don't worry about code or dependencies, I will take care of that

1data "archive_file" "dummy" {
2 output_path = "${path.module}/dist.zip"
3 type = "zip"
4 source {
5 content = "dummy dummy"
6 filename = "dummy.txt"
7 }
8}
9
10resource "aws_lambda_function" "function" {
11 function_name = local.prefix
12 publish = true
13 role = aws_iam_role.role.arn
14
15 # runtime
16 runtime = var.runtime
17 handler = var.handler
18
19 # resources
20 memory_size = var.memory_size
21 timeout = var.timeout
22
23 # dummy package, package is delegated to CI pipeline
24 filename = data.archive_file.dummy.output_path
25
26 environment {
27 variables = var.envs
28 }
29
30 tags = merge(
31 local.common_tags,
32 map(
33 "description", var.description
34 )
35 )
36
37 # LAMBDA CI is done through codebuild/codepipeline
38 lifecycle {
39 ignore_changes = [s3_key, s3_bucket, layers, filename]
40 }
41}

The dummy package we have created and provided to Lambda resource is important, because during Lambda resource creation Terraform will not be able to create the Lambda function without a code package.

We also added a lifecycle ignore_changes meta-argument instructing terraform to ignore changes to code [s3_key, s3_bucket, filename] and dependencies [layers].

Lambda alias

A Lambda alias is like a pointer to a specific function version. We can access the function version using the alias Amazon Resource Name (ARN). Since the Lambda version will be changed constantly, we need to create a latest alias and on each deploy we have to update the alias with the new lambda version.

1# Required by lambda provisioned concurrency
2resource "aws_lambda_alias" "alias" {
3 name = "latest"
4 description = "alias pointing to the latest published version of the lambda"
5 function_name = aws_lambda_function.function.function_name
6 function_version = aws_lambda_function.function.version
7
8 lifecycle {
9 ignore_changes = [
10 description,
11 routing_config
12 ]
13 }
14}

The main advantage of Lambda alias is to avoid other resources depending on the lambda function from invoking a broken lambda function version, and they will always invoke the latest stable version that the CI process have tagged with latest alias at the end of the pipeline.

The other advantage is the ability for other services to invoke latest version without needing to update them whenever the function version changes.

We can also specify routing configuration on an alias to send a portion of the traffic to a second function version. For example, we can reduce the risk of deploying a new version by configuring the alias to send most of the traffic to the existing version, and only a small percentage of traffic to the new version.

A lambda alias is required in case you want to implement provisioned concurrency afterwards.

Calling the module

We have prepared terraform-aws-codeless-lambda reusable terraform module for you here, you can call it like this for creating Lambda functions ready for External CI:

1module "lambda" {
2 source = "git::https://github.com/obytes/terraform-aws-codeless-lambda.git//modules/lambda"
3 prefix = "demo-api"
4 common_tags = {env = "test", stack = "demos"}
5 description = "Terraform is my creator but Codepipeline is the demon I listen to"
6
7 envs = {API_KEY = "not-secret"}
8 runtime = "python3.9"
9 handler = "app.handler"
10 timeout = 29
11 memory_size = 1024
12
13 policy_json = data.aws_iam_policy_document.policy.json
14 logs_retention_in_days = 14
15}

The module will output the required attributes that will be used by CI process and by other components using/invoking the lambda, like API Gateway, etc.

1output "lambda" {
2 value = {
3 name = aws_lambda_function.function.function_name
4 arn = aws_lambda_function.function.arn
5 runtime = aws_lambda_function.function.runtime
6 alias = aws_lambda_alias.alias.name
7 invoke_arn = aws_lambda_alias.alias.invoke_arn
8 }
9}

CI Process Workflow

PATIENCE

Before writing the CI script, we need to design the workflow of the process.

Lambda CI workflow

Check

In the first stage we checkout the new packages’ descriptor, and check if the previous packages’ descriptor exists in S3, which is an edge case for the first execution. And if the package descriptor exists we need to download it for next stage.

We also checkout the new application code and current source code hash that will be used in the next stage!

Bundle

If it is the first build we can jump directly and build/package the dependencies in the packages’ descriptor

Otherwise, if it is not the first build, and the previous descriptor is already cached in S3, we will compare the previously cached descriptor with the new descriptor.

If they are not the same, then the descriptor is changed, and we will have to build and package the dependencies again, or else we can jump directly to the next stage.

Next, we package the new application source code, and we calculate its hash and compare it with the current source code hash. if they are different, then we have to push the new source code to S3.

For building lambda application dependencies we will use lambci/lambda docker image which is a sandboxed local environment that replicates the live AWS Lambda environment almost identically – including installed software and libraries.


Uploading artifacts

After building and packaging application code and/or dependencies, it’s time to upload the packaged files to S3. we need to upload the artifacts in two different names:

  • The first package should include the word latest in the artifact name.
  • The second package should include the commit digest hash in the artifact name.

The reason for this redundancy is to ensure we don’t update the existing Lambda code with broken packages, so the previous lambda version will keep pointing to the previous code/dependencies, and the newer lambda version will point to the new code with the new commit digest hash as the artifact name.

This is an example of the files that will be uploaded to S3:

  • APP version S3 KEY - lambda-ci-demo/{hash}/app.zip
  • APP latest S3 KEY - lambda-ci-demo/latest/app.zip
  • DEP version S3 KEY - lambda-ci-demo/{hash}/deps.zip
  • DEP latest S3 KEY - lambda-ci-demo/latest/deps.zip

AWS Lambda size limit is 50mb when you upload the code directly to the Lambda service. However, if the code deployment package size is more than that, we have an option to upload it to S3 and download it while triggering the function invocation. That’s why we are going to use S3 to support large deployment packages.


Deploying dependencies and application

Now that we have the dependencies stored in S3, we can publish a new layer version that points to the new dependencies with publish_layer_version(..) and update the lambda function to use the new layer version with update_function_configuration(..).

After a successful deployment, we cache the new packages’ descriptor in order for subsequent runs to be faster.

After that, we have to deploy the actual lambda code, so we update the Lambda function to point to the newer code package in S3 using update_function_code(..).

Publishing application

Yes, we have deployed our Lambda function with newer dependencies and newer code but the services depending on it are still pointing to the old Lambda function version tagged with latest alias, so we need to shift traffic to the new version.

First, we should create a new lambda function version using publish_version(..) and then we have to update the lambda alias to point to the newly created lambda version using update_alias(..).

Implementation

The stages in the above workflow are implemented in this package https://github.com/obytes/aws-lambda-ci and published in a public PyPi repository, you can install it with pip:

1pip3 install aws-lambda-ci

Local CI with standalone CI Script

automated_ci

In case you are still developing your function locally and you want to test it but you still don’t have codebuild and codepipeline resources in development environment, you can use aws-lambda-ci like this:

1aws-lambda-ci \
2--app-s3-bucket "demo-artifacts" \
3--function-name "demo-api" \
4--function-runtime "python3.9" \
5--function-alias-name "latest" \
6--function-layer-name "demo-api-deps" \
7--app-src-path "src" \
8--app-packages-descriptor-path "requirements.txt" \
9--source-version "1.0.2" \
10--aws-profile-name "kodhive_prd" \
11--watch-log-stream

First usage will take a longer time, because it needs to pull lambci/lambda from dockerhub.

Code and dependencies changed

If both code and dependencies changed, the pipeline will publish both changes.

demo_code_changed_deps_changed

Just code changed

If code changed but not dependencies, the pipeline will publish new code and the dependencies will be left intact.

demo_just_code_changed

Nothing changed

If both code and dependencies not changed, the pipeline will not publish anything.

demo_nothing_changed

Leveraging codebuild/codepipeline for automated CI

fun_begins

After preparing the python CI package, we can now configure codebuild and codepipeline to use it.

TLDR: We’ve published a reusable terraform module for Lambda CI using Codepipeline/Codebuild, you can check it here.

Build specification

Let’s start by creating the build specification file. It will be used by the codebuild project when the build is triggered by codepipeline. In the install phase, we instruct codebuild to use runtimes shipped with docker.

In the pre-build phase, we get the source version that triggered the build, which can be a commit hash for pushes events or SemVer tag for releases events and we install aws-lambda-ci.

Finally, during build phase, we call aws-lambda-ci that should take care of the next CI phases.

1version: 0.2
2
3phases:
4 install:
5 runtime-version:
6 docker: 19
7
8 pre_build:
9 commands:
10 - SOURCE_VERSION=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION)
11 - pip3 install aws-lambda-ci
12
13 build:
14 commands:
15 - |
16 aws-lambda-ci \
17 --source-version "$SOURCE_VERSION" \
18 --function-name "$FUNCTION_NAME" \
19 --function-runtime "$FUNCTION_RUNTIME" \
20 --function-alias-name "$FUNCTION_ALIAS_NAME" \
21 --function-layer-name "$FUNCTION_NAME-deps" \
22 --app-s3-bucket "$APP_S3_BUCKET" \
23 --app-src-path "$APP_SRC_PATH" \
24 --app-packages-descriptor-path "$APP_PACKAGES_DESCRIPTOR_PATH"

Codebuild

After preparing the build specification file, we can now pass it to codebuild project along with environment variables that will be forwarded to aws-lambda-ci.

It’s important to set the source to CODEPIPELINE as codepipeline will trigger the codebuild project in response to Github webhook and will prevent concurrent builds that can cause issues during deployment.

1resource "aws_codebuild_project" "default" {
2 name = local.prefix
3 description = "Build ${var.github_repository.branch} of ${var.github_repository.name} and deploy"
4 build_timeout = var.build_timeout
5 service_role = aws_iam_role.role.arn
6
7 source {
8 type = "CODEPIPELINE"
9 buildspec = file("${path.module}/templates/buildspec.yml")
10 }
11
12 artifacts {
13 type = "CODEPIPELINE"
14 }
15
16 cache {
17 type = "LOCAL"
18 modes = ["LOCAL_DOCKER_LAYER_CACHE"]
19 }
20
21 environment {
22 compute_type = var.compute_type
23 image = var.image
24 type = var.type
25 privileged_mode = var.privileged_mode
26
27 # Bucket
28 # ------
29 environment_variable {
30 name = "APP_S3_BUCKET"
31 value = var.s3_artifacts.bucket
32 }
33
34 # Build
35 # -----
36 environment_variable {
37 name = "APP_SRC_PATH"
38 value = var.app_src_path
39 }
40
41 environment_variable {
42 name = "APP_PACKAGES_DESCRIPTOR_PATH"
43 value = var.packages_descriptor_path
44 }
45
46 # Function
47 # --------
48 environment_variable {
49 name = "FUNCTION_NAME"
50 value = var.lambda.name
51 }
52
53 environment_variable {
54 name = "FUNCTION_RUNTIME"
55 value = var.lambda.runtime
56 }
57
58 environment_variable {
59 name = "FUNCTION_ALIAS_NAME"
60 value = var.lambda.alias
61 }
62
63 environment_variable {
64 name = "FUNCTION_LAYER_NAME"
65 value = "${var.lambda.name}-deps"
66 }
67 }
68
69 tags = local.common_tags
70}

Codepipeline

Now that we have prepared the codebuild project, we will create the pipeline that will orchestrate the CI runs/stages.

During Source stage, we set up codepipeline to detect any change to target git branch if the environment is a pre-release like qa or staging, checkout the lambda git repository and send it to next stage through artifacts bucket.

In BuildAndDeploy stage, codepipeline will trigger codebuild and wait for codebuild execution to deploy and publish the lambda function.

1resource "aws_codepipeline" "default" {
2 name = local.prefix
3 role_arn = aws_iam_role.role.arn
4
5 ##########################
6 # Artifact Store S3 Bucket
7 ##########################
8 artifact_store {
9 location = var.s3_artifacts.bucket
10 type = "S3"
11 }
12
13 #########################
14 # Pull source from Github
15 #########################
16 stage {
17 name = "Source"
18
19 action {
20 name = "Source"
21 category = "Source"
22 owner = "AWS"
23 provider = "CodeStarSourceConnection"
24 version = "1"
25 output_artifacts = ["code"]
26
27 configuration = {
28 FullRepositoryId = "${var.github.owner}/${var.github_repository.name}"
29 BranchName = var.github_repository.branch
30 DetectChanges = var.pre_release
31 ConnectionArn = var.github.connection_arn
32 OutputArtifactFormat = "CODEBUILD_CLONE_REF"
33 }
34 }
35 }
36
37 #########################
38 # Build & Deploy to S3
39 #########################
40 stage {
41 name = "BuildAndDeploy"
42
43 action {
44 name = "BuildAndDeploy"
45 category = "Build"
46 owner = "AWS"
47 provider = "CodeBuild"
48 input_artifacts = ["code"]
49 output_artifacts = ["package"]
50 version = "1"
51
52 configuration = {
53 ProjectName = var.codebuild_project_name
54 }
55 }
56 }
57}

Release webhook

To support triggering the pipeline in response to github releases, we have to configure webhook resources that will only be created if var.pre_release is set to false.

1# Webhooks (Only for github releases)
2resource "aws_codepipeline_webhook" "default" {
3 count = var.pre_release ? 0:1
4 name = local.prefix
5 authentication = "GITHUB_HMAC"
6 target_action = "Source"
7 target_pipeline = aws_codepipeline.default.name
8
9 authentication_configuration {
10 secret_token = var.github.webhook_secret
11 }
12
13 filter {
14 json_path = "$.action"
15 match_equals = "published"
16 }
17}
18
19resource "github_repository_webhook" "default" {
20 count = var.pre_release ? 0:1
21 repository = var.github_repository.name
22
23 configuration {
24 url = aws_codepipeline_webhook.default.0.url
25 secret = var.github.webhook_secret
26 content_type = "json"
27 insecure_ssl = true
28 }
29
30 events = [ "release" ]
31}

Slack Notifications

To receive slack alerts for pipeline’s failure and success events, we will add two notification rules. the first one will notify started and succeeded events and the second will notify failure events.

1resource "aws_codestarnotifications_notification_rule" "notify_info" {
2 name = "${local.prefix}-info"
3 resource = aws_codepipeline.default.arn
4 detail_type = "FULL"
5 status = "ENABLED"
6
7 event_type_ids = [
8 "codepipeline-pipeline-pipeline-execution-started",
9 "codepipeline-pipeline-pipeline-execution-succeeded"
10 ]
11
12 target {
13 type = "AWSChatbotSlack"
14 address = "${local.chatbot}/${var.ci_notifications_slack_channels.info}"
15 }
16}
17
18resource "aws_codestarnotifications_notification_rule" "notify_alert" {
19 name = "${local.prefix}-alert"
20 resource = aws_codepipeline.default.arn
21 detail_type = "FULL"
22 status = "ENABLED"
23
24 event_type_ids = [
25 "codepipeline-pipeline-pipeline-execution-failed",
26 ]
27
28 target {
29 type = "AWSChatbotSlack"
30 address = "${local.chatbot}/${var.ci_notifications_slack_channels.alert}"
31 }
32}

Put it all together

finally

We can structure the CI components into two modules, one for the build and another for the pipeline.

1module "code_build_project" {
2 source = "../../components/build"
3 prefix = local.prefix
4 common_tags = var.common_tags
5
6 # S3
7 s3_artifacts = var.s3_artifacts
8 lambda = var.lambda
9
10 # App
11 app_src_path = var.app_src_path
12 packages_descriptor_path = var.packages_descriptor_path
13
14 # Github
15 connection_arn = var.github.connection_arn
16 github_repository = var.github_repository
17}
18
19module "code_pipeline_project" {
20 source = "../../components/pipeline"
21 prefix = local.prefix
22 common_tags = local.common_tags
23
24 # Github
25 github = var.github
26 pre_release = var.pre_release
27 github_repository = var.github_repository
28
29 # S3
30 s3_artifacts = var.s3_artifacts
31
32 # Codebuild
33 codebuild_project_name = module.code_build_project.codebuild_project_name
34
35 ci_notifications_slack_channels = var.ci_notifications_slack_channels
36}

Finally, we will wrap the two modules into a single module and call it like this.

1module "lambda_ci" {
2 source = "git::https://github.com/obytes/terraform-aws-lambda-ci.git//modules/ci"
3 prefix = "demo-api-ci"
4 common_tags = {env = "test", stack = "demos-ci"}
5
6 lambda = module.lambda.lambda
7 app_src_path = "src"
8 packages_descriptor_path = "requirements.txt"
9
10 # Github
11 s3_artifacts = {
12 arn = aws_s3_bucket.artifacts.arn
13 bucket = aws_s3_bucket.artifacts.bucket
14 }
15 pre_release = true
16 github = {
17 owner = "obytes"
18 token = "gh_123456789876543234567845678"
19 webhook_secret = "not-secret"
20 connection_arn = "[GH_CODESTAR_CONNECTION_ARN]"
21 }
22 github_repository = {
23 name = "demo-api"
24 branch = "main"
25 }
26
27 # Notifications
28 ci_notifications_slack_channels = {
29 info = "ci-info"
30 alert = "ci-alert"
31 }
32}

What’s next?

Now that we have implemented our Lambda CI reusable pipeline, in the next part we will use it to deploy nodejs and flask APIs.

Share the article if you find it useful! See you next time.

party-over-star-wars

More articles from Obytes

GO Serverless Part 1 Serverless Architecture and Components Integration Patterns

Cloud serverless architectures became more and more popular in the past years, and many enterprises started leveraging it and even migrating…

September 15th, 2021 · 18 min read

Handle and track chat messages delivery using React

Make your chat look like Whatsapp In this article, we'll be talking about tracking messages delivery inside a chat application using React…

September 2nd, 2021 · 2 min read

ABOUT US

Our mission and ambition is to challenge the status quo, by doing things differently we nurture our love for craft and technology allowing us to create the unexpected.