# GO Serverless! Part 2 - Terraform and AWS Lambda External CI

- URL: https://www.obytes.com/blog/aws-lambda-ci
- Published: 2021-09-29
- Author: Hamza Adami
- Topics: DevOps
- Summary: Keep Terraform for infrastructure and move AWS Lambda deployments to an external CodePipeline and CodeBuild CI using a dummy package and Lambda aliases.

> If you didn't read the [previous part](https://www.obytes.com/blog/go-serverless-part-1-serverless-architecture-and-components-integration-patterns), we highly advise you to do that!

In the [previous part](https://www.obytes.com/blog/go-serverless-part-1-serverless-architecture-and-components-integration-patterns) we have discussed the pros and cons of serverless architectures, different serverless tools
and technologies, integration patterns between serverless components and more.

In this part, we are going to discuss how we can decouple Lambda **`code`** and **`dependencies`**
continuous-integration from Terraform and delegate it to external CI services like **`Codebuild`**
and **`Codepipeline`**.

## Why Lambda CI using Terraform is a bad and difficult approach?

![Difficult](https://www.obytes.com/assets/posts/2021-09-29-aws-lambda-ci/images/difficult_sometimes.gif)

There are many drawbacks when using Terraform to continuously integrate new lambda code/dependencies changes:

❌ **`From a structural perspective:`** it is considered a bad practice to mix business logic (code) and packages
(dependencies) with Terraform HCL files in the same codebase, because the engineering team building lambda code should
not worry about cloning the infrastructure repository and committing to it. On the other hand, the DevOps team should
have a clean Terraform codebase without any business logic.

❌ **`From an infrastructure CI perspective:`** if your DevOps team is using Terraform Cloud for infrastructure CI, each
time a backend engineer pushes a new commit to Lambda code, Terraform Cloud will have to start plan/apply runs, and if
the lambda code is actively updated, for example the Lambda is an API and the engineering team is updating it many times
during the day, this will slow down Terraform Cloud runs queue as terraform can only have one active run per workspace,
and, eventually, the DevOps team will be frustrated because they will have to wait for the runs to finish.

❌ **`No way to support releases:`** you can't have a QA Lambda which is updated on merges to master, and a
Production Lambda which is updated if a tag/release is created, because the lambda is not version controlled on its own
git repository, and it is part of the infrastructure repository.

❌ **`No proper way to build lambda application dependencies:`** if the package's descriptor changed, there is no way to
build and publish the dependencies again and the process should be done manually each time there is a change.

❌ **`For compatibility:`** Lambda application dependencies should be built using a runtime similar to the actual Lambda
runtime, and even worse than that, if you decide to build dependencies using Terraform Cloud, you can't. because you
don't have control over terraform instances, and you don't know if a specific nodejs or python version is already
installed.

## What are we trying to achieve?

![Bring Balance](https://www.obytes.com/assets/posts/2021-09-29-aws-lambda-ci/images/bring_balance.gif)

Before we start implementing the solution we should agree on the set of capabilities the solution must adhere to:

✅ **`Support famous runtimes:`** The CI process should be able to support at least the two famous lambda runtimes
python and nodejs.

✅ **`Support custom packages:`** installing custom packages that does not exist in Lambda runtime passed to CI process
as a package's descriptor file path in git repository. *`packages.json`* or *`requirements.txt`*.

✅ **`Caching:`** the integration/deployment process should be fast thanks to code and dependencies caching.

✅ **`Resiliency:`** during deployment, the process should not break the currently published version and traffic
shifting between the current and new deployment should be seamless.

✅ **`Compatibility:`** Lambda dependencies packages should be built in a sandboxed local environment that replicates
the live AWS Lambda environment almost identically – including installed software and libraries.

✅ **`Loose coupling:`** Lambda code should be decoupled entirely from Terraform.

✅ **`Interoperability:`** The CI process should be able to deploy Lambdas versioned in github repositories.

✅ **`Support mono-repos:`** The solution should be able to support mono-repos and dedicated repos, so all lambdas can
be placed in a single git repositories, or it can be versioned in a dedicated repository.

✅ **`Alerts:`** The CI pipeline should notify the failure or success of the process through slack.

## Prepare terraform to delegate CI to codepipeline/codebuild

![are\_we\_realy\_doing\_this](https://www.obytes.com/assets/posts/2021-09-29-aws-lambda-ci/images/are_we_realy_doing_this.gif)

In order to delegate Lambda CI to codepipeline and codebuild, we should decouple lambda code and dependencies from
terraform resources.

### Lambda dummy code package

The first thing we need to do is providing a **`dummy code package`** to Terraform Lambda resource and instructing
Terraform to ignore any changes affecting **`s3_key`**, **`s3_bucket`**, **`layers`** and **`filename`** as they will be
changed by the CI process and we don't want Terraform to revert those changes on every deployment.

In other words, the external CI process will say to terraform: *`Don't worry about code or dependencies, I will take
care of that`*

```hcl
data "archive_file" "dummy" {
  output_path = "${path.module}/dist.zip"
  type        = "zip"
  source {
    content  = "dummy dummy"
    filename = "dummy.txt"
  }
}

resource "aws_lambda_function" "function" {
  function_name = local.prefix
  publish       = true
  role          = aws_iam_role.role.arn

  # runtime
  runtime = var.runtime
  handler = var.handler

  # resources
  memory_size = var.memory_size
  timeout     = var.timeout

  # dummy package, package is delegated to CI pipeline
  filename = data.archive_file.dummy.output_path

  environment {
    variables = var.envs
  }

  tags = merge(
    local.common_tags,
    map(
      "description", var.description
    )
  )

  # LAMBDA CI is done through codebuild/codepipeline
  lifecycle {
    ignore_changes = [s3_key, s3_bucket, layers, filename]
  }
}
```

The dummy package we have created and provided to Lambda resource is important, because during Lambda resource creation
Terraform will not be able to create the Lambda function without a code package.

We also added a lifecycle **`ignore_changes`** meta-argument instructing terraform to ignore changes to code
**`[s3_key, s3_bucket, filename]`** and dependencies **`[layers]`**.

### Lambda alias

A Lambda alias is like a pointer to a specific function version. We can access the function version using the alias
**`Amazon Resource Name (ARN)`**. Since the Lambda version will be changed constantly, we need to create a **`latest`**
alias and on each deploy we have to update the alias with the new lambda version.

```hcl
# Required by lambda provisioned concurrency
resource "aws_lambda_alias" "alias" {
  name             = "latest"
  description      = "alias pointing to the latest published version of the lambda"
  function_name    = aws_lambda_function.function.function_name
  function_version = aws_lambda_function.function.version

  lifecycle {
    ignore_changes = [
      description,
      routing_config
    ]
  }
}
```

The main advantage of Lambda alias is to avoid other resources depending on the lambda function from invoking a broken
lambda function version, and they will always invoke the latest stable version that the CI process have tagged with
**`latest`** alias at the end of the pipeline.

The other advantage is the ability for other services to invoke **`latest`** version without needing to update them
whenever the function version changes.

We can also specify routing configuration on an alias to send a portion of the traffic to a second function version. For
example, we can reduce the risk of deploying a new version by configuring the alias to send most of the traffic to the
existing version, and only a small percentage of traffic to the new version.

> A lambda alias is required in case you want to implement provisioned concurrency afterwards.

### Calling the module

We have prepared **`terraform-aws-codeless-lambda`** reusable terraform module for you [here](https://github.com/obytes/terraform-aws-codeless-lambda), you can call it like this
for creating Lambda functions ready for External CI:

```hcl
module "lambda" {
  source      = "git::https://github.com/obytes/terraform-aws-codeless-lambda.git//modules/lambda"
  prefix      = "demo-api"
  common_tags = {env = "test", stack = "demos"}
  description = "Terraform is my creator but Codepipeline is the demon I listen to"

  envs        = {API_KEY = "not-secret"}
  runtime     = "python3.9"
  handler     = "app.handler"
  timeout     = 29
  memory_size = 1024

  policy_json            = data.aws_iam_policy_document.policy.json
  logs_retention_in_days = 14
}
```

The module will output the required attributes that will be used by CI process and by other components using/invoking
the lambda, like API Gateway, etc.

```hcl
output "lambda" {
  value = {
    name       = aws_lambda_function.function.function_name
    arn        = aws_lambda_function.function.arn
    runtime    = aws_lambda_function.function.runtime
    alias      = aws_lambda_alias.alias.name
    invoke_arn = aws_lambda_alias.alias.invoke_arn
  }
}
```

## CI Process Workflow

![PATIENCE](https://www.obytes.com/assets/posts/2021-09-29-aws-lambda-ci/images/patience.gif)

Before writing the CI script, we need to design the workflow of the process.

![Lambda CI workflow](https://www.obytes.com/assets/posts/2021-09-29-aws-lambda-ci/images/lambda-ci-workflow.svg)

#### Check

In the first stage we checkout the new packages' descriptor, and check if the previous packages' descriptor exists in
**`S3`**, which is an edge case for the first execution. And if the package descriptor exists we need to download it for
next stage.

We also checkout the new application code and current source code hash that will be used in the next stage!

#### Bundle

If it is the first build we can jump directly and build/package the dependencies in the packages' descriptor

Otherwise, if it is not the first build, and the previous descriptor is already cached in S3, we will compare the
previously cached descriptor with the new descriptor.

If they are not the same, then the descriptor is changed, and we will have to build and package the dependencies again,
or else we can jump directly to the next stage.

Next, we package the new application source code, and we calculate its hash and compare it with the current source code
hash. if they are different, then we have to push the new source code to S3.

> For building lambda application dependencies we will use **`lambci/lambda`** docker image which is a sandboxed local
> environment that replicates the live AWS Lambda environment almost identically – including installed software and
> libraries.

#### Uploading artifacts

After building and packaging application code and/or dependencies, it's time to upload the packaged files to S3. we need
to upload the artifacts in two different names:

- The first package should include the word **`latest`** in the artifact name.
- The second package should include the **`commit digest hash`** in the artifact name.

The reason for this redundancy is to ensure we don't update the existing Lambda code with broken packages, so the
previous lambda version will keep pointing to the previous code/dependencies, and the newer lambda version will point to
the new code with the new commit digest hash as the artifact name.

This is an example of the files that will be uploaded to S3:

- **APP version S3 KEY** - *`lambda-ci-demo/{hash}/app.zip`*
- **APP latest S3 KEY** - *`lambda-ci-demo/latest/app.zip`*
- **DEP version S3 KEY** - *`lambda-ci-demo/{hash}/deps.zip`*
- **DEP latest S3 KEY** - *`lambda-ci-demo/latest/deps.zip`*

> AWS Lambda size limit is 50mb when you upload the code directly to the Lambda service. However, if the code
> deployment package size is more than that, we have an option to upload it to S3 and download it while triggering the
> function invocation. That's why we are going to use S3 to support large deployment packages.

#### Deploying dependencies and application

Now that we have the dependencies stored in S3, we can publish a new layer version that points to the new dependencies
with **`publish_layer_version(..)`** and update the lambda function to use the new layer version with
**`update_function_configuration(..)`**.

After a successful deployment, we cache the new packages' descriptor in order for subsequent runs to be faster.

After that, we have to deploy the actual lambda code, so we update the Lambda function to point to the newer code
package in S3 using **`update_function_code(..)`**.

#### Publishing application

Yes, we have deployed our Lambda function with newer dependencies and newer code but the services depending on it are
still pointing to the old Lambda function version tagged with **`latest`** alias, so we need to shift traffic to the
new version.

First, we should create a new lambda function version using **`publish_version(..)`** and then we have to update the
lambda alias to point to the newly created lambda version using **`update_alias(..)`**.

### Implementation

The stages in the above workflow are implemented in this package `https://github.com/obytes/aws-lambda-ci` and published
in a public PyPi repository, you can install it with **`pip`**:

```
pip3 install aws-lambda-ci
```

## Local CI with standalone CI Script

![automated\_ci](https://www.obytes.com/assets/posts/2021-09-29-aws-lambda-ci/images/automated_ci.jpeg)

In case you are still developing your function locally and you want to test it but you still don't have codebuild and
codepipeline resources in development environment, you can use **`aws-lambda-ci`** like this:

```
aws-lambda-ci \
--app-s3-bucket "demo-artifacts" \
--function-name "demo-api" \
--function-runtime "python3.9" \
--function-alias-name "latest" \
--function-layer-name "demo-api-deps" \
--app-src-path "src" \
--app-packages-descriptor-path "requirements.txt" \
--source-version "1.0.2" \
--aws-profile-name "kodhive_prd" \
--watch-log-stream
```

> First usage will take a longer time, because it needs to pull **`lambci/lambda`** from dockerhub.

### Code and dependencies changed

If both code and dependencies changed, the pipeline will publish both changes.

![demo\_code\_changed\_deps\_changed](https://www.obytes.com/assets/posts/2021-09-29-aws-lambda-ci/images/demo_code_changed_deps_changed.gif)

### Just code changed

If code changed but not dependencies, the pipeline will publish new code and the dependencies will be left intact.

![demo\_just\_code\_changed](https://www.obytes.com/assets/posts/2021-09-29-aws-lambda-ci/images/demo_just_code_changed.gif)

### Nothing changed

If both code and dependencies not changed, the pipeline will not publish anything.

![demo\_nothing\_changed](https://www.obytes.com/assets/posts/2021-09-29-aws-lambda-ci/images/demo_nothing_changed.gif)

## Leveraging codebuild/codepipeline for automated CI

![fun\_begins](https://www.obytes.com/assets/posts/2021-09-29-aws-lambda-ci/images/fun_begins.gif)

After preparing the python CI package, we can now configure codebuild and codepipeline to use it.

> **TLDR:** We've published a reusable terraform module for Lambda CI using Codepipeline/Codebuild, you can check it [here](https://github.com/obytes/terraform-aws-lambda-ci).

### Build specification

Let's start by creating the build specification file. It will be used by the codebuild project when the build is
triggered by codepipeline. In the **`install`** phase, we instruct codebuild to use runtimes shipped with docker.

In the **`pre-build`** phase, we get the source version that triggered the build, which can be a commit hash for pushes
events or SemVer tag for releases events and we install **`aws-lambda-ci`**.

Finally, during **`build`** phase, we call **`aws-lambda-ci`** that should take care of the next CI phases.

```yaml
version: 0.2

phases:
  install:
    runtime-version:
      docker: 19

  pre_build:
    commands:
      - SOURCE_VERSION=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION)
      - pip3 install aws-lambda-ci

  build:
    commands:
      - |
        aws-lambda-ci \
        --source-version "$SOURCE_VERSION" \
        --function-name "$FUNCTION_NAME" \
        --function-runtime "$FUNCTION_RUNTIME" \
        --function-alias-name "$FUNCTION_ALIAS_NAME" \
        --function-layer-name "$FUNCTION_NAME-deps" \
        --app-s3-bucket "$APP_S3_BUCKET" \
        --app-src-path "$APP_SRC_PATH" \
        --app-packages-descriptor-path "$APP_PACKAGES_DESCRIPTOR_PATH"
```

### Codebuild

After preparing the build specification file, we can now pass it to codebuild project along with environment variables
that will be forwarded to **`aws-lambda-ci`**.

It's important to set the source to **`CODEPIPELINE`** as codepipeline will trigger the codebuild project in response
to Github webhook and will prevent concurrent builds that can cause issues during deployment.

```hcl
resource "aws_codebuild_project" "default" {
  name          = local.prefix
  description   = "Build ${var.github_repository.branch} of ${var.github_repository.name} and deploy"
  build_timeout = var.build_timeout
  service_role  = aws_iam_role.role.arn

  source {
    type      = "CODEPIPELINE"
    buildspec = file("${path.module}/templates/buildspec.yml")
  }

  artifacts {
    type = "CODEPIPELINE"
  }

  cache {
    type  = "LOCAL"
    modes = ["LOCAL_DOCKER_LAYER_CACHE"]
  }

  environment {
    compute_type    = var.compute_type
    image           = var.image
    type            = var.type
    privileged_mode = var.privileged_mode

    # Bucket
    # ------
    environment_variable {
      name  = "APP_S3_BUCKET"
      value = var.s3_artifacts.bucket
    }

    # Build
    # -----
    environment_variable {
      name  = "APP_SRC_PATH"
      value = var.app_src_path
    }

    environment_variable {
      name  = "APP_PACKAGES_DESCRIPTOR_PATH"
      value = var.packages_descriptor_path
    }

    # Function
    # --------
    environment_variable {
      name  = "FUNCTION_NAME"
      value = var.lambda.name
    }

    environment_variable {
      name  = "FUNCTION_RUNTIME"
      value = var.lambda.runtime
    }

    environment_variable {
      name  = "FUNCTION_ALIAS_NAME"
      value = var.lambda.alias
    }

    environment_variable {
      name  = "FUNCTION_LAYER_NAME"
      value = "${var.lambda.name}-deps"
    }
  }

  tags = local.common_tags
}
```

### Codepipeline

Now that we have prepared the codebuild project, we will create the pipeline that will orchestrate the CI runs/stages.

During **`Source`** stage, we set up codepipeline to detect any change to target git branch if the environment is a
**`pre-release`** like **`qa`** or **`staging`**, checkout the lambda git repository and send it to next stage through
artifacts bucket.

In **`BuildAndDeploy`** stage, codepipeline will trigger codebuild and wait for codebuild execution to deploy and
publish the lambda function.

```hcl
resource "aws_codepipeline" "default" {
  name     = local.prefix
  role_arn = aws_iam_role.role.arn

  ##########################
  # Artifact Store S3 Bucket
  ##########################
  artifact_store {
    location = var.s3_artifacts.bucket
    type     = "S3"
  }

  #########################
  # Pull source from Github
  #########################
  stage {
    name = "Source"

    action {
      name             = "Source"
      category         = "Source"
      owner            = "AWS"
      provider         = "CodeStarSourceConnection"
      version          = "1"
      output_artifacts = ["code"]

      configuration = {
        FullRepositoryId     = "${var.github.owner}/${var.github_repository.name}"
        BranchName           = var.github_repository.branch
        DetectChanges        = var.pre_release
        ConnectionArn        = var.github.connection_arn
        OutputArtifactFormat = "CODEBUILD_CLONE_REF"
      }
    }
  }

  #########################
  # Build & Deploy to S3
  #########################
  stage {
    name = "BuildAndDeploy"

    action {
      name             = "BuildAndDeploy"
      category         = "Build"
      owner            = "AWS"
      provider         = "CodeBuild"
      input_artifacts  = ["code"]
      output_artifacts = ["package"]
      version          = "1"

      configuration = {
        ProjectName = var.codebuild_project_name
      }
    }
  }
}
```

### Release webhook

To support triggering the pipeline in response to github releases, we have to configure webhook resources that will only
be created if **`var.pre_release`** is set to **`false`**.

```hcl
# Webhooks (Only for github releases)
resource "aws_codepipeline_webhook" "default" {
  count           = var.pre_release ? 0:1
  name            = local.prefix
  authentication  = "GITHUB_HMAC"
  target_action   = "Source"
  target_pipeline = aws_codepipeline.default.name

  authentication_configuration {
    secret_token = var.github.webhook_secret
  }

  filter {
    json_path    = "$.action"
    match_equals = "published"
  }
}

resource "github_repository_webhook" "default" {
  count      = var.pre_release ? 0:1
  repository = var.github_repository.name

  configuration {
    url          = aws_codepipeline_webhook.default.0.url
    secret       = var.github.webhook_secret
    content_type = "json"
    insecure_ssl = true
  }

  events = [ "release" ]
}
```

### Slack Notifications

To receive slack alerts for pipeline's **`failure`** and **`success`** events, we will add two notification rules. the
first one will notify **`started`** and **`succeeded`** events and the second will notify **`failure`** events.

```hcl
resource "aws_codestarnotifications_notification_rule" "notify_info" {
  name        = "${local.prefix}-info"
  resource    = aws_codepipeline.default.arn
  detail_type = "FULL"
  status      = "ENABLED"

  event_type_ids = [
    "codepipeline-pipeline-pipeline-execution-started",
    "codepipeline-pipeline-pipeline-execution-succeeded"
  ]

  target {
    type    = "AWSChatbotSlack"
    address = "${local.chatbot}/${var.ci_notifications_slack_channels.info}"
  }
}

resource "aws_codestarnotifications_notification_rule" "notify_alert" {
  name        = "${local.prefix}-alert"
  resource    = aws_codepipeline.default.arn
  detail_type = "FULL"
  status      = "ENABLED"

  event_type_ids = [
    "codepipeline-pipeline-pipeline-execution-failed",
  ]

  target {
    type    = "AWSChatbotSlack"
    address = "${local.chatbot}/${var.ci_notifications_slack_channels.alert}"
  }
}
```

## Put it all together

![finally](https://www.obytes.com/assets/posts/2021-09-29-aws-lambda-ci/images/finally.gif)

We can structure the CI components into two modules, one for the build and another for the pipeline.

```hcl
module "code_build_project" {
  source             = "../../components/build"
  prefix             = local.prefix
  common_tags        = var.common_tags

  # S3
  s3_artifacts         = var.s3_artifacts
  lambda               = var.lambda

  # App
  app_src_path                = var.app_src_path
  packages_descriptor_path    = var.packages_descriptor_path

  # Github
  connection_arn    = var.github.connection_arn
  github_repository = var.github_repository
}

module "code_pipeline_project" {
  source      = "../../components/pipeline"
  prefix      = local.prefix
  common_tags = local.common_tags

  # Github
  github            = var.github
  pre_release       = var.pre_release
  github_repository = var.github_repository

  # S3
  s3_artifacts = var.s3_artifacts

  # Codebuild
  codebuild_project_name = module.code_build_project.codebuild_project_name

  ci_notifications_slack_channels = var.ci_notifications_slack_channels
}
```

Finally, we will wrap the two modules into a single module and call it like this.

```hcl
module "lambda_ci" {
  source      = "git::https://github.com/obytes/terraform-aws-lambda-ci.git//modules/ci"
  prefix      = "demo-api-ci"
  common_tags = {env = "test", stack = "demos-ci"}

  lambda                   = module.lambda.lambda
  app_src_path             = "src"
  packages_descriptor_path = "requirements.txt"

  # Github
  s3_artifacts = {
    arn    = aws_s3_bucket.artifacts.arn
    bucket = aws_s3_bucket.artifacts.bucket
  }
  pre_release  = true
  github       = {
    owner          = "obytes"
    token          = "gh_123456789876543234567845678"
    webhook_secret = "not-secret"
    connection_arn = "[GH_CODESTAR_CONNECTION_ARN]"
  }
  github_repository = {
    name   = "demo-api"
    branch = "main"
  }

  # Notifications
  ci_notifications_slack_channels = {
    info  = "ci-info"
    alert = "ci-alert"
  }
}
```

## What's next?

Now that we have implemented our Lambda CI reusable pipeline, in the next part we will use it to deploy nodejs and
flask APIs.

Share the article if you find it useful! See you next time.

![party-over-star-wars](https://www.obytes.com/assets/posts/2021-09-29-aws-lambda-ci/images/party-over-star-wars.gif)