GO Serverless! Part 3 - Deploy HTTP API to AWS Lambda and Expose it via API Gateway

Hamza Adami
November 2nd, 2021 · 10 min read

If you didn’t read the previous parts, we highly advise you to do that!


  • Part 1: Serverless Architecture and Components Integration Patterns
  • Part 2: Terraform and AWS Lambda External CI

In the previous parts we have discussed the pros and cons of serverless architectures, different serverless tools and technologies, integration patterns between serverless components and more.

We’ve built the reusable Terraform modules that helped us decouple Lambda code and dependencies continuous-integration from Terraform and delegate it to external CI services like Codebuild and Codepipeline.

Throughout this part, we will build a simple Flask API, deploy it as a Lambda Function with the CI Terraform module we previously built, and Expose it through AWS API Gateway V2 as an HTTP API.

TLDR

If you are in a rush and you don’t have time to read the entire article, we’ve got your back!

TLDR

Our goal is to build a reusable Terraform module for provisioning all resources needed for a functional, low budget and serverless HTTP API, to achieve this we will leverage the Terraform modules from our last part, and we will build other modules, so this is what we need:

  1. AWS Lambda API Code compatible with API Gateway - an API built with any Restfull API framework like Flask API and Fast API, and able to adapt Lambda API Gateway events into HTTP Requests and HTTP Responses into API Gateway Responses. an AWS Lambda Fast API Starter and AWS Lambda Flask API Starter are provided.

  2. Codeless AWS Lambda Function - from the previous part, a reusable Terraform module for provisioning codeless Lambda resources in order for code/dependencies build and deployment to be delegated to an external CI/CD process.

  3. AWS Lambda Function CI/CD - from the previous part, a reusable Terraform module for provisioning pipeline resources that will take care of the Lambda code and dependencies continuous integration/deployment.

  4. AWS API Gateway HTTP API - a Terraform reusable module for provisioning an AWS API Gateway HTTP API that integrates with an Upstream Lambda Function, authorize and proxy requests.

  5. AWS API Gateway APIs Exposer - a Terraform reusable module for provisioning AWS API Gateway resources for a custom domain, mapping the domain with the API Gateway HTTP API and exposing the API through route53 or cloudflare records.

  6. An Identity as a Service Provider - any JWT compatible provider that API Gateway can integrate with for authorizing requests based on users JWT access tokens.

  7. AWS Lambda API - This is the main reusable Terraform module (The Brain!) which processes/trigger other modules that are listed above. It encapsulates Codeless Lambda Function(2), Lambda Function CI/CD(3) and API Gateway HTTP API(4).

The API Gateway APIs Exposer(5) is placed in a separate module because it’s more generic and it can be used to expose multiple API Gateway HTTP APIs(4) or even API Gateway Websocket APIs that we will discuss more in the next article.

The Lambda API Code compatible with API Gateway(1) and Identity as a Service Provider(6) components are your responsibility to build and configure but a AWS Lambda Fast API Starter and AWS Lambda Flask API Starter are provided to you for inspiration and demo.

Prerequisites

  • HTTP API application compatible with Lambda/AWS-APIGW (A starter app is provided)
  • Route53 or Cloudflare zone (That you own of course)
  • ACM certificate for your AWS API Gateway custom domain (For HTTPs)
  • Codestar connection to your Github account.
  • S3 bucket for holding CI/CD artifacts and Lambda Code/Dependencies
  • Firebase project or a project in any other IaaS providers.
  • Slack channel(s) for notifying success adn failure deployments.

Usage

  • Version Control your Lambda Function API source code in Github and Provision the AWS Lambda API(7)
1module "aws_flask_lambda_api" {
2 source = "git::https://github.com/obytes/terraform-aws-lambda-api.git//modules/api"
3 prefix = "${local.prefix}-flask"
4 common_tags = local.common_tags
5
6 # Lambda API
7 description = "Flask Lambda API"
8 runtime = "python3.7"
9 handler = "app.runtime.lambda.main.handler"
10 memory_size = 512
11 envs = {
12 FIREBASE_APP_API_KEY = "AIzaSyAbiq3L6lVT9TyM_Lik6C5rgSLEGCiqJhM"
13 AWS_API_GW_MAPPING_KEY = "flask"
14 }
15 policy_json = null
16 logs_retention_in_days = 3
17 jwt_authorization_groups_attr_name = "groups"
18
19 # CI/CD
20 github = {
21 owner = "obytes"
22 webhook_secret = "not-secret"
23 connection_arn = "arn:aws:codestar-connections:us-east-1:{ACCOUNT_ID}:connection/{CONNECTION_ID}"
24 }
25 pre_release = true
26 github_repository = {
27 name = "lambda-flask-api"
28 branch = "main"
29 }
30 s3_artifacts = {
31 arn = aws_s3_bucket.artifacts.arn
32 bucket = aws_s3_bucket.artifacts.bucket
33 }
34 app_src_path = "src"
35 packages_descriptor_path = "src/requirements/lambda.txt"
36 ci_notifications_slack_channels = {
37 info = "ci-info"
38 alert = "ci-alert"
39 }
40
41 # API Gateway
42 stage_name = "mvp"
43 jwt_authorizer = {
44 issuer = "https://securetoken.google.com/flask-lambda"
45 audience = [ "flask-lambda" ]
46 }
47 routes_definitions = {
48 health_check = {
49 operation_name = "Service Health Check"
50 route_key = "GET /v1/manage/hc"
51 }
52 token = {
53 operation_name = "Get authorization token"
54 route_key = "POST /v1/auth/token"
55 }
56 whoami = {
57 operation_name = "Get user claims"
58 route_key = "GET /v1/users/whoami"
59 # Authorization
60 api_key_required = false
61 authorization_type = "JWT"
62 authorization_scopes = []
63 }
64 site_map = {
65 operation_name = "Get endpoints list"
66 route_key = "GET /v1/admin/endpoints"
67 # Authorization
68 api_key_required = false
69 authorization_type = "JWT"
70 authorization_scopes = []
71 }
72 swagger_specification = {
73 operation_name = "Swagger Specification"
74 route_key = "GET /v1/swagger.json"
75 }
76 swagger_ui = {
77 operation_name = "Swagger UI"
78 route_key = "GET /v1/docs"
79 }
80 }
81 access_logs_retention_in_days = 3
82}
  • For FastAPI lovers, we’ve got your back:
1module "aws_fast_lambda_api" {
2 source = "git::https://github.com/obytes/terraform-aws-lambda-api.git//modules/api"
3 prefix = "${local.prefix}-fast"
4 common_tags = local.common_tags
5
6 # Lambda API
7 description = "Fast Lambda API"
8 runtime = "python3.7"
9 handler = "app.runtime.lambda.main.handler"
10 memory_size = 512
11 envs = {
12 FIREBASE_APP_API_KEY = "AIzaSyAbiq3L6lVT9TyM_Lik6C5rgSLEGCiqJhM"
13 AWS_API_GW_MAPPING_KEY = "fast"
14 }
15 policy_json = null
16 logs_retention_in_days = 3
17 jwt_authorization_groups_attr_name = "groups"
18
19 # CI/CD
20 github = {
21 owner = "obytes"
22 webhook_secret = "not-secret"
23 connection_arn = "arn:aws:codestar-connections:us-east-1:{ACCOUNT_ID}:connection/{CONNECTION_ID}"
24 }
25 pre_release = true
26 github_repository = {
27 name = "lambda-fast-api"
28 branch = "main"
29 }
30 s3_artifacts = {
31 arn = aws_s3_bucket.artifacts.arn
32 bucket = aws_s3_bucket.artifacts.bucket
33 }
34 app_src_path = "src"
35 packages_descriptor_path = "src/requirements/lambda.txt"
36 ci_notifications_slack_channels = {
37 info = "ci-info"
38 alert = "ci-alert"
39 }
40
41 # API Gateway
42 stage_name = "mvp"
43 jwt_authorizer = {
44 issuer = "https://securetoken.google.com/flask-lambda"
45 audience = [ "flask-lambda" ]
46 }
47 routes_definitions = {
48 health_check = {
49 operation_name = "Service Health Check"
50 route_key = "GET /v1/manage/hc"
51 }
52 token = {
53 operation_name = "Get authorization token"
54 route_key = "POST /v1/auth/token"
55 }
56 whoami = {
57 operation_name = "Get user claims"
58 route_key = "GET /v1/users/whoami"
59 # Authorization
60 api_key_required = false
61 authorization_type = "JWT"
62 authorization_scopes = []
63 }
64 site_map = {
65 operation_name = "Get site map"
66 route_key = "GET /v1/admin/endpoints"
67 # Authorization
68 api_key_required = false
69 authorization_type = "JWT"
70 authorization_scopes = []
71 }
72 openapi = {
73 operation_name = "OpenAPI Specification"
74 route_key = "GET /v1/openapi.json"
75 }
76 swagger = {
77 operation_name = "Swagger UI"
78 route_key = "GET /v1/docs"
79 }
80 redoc = {
81 operation_name = "ReDoc UI"
82 route_key = "GET /v1/redoc"
83 }
84 }
85 access_logs_retention_in_days = 3
86}
  • Provision The AWS API Gateway APIs Exposer(5)
1module "gato" {
2 source = "git::https://github.com/obytes/terraform-aws-gato.git//modules/core-route53"
3 prefix = local.prefix
4 common_tags = local.common_tags
5
6 # DNS
7 r53_zone_id = aws_route53_zone.prerequisite.zone_id
8 cert_arn = aws_acm_certificate.prerequisite.arn
9 domain_name = "kodhive.com"
10 sub_domains = {
11 stateless = "api"
12 statefull = "ws"
13 }
14
15 # Rest APIS
16 http_apis = [
17 {
18 id = module.aws_flask_lambda_api.http_api_id
19 key = "flask"
20 stage = module.aws_flask_lambda_api.http_api_stage_name
21 },
22 {
23 id = module.aws_fast_lambda_api.http_api_id
24 key = "fast"
25 stage = module.aws_fast_lambda_api.http_api_stage_name
26 },
27 ]
28 ws_apis = []
29}

With this configuration, our Lambda API Gateway final base URLs will be https://api.kodhive.com/flask/ for Flask API and https://api.kodhive.com/fast/ for Fast API and these endpoints will be exposed:

Flask Docs
Fast Docs

Demo time, Bring it on!

APIGW Ready

Demo Credentials

Demo

  • Public Endpoint [ALLOW]
1curl -X GET https://api.kodhive.com/flask/v1/manage/hc
2{
3 "status": "I'm sexy and I know It"
4}
  • Auth Endpoint [DENY]
1curl -X POST -F '[email protected]' -F 'password=not-secret' https://api.kodhive.com/flask/v1/auth/token
2{
3 "error": {
4 "code": "002401",
5 "title": "Unauthorized",
6 "message": "Access unauthorized",
7 "reason": "EMAIL_NOT_FOUND"
8 },
9 "message": "EMAIL_NOT_FOUND"
10}
  • Auth Endpoint [ALLOW]
1curl -X POST -F '[email protected]' -F 'password=not-secret' https://api.kodhive.com/flask/v1/auth/token
2{
3 "kind": "identitytoolkit#VerifyPasswordResponse",
4 "localId": "gf30eciYKjVJrA5XMHK0NKDbKeC2",
5 "email": "[email protected]",
6 "displayName": "Super Admin",
7 "registered": true,
8 "profilePicture": "https://img2.freepng.fr/20180402/ogw/kisspng-computer-icons-user-profile-clip-art-user-avatar-5ac208105c03d6.9558906215226654883769.jpg",
9 "refreshToken": "TOO_LONG_TOKEN",
10 "expiresIn": "3600",
11 "token_type": "bearer",
12 "access_token": "TOO_LONG_TOKEN"
13}
  • Private Endpoint [DENY]
1curl -X GET https://api.kodhive.com/flask/v1/users/whoami
2{"message":"Unauthorized"}%
  • Private Endpoint [ALLOW]
1curl -X GET https://api.kodhive.com/flask/v1/users/whoami -H "Authorization: Bearer NORMAL_USER_FIREBASE_JWT_TOKEN"
2{
3 "claims": {
4 "aud": "flask-lambda",
5 "auth_time": "1635015339",
6 "email": "[email protected]",
7 "email_verified": "true",
8 "exp": "1635018939",
9 "firebase": "map[identities:map[email:[[email protected]]] sign_in_provider:password]",
10 "groups": "[USERS ADMINS]",
11 "iat": "1635015339",
12 "iss": "https://securetoken.google.com/flask-lambda",
13 "name": "Hamza Adami",
14 "picture": "https://siasky.net/_AlWdFnwvbHwXoDeVk-4DrMKcmQajKIJ2z-maOkXsDfYNw",
15 "sub": "NcpGCnZ9B0cFDqRllYbtTYG8awE2",
16 "user_id": "NcpGCnZ9B0cFDqRllYbtTYG8awE2"
17 }
18}
  • Admin Endpoint [DENY]
1curl -X GET https://api.kodhive.com/flask/v1/admin/endpoints -H "Authorization: Bearer NORMAL_USER_FIREBASE_JWT_TOKEN"
2{
3 "error": {
4 "code": "002401",
5 "title": "Unauthorized",
6 "message": "Access unauthorized",
7 "reason": "Only ['ADMINS'] can access this endpoint"
8 },
9 "message": "Only ['ADMINS'] can access this endpoint"
10}
  • Admin Endpoint [ALLOW]
1curl -X GET https://api.kodhive.com/flask/v1/admin/endpoints -H "Authorization: Bearer ADMIN_USER_FIREBASE_JWT_TOKEN"
2{
3 "endpoints": [
4 {
5 "path": "/v1/manage/hc",
6 "name": "api.manage_health_check"
7 },
8 {
9 "path": "/v1/admin/endpoints",
10 "name": "api.admin_list_endpoints"
11 },
12 {
13 "path": "/v1/users/whoami",
14 "name": "api.users_who_am_i"
15 },
16 {
17 "path": "/v1/swagger.json",
18 "name": "api.specs"
19 },
20 {
21 "path": "/v1/docs",
22 "name": "api.doc"
23 },
24 {
25 "path": "/v1/",
26 "name": "api.root"
27 }
28 ]
29}

Notice that even though the request URL has the flask mapping key prefix on it and also the stage name is added in the background by APIGW to the HTTP path sent to Lambda Function but it still works! this is thanks to API Gateway Mapping that strips the mapping key flask from the original path in the background and the lambda function adapter that strips the stage name mvp before matching the APIGW path with Flask route path that does not have the mapping key nor the stage name.

It's working

That’s all. enjoy your low budget API. However, if you are not in a rush, continue the article to see how we’ve built it. you will not regret that.

“You take the blue pill, the story ends, you wake up in your bed and believe whatever you want to believe. You take the red pill, you stay in wonderland, and I show you how deep the rabbit hole goes.” Morpheus

Lambda Compatible API?

Source: AWS Lambda Flask API Starter Source: AWS Lambda Fast API Starter

Possible

You can make any API compatible with Lambda and API Gateway as long as it’s written in a supported Lambda runtime and respects the HTTP specifications and standards. for this article we will take Flask APIs as an example.

Well, Most of you knows how to build a Flask application, so we will skip that and go directly to the important part which is how to make the Flask application compatible with Lambda Function and capable of serving requests originated from API Gateway.

Structure

To achieve that, we will need an event/request adapter, The adapter will be responsible of the following:

  • Adapt the received API Gateway event and translate it to a WSGI environ that contains information about the server configuration and client request.

  • Create a WSGI HTTP request, with headers and body taken from the WSGI environment. Which has properties and methods for using the functionality defined by various HTTP specs.

  • Start a WSGI Application to process the request, dispatch it to the target route and return a WSGI HTTP response with body, status, and headers.

  • Adapt the WSGI HTTP Response to a format that API Gateway understand and return it.

1# src/app/runtime/lambda/flask_lambda.py
2import base64
3import sys
4
5from app.api.conf.settings import AWS_API_GW_STAGE_NAME
6from .warmer import warmer
7
8try:
9 from urllib import urlencode
10except ImportError:
11 from urllib.parse import urlencode
12
13from flask import Flask
14from io import BytesIO
15
16from werkzeug._internal import _to_bytes
17
18
19def strip_api_gw_stage_name(path: str) -> str:
20 if path.startswith(f"/{AWS_API_GW_STAGE_NAME}"):
21 return path[len(f"/{AWS_API_GW_STAGE_NAME}"):]
22 return path
23
24
25def adapt(event):
26 environ = {'SCRIPT_NAME': ''}
27
28 context = event['requestContext']
29 http = context['http']
30
31 # Construct HEADERS
32 for hdr_name, hdr_value in event['headers'].items():
33 hdr_name = hdr_name.replace('-', '_').upper()
34 if hdr_name in ['CONTENT_TYPE', 'CONTENT_LENGTH']:
35 environ[hdr_name] = hdr_value
36 continue
37
38 http_hdr_name = 'HTTP_%s' % hdr_name
39 environ[http_hdr_name] = hdr_value
40
41 # Construct QUERY Params
42 qs = event.get('queryStringParameters')
43 environ['QUERY_STRING'] = urlencode(qs) if qs else ''
44
45 # Construct HTTP
46 environ['REQUEST_METHOD'] = http['method']
47 environ['PATH_INFO'] = strip_api_gw_stage_name(http['path'])
48 environ['SERVER_PROTOCOL'] = http['protocol']
49 environ['REMOTE_ADDR'] = http['sourceIp']
50 environ['HOST'] = '%(HTTP_HOST)s:%(HTTP_X_FORWARDED_PORT)s' % environ
51 environ['SERVER_PORT'] = environ['HTTP_X_FORWARDED_PORT']
52 environ['wsgi.url_scheme'] = environ['HTTP_X_FORWARDED_PROTO']
53
54 # Authorizer
55 environ['AUTHORIZER'] = context.get('authorizer')
56 environ['IDENTITY'] = context.get('identity')
57
58 # Body
59 body = event.get(u"body", "")
60 if event.get("isBase64Encoded", False):
61 body = base64.b64decode(body)
62 if isinstance(body, (str,)):
63 body = _to_bytes(body, charset="utf-8")
64 environ['CONTENT_LENGTH'] = str(len(body))
65
66 # WSGI
67 environ['wsgi.input'] = BytesIO(body)
68 environ['wsgi.version'] = (1, 0)
69 environ['wsgi.errors'] = sys.stderr
70 environ['wsgi.multithread'] = False
71 environ['wsgi.run_once'] = True
72 environ['wsgi.multiprocess'] = False
73
74 return environ
75
76
77class LambdaResponse(object):
78 def __init__(self):
79 self.status = None
80 self.response_headers = None
81
82 def start_response(self, status, response_headers, exc_info=None):
83 self.status = int(status[:3])
84 self.response_headers = dict(response_headers)
85
86
87class FlaskLambda(Flask):
88
89 @warmer(send_metric=False)
90 def __call__(self, event, context):
91 response = LambdaResponse()
92 response_body = next(self.wsgi_app(
93 adapt(event),
94 response.start_response
95 ))
96 res = {
97 'statusCode': response.status,
98 'headers': response.response_headers,
99 'body': response_body.decode("utf-8")
100 }
101 return res

AWS API Gateway sends Requests HTTP Paths that already contains a stage name to Lambda Function and the Flask application will not be able to match the request with the available target routes.

To make sure all blueprints routes match the path sent from API Gateway, for each request the adapter will strip the API Gateway stage name AWS_API_GW_STAGE_NAME from the original HTTP Path.

The Terraform AWS Lambda API reusable modules will ensure that the same stage name is used for both AWS API Gateway and Flask Application so this logic can work.

Now that we have the adapter in place, we can now use it to create the Flask application and make it as our Lambda Function entrypoint, AKA the handler. additionally, we have to make sure that the handler is set to the Adapter Object which is in our case app.runtime.lambda.main.handler.

1# src/app/runtime/lambda/main.py
2from __future__ import print_function
3
4from app.blueprints import register_blueprints
5from .flask_lambda import FlaskLambda
6
7
8def create_app():
9 # Init app
10 app = FlaskLambda(__name__)
11 app.url_map.strict_slashes = False
12 app.config["RESTX_JSON"] = {"indent": 4}
13 register_blueprints(app)
14 return app
15
16
17handler = create_app()

The app is registering a root blueprint for our v1 root resource and 3 sub blueprints:

1# src/app/blueprints.py
2from app.api.api_v1 import api_v1_blueprint
3from app.api.internal.manage import manage_bp
4from app.api.internal.admin import admin_bp
5from app.api.routes.users import users_bp
6from app.api.routes.auth import auth_bp
7
8
9def register_blueprints(app):
10 # Register blueprints
11 app.register_blueprint(api_v1_blueprint)
12 app.register_blueprint(manage_bp)
13 app.register_blueprint(auth_bp)
14 app.register_blueprint(admin_bp)
15 app.register_blueprint(users_bp)

We are overriding the base_path because it will be used by flask_restx to generate swagger specification base URL, it should include the AWS_API_GW_MAPPING_KEY which is flask in our case in order for the browser to fetch /flask/v1/swagger.json correctly.

We are also adding an OAuth authorizer to the swagger UI, so we can authenticate and test our endpoints directly from swagger instead of using cURL.

1# src/app/api/api_v1.py
2import os
3
4from flask import Blueprint
5from flask_restx import Api
6
7from app.api.conf.settings import AWS_API_GW_MAPPING_KEY
8from app.api.docs import get_swagger_ui
9from app.api.errors.errors_handler import handle_errors
10
11api_v1_blueprint = Blueprint('api', __name__, url_prefix=f'/v1')
12
13
14class FlaskAPI(Api):
15
16 @Api.base_path.getter
17 def base_path(self):
18 """
19 The API path
20
21 :rtype: str
22 """
23 return os.path.join(f"/{AWS_API_GW_MAPPING_KEY}", "v1")
24
25
26authorizations = {
27 'oauth2': {
28 'type': 'oauth2',
29 'flow': 'password',
30 'tokenUrl': os.path.join(f"/{AWS_API_GW_MAPPING_KEY}", "v1/auth/token"),
31 'refreshUrl': os.path.join(f"/{AWS_API_GW_MAPPING_KEY}", "v1/auth/refresh"),
32 }
33}
34
35api_v1 = FlaskAPI(
36 api_v1_blueprint,
37 version='0.1.0',
38 title="Lambda Flask API Starter",
39 description="Fast API Starter, Deployed on AWS Lambda and served with AWS API Gateway",
40 doc="/docs",
41 authorizations=authorizations
42)
43api_v1.documentation(get_swagger_ui)
44handle_errors(api_v1)

Endpoints

Test

To test all use cases we will cook a public endpoint, an authentication endpoint, a private endpoint and an admin endpoint:

  • Public Endpoint: simple health check endpoint.
1# src/app/api/internal/manage.py
2import logging
3
4from flask import Blueprint
5from flask_restx import Resource
6
7from app.api.api_v1 import api_v1
8
9manage_bp = Blueprint('manage', __name__, url_prefix='/v1/manage')
10manage_ns = api_v1.namespace('manage', 'Operations related to management.')
11
12logger = logging.getLogger(__name__)
13
14
15@manage_ns.route('/hc')
16class HealthCheck(Resource):
17
18 def get(self):
19 """
20 Useful to prevent cold start, should be called periodically by another lambda
21 """
22 return {"status": "I'm sexy and I know It"}, 200
  • Auth Endpoint: OAuth Password Authentication Flow.
1# src/app/api/routes/auth.py
2import logging
3
4import requests
5from flask import Blueprint, request
6from flask_restx import Resource
7
8from app.api.api_v1 import api_v1
9from app.api.conf.settings import FIREBASE_APP_API_KEY
10from app.api.exceptions import LambdaAuthorizationError
11
12auth_bp = Blueprint('auth', __name__, url_prefix='/v1/auth')
13auth_ns = api_v1.namespace('auth', 'Operations related to auth.')
14
15logger = logging.getLogger(__name__)
16
17
18@auth_ns.route("/token")
19class Token(Resource):
20
21 def post(self):
22 base_path = "https://identitytoolkit.googleapis.com"
23 payload = {
24 "email": request.form["username"],
25 "password": request.form["password"],
26 'returnSecureToken': True
27 }
28 # Post request
29 r = requests.post(
30 f"{base_path}/v1/accounts:signInWithPassword?key={FIREBASE_APP_API_KEY}",
31 data=payload
32 )
33 keys = r.json().keys()
34 # Check for errors
35 if "error" in keys:
36 error = r.json()["error"]
37 raise LambdaAuthorizationError(
38 errors=error["message"]
39 )
40 # success
41 auth = r.json()
42 auth["token_type"] = "bearer"
43 auth["access_token"] = auth.pop("idToken")
44 return auth
  • Private Endpoint: whoami endpoint that returns to the calling user his JWT decoded claims.
1# src/app/api/routes/users.py
2import logging
3
4from flask import Blueprint, g
5from flask_restx import Resource
6
7from app.api.api_v1 import api_v1
8from app.api.decorators import auth
9
10users_bp = Blueprint('users', __name__, url_prefix='/v1/users')
11users_ns = api_v1.namespace('users', 'Operations related to users.')
12
13logger = logging.getLogger(__name__)
14
15
16@users_ns.route('/whoami')
17class WhoAmI(Resource):
18
19 @users_ns.doc(security=[{'oauth2': []}])
20 @auth(allowed_groups=['USERS', 'ADMINS']) # OR Simply @auth
21 def get(self):
22 """
23 Return user specific JWT decoded claims
24 """
25 return {"claims": g.claims}, 200
  • Admin Endpoint: returns to site admins the available Flask routes as a list.
1# src/app/api/internal/admin.py
2import logging
3
4from flask import Blueprint, current_app, url_for
5from flask_restx import Resource
6
7from app.api.api_v1 import api_v1
8from app.api.decorators import auth
9
10admin_bp = Blueprint('admin', __name__, url_prefix='/v1/admin')
11admin_ns = api_v1.namespace('admin', 'Operations related to administration')
12
13logger = logging.getLogger(__name__)
14
15
16def has_no_empty_params(rule):
17 defaults = rule.defaults if rule.defaults is not None else ()
18 arguments = rule.arguments if rule.arguments is not None else ()
19 return len(defaults) >= len(arguments)
20
21
22@admin_ns.route('/endpoints')
23class ListEndpoints(Resource):
24
25 @admin_ns.doc(security=[{'oauth2': []}])
26 @auth(allowed_groups=['ADMINS'])
27 def get(self):
28 """
29 Get flask app available urls
30 """
31 endpoints = []
32 for rule in current_app.url_map.iter_rules():
33 # Filter out rules we can't navigate to in a browser
34 # and rules that require parameters
35 if "GET" in rule.methods and has_no_empty_params(rule):
36 url = url_for(rule.endpoint, **(rule.defaults or {}))
37 endpoints.append({"path": url, "name": rule.endpoint})
38 return {"endpoints": endpoints}, 200

Authentication & Authorization

Authorization

Authentication

The public endpoint will be open for all users without prior authentication but how about the private and admin endpoints? They certainly need an authentication system in place, for that we will not reinvent the wheel, and we will leverage an IaaS (Identity as a Service) provider like Firebase.

We have agreed to use an IaaS to authenticate users but how can we verify the users issued JWT access tokens? fortunately, AWS API Gateway can take that burden as it is capable of:

  • Allowing only access tokens that passed the integrity check.
  • Verify that access tokens are not yet expired.
  • Verify that access tokens are issued for an audience which is in the whitelisted audiences list.
  • Verify that access tokens have sufficient OAuth scopes to consume the endpoints.

NOTE: audiences and scopes checks are Authorization checks and not Authentication checks

Authorization

Authorization is an important aspect when building APIs, so we want certain functionalities/endpoints to be available to only a subset of our users. to tackle this, there are two famous approaches Role Based Access Control (RBAC) and OAuth Scopes Authorization.

Role Based Access Control (RBAC)

Authorization can be achieved by implementing a Role Based Access Control (RBAC) model. where we assign each user a role or multiple roles by adding them to groups and then decorate each route with the list of groups that can consume it.

When using an Identity as a Service Provider like Auth0, Firebase or Cognito we have to make sure to assign users to groups and during user’s authentication, the JWT token service will embed the user’s groups into the JWT Access/ID tokens claims.

After authenticating to Identity Provider, the user can send its JWT access token to API Gateway that will verify the token integrity/expiration and dispatch the request with decoded JWT token to Lambda Function. Finally, the Lambda Function will compare user’s groups claim with the whitelisted groups at route level and decide whether to allow it or forbid it.

1# src/app/api/decorators.py
2from functools import wraps
3from typing import List
4
5from flask import g, request
6
7from app.api.conf import settings
8from app.api.exceptions import LambdaAuthorizationError
9from app.api.auth import decode_jwt_token
10
11
12def get_claims():
13 g.access_token = request.headers.get('Authorization').split()[1]
14 if settings.RUNTIME == 'LAMBDA':
15 g.claims = request.environ['AUTHORIZER']['jwt']['claims']
16 g.username = g.claims['sub']
17 g.groups = g.claims.get(settings.JWT_AUTHORIZATION_GROUPS_ATTR_NAME, "[]").strip("[]").split()
18 elif settings.RUNTIME == 'CONTAINERIZED':
19 g.claims = decode_jwt_token(g.access_token)
20 g.username = g.claims['sub']
21 g.groups = g.claims.get(settings.JWT_AUTHORIZATION_GROUPS_ATTR_NAME, [])
22 else:
23 raise Exception("No runtime specified, Please set RUNTIME environment variable!")
24
25
26def auth(_route=None, allowed_groups: List = None):
27 def decorator(route):
28 @wraps(route)
29 def wrapper(*args, **kwargs):
30 get_claims()
31 if allowed_groups:
32 if not g.groups:
33 raise LambdaAuthorizationError(
34 'The endpoint has authorization check and the caller does not belong to any groups'
35 )
36 else:
37 if not any(group in allowed_groups for group in g.groups):
38 raise LambdaAuthorizationError(f'Only {allowed_groups} can access this endpoint')
39 return route(*args, **kwargs)
40
41 return wrapper
42
43 if _route:
44 return decorator(_route)
45 return decorator

This approach comes with many benefits but also with drawbacks:

  • Requests will not be authorized at the API Gateway level, and they need to travel to Lambda Function to run authorization logic.

  • Authorization rules will be writen in backend code, which will be messy from a DevOps perspective but backend developers will favour that because they will have a better visibility when coding/debugging, and they will easily know who can call any endpoint without going to infrastructure code.

OAuth Scopes Authorization

The second approach is by using OAuth Scopes Authorization model, and for each functionality/route we have to:

  • Create an OAuth scope.
  • Assign to users the list of OAuth scopes that they can claim.
  • At AWS API Gateway level specify the list of OAuth scopes that the user should have at least one of them for the API Gateway to let it reach the Lambda Function API.

The advantages of this approach are:

  • The ability to change permissions scopes at Identity Provider and AWS API Gateway Level without changing/deploying new code.
  • Unauthorized requests will be revoked at API Gateway Level and before reaching the Lambda Function.

The Terraform AWS Lambda API module supports this authorization model and you can customize it using the module’s routes_definitions Terraform variable.

Deploy The Lambda Function

Source 1: Codeless AWS Lambda Function Source 2: AWS Lambda Function CI/CD

Deploy

To deploy our Lambda Flask Application we will leverage the reusable modules we built in the previous part, the first module will provision the Lambda Function and related resources and the second module will provision the External CI module:

1module "flask_api" {
2 source = "git::https://github.com/obytes/terraform-aws-codeless-lambda.git//modules/lambda"
3 prefix = "${local.prefix}-flask-api"
4 common_tags = local.common_tags
5
6 envs = {
7 RUNTIME = "LAMBDA"
8
9 # API
10 AWS_API_GW_STAGE_NAME = "mvp"
11
12 # Authentication/Authorization
13 JWT_AUTHORIZATION_GROUPS_ATTR_NAME = "groups"
14 }
15}
16
17module "flask_api_ci" {
18 source = "git::https://github.com/obytes/terraform-aws-lambda-ci.git//modules/ci"
19 prefix = "${local.prefix}-flask-api-ci"
20 common_tags = local.common_tags
21
22 # Lambda
23 lambda = module.flask_api.lambda
24 app_src_path = "src"
25 packages_descriptor_path = "src/requirements/lambda.txt"
26
27 # Github
28 s3_artifacts = {
29 arn = aws_s3_bucket.artifacts.arn
30 bucket = aws_s3_bucket.artifacts.bucket
31 }
32 github = {
33 connection_arn = "arn:aws:codestar-connections:us-east-1:[YOUR_ACCOUNT_ID]:connection/[YOUR_CONNECTION_ID]"
34 owner = "YOUR_GITHUB_USERNAME"
35 webhook_secret = "YOUR_WEBHOOK_SECRET"
36 }
37 pre_release = true
38 github_repository = {
39 name = "lambda-flask-api"
40 branch = "main"
41 }
42
43 # Notifications
44 ci_notifications_slack_channels = var.ci_notifications_slack_channels
45}

The RUNTIME environment variable is used by Flask Application to distinguish between LAMBDA and CONTAINERIZED runtimes so the application can work as a lambda or as a docker container.

The AWS_API_GW_STAGE_NAME environment variable as discussed earlier, It should be set to the same stage as the AWS API Gateway.

The JWT_AUTHORIZATION_GROUPS_ATTR_NAME environment is passed to the Flask application as the claim attribute name that will be used by authentication decorator for RBAC authorization.

The AWS API Gateway

Source: AWS API Gateway HTTP API

Home

The first thing we do is to create an API Gateway V2 HTTP API, we specify the selection expressions for method, path and api key, and we configure CORS to allow all headers, methods and origins.

1# modules/gw/gateway_http_api.tf
2resource "aws_apigatewayv2_api" "_" {
3 name = "${local.prefix}-http-api"
4 description = "Lambda HTTP API"
5 protocol_type = "HTTP"
6
7 route_selection_expression = "$request.method $request.path"
8 api_key_selection_expression = "$request.header.x-api-key"
9
10 cors_configuration {
11 allow_credentials = false
12
13 allow_headers = [
14 "*"
15 ]
16 allow_methods = [
17 "*"
18 ]
19 allow_origins = [
20 "*"
21 ]
22 expose_headers = [
23 "*"
24 ]
25 }
26
27 tags = local.common_tags
28}

Next, we create the integration with the upstream service which is in this case our Lambda Function, the integration type is set to AWS_PROXY for direct interactions between the client and the integrated Lambda function. and we’ve chosen INTERNET as connection type because the connection is through the public routable internet and not inside a VPC.

1# modules/gw/gateway_integrations.tf
2resource "aws_apigatewayv2_integration" "_" {
3 api_id = aws_apigatewayv2_api._.id
4 description = "Lambda Serverless Upstream Service"
5
6 passthrough_behavior = "WHEN_NO_MATCH"
7 payload_format_version = "2.0"
8
9 # Upstream
10 integration_type = "AWS_PROXY"
11 integration_uri = var.api_lambda.invoke_arn
12 connection_type = "INTERNET"
13 integration_method = "POST"
14 timeout_milliseconds = 29000
15
16 lifecycle {
17 ignore_changes = [
18 passthrough_behavior
19 ]
20 }
21}

For the authorizers, we need a JWT Authorizer because we will leverage a token-based Authentication and Authorization standard to allow an application to access our API. however, the routes can also support NONE for open access mode and IAM for authorization with IAM STS tokens generated by Cognito Identity Pools.

The JWT issuer(iss) and audience(aud) depends on the IaaS provider that you will use. in our case we are using Firebase. so these are the issuer and audience format:

For AWS Cognito:

For Auth0:

1# modules/gw/gateway_authorizers.tf
2resource "aws_apigatewayv2_authorizer" "_" {
3 name = "${var.prefix}-jwt-authz"
4 api_id = aws_apigatewayv2_api._.id
5 authorizer_type = "JWT"
6 identity_sources = ["$request.header.Authorization"]
7
8 jwt_configuration {
9 issuer = var.jwt_authorizer.issuer
10 audience = var.jwt_authorizer.audience
11 }
12}
13
14locals {
15 authorizers_ids = {
16 JWT = aws_apigatewayv2_authorizer._.id
17 IAM = null
18 NONE = null
19 }
20}

After that, we will create the most important resources which are the routes, we are using the for_each to create a route for every route definition element provided in routes definitions variable, the upstream target is our API Gateway Lambda integration and the authorization will be dynamically configured:

1resource "aws_apigatewayv2_route" "routes" {
2 for_each = var.routes_definitions
3 api_id = aws_apigatewayv2_api._.id
4
5 # UPSTREAM
6 target = "integrations/${aws_apigatewayv2_integration._.id}"
7 route_key = each.value.route_key
8 operation_name = each.value.operation_name
9
10 # AUTHORIZATION
11 authorizer_id = lookup(local.authorizers_ids, lookup(each.value, "authorization_type", "NONE"), null)
12 api_key_required = lookup(each.value, "api_key_required", false)
13 authorization_type = lookup(each.value, "authorization_type", "NONE")
14 authorization_scopes = lookup(each.value, "authorization_scopes", null)
15}

Now that we have prepared all the required resources for the API Gateway to integrate with Lambda Function we should add a permission rule to the Upstream Lambda Function allowing invocations from API Gateway:

1# modules/gw/permission.tf
2resource "aws_lambda_permission" "_" {
3 statement_id = "AllowExecutionFromAPIGatewayV2"
4 action = "lambda:InvokeFunction"
5 function_name = var.api_lambda.name
6 principal = "apigateway.amazonaws.com"
7 qualifier = var.api_lambda.alias
8 source_arn = "${aws_apigatewayv2_api._.execution_arn}/${aws_apigatewayv2_stage._.name}/*/*"
9}

Finally, we deploy our HTTP API to an API Gateway Stage, we’ve opted for automatic deploy to avoid manual deploys each time we change routes definitions:

1# modules/gw/gateway_stage.tf
2resource "aws_apigatewayv2_stage" "_" {
3 name = var.stage_name
4 api_id = aws_apigatewayv2_api._.id
5 description = "Default Stage"
6 auto_deploy = true
7
8 access_log_settings {
9 format = jsonencode(local.access_logs_format)
10 destination_arn = aws_cloudwatch_log_group.access.arn
11 }
12
13 lifecycle {
14 ignore_changes = [
15 deployment_id,
16 default_route_settings
17 ]
18 }
19}

The reusable Terraform Lambda APIGW module can be called like this:

1module "flask_api_gw" {
2 source = "git::https://github.com/obytes/terraform-aws-lambda-apigw.git//modules/gw"
3 prefix = local.prefix
4 common_tags = local.common_tags
5
6 stage_name = "mvp"
7 api_lambda = {
8 name = aws_lambda_function.function.function_name
9 arn = aws_lambda_function.function.arn
10 runtime = aws_lambda_function.function.runtime
11 alias = aws_lambda_alias.alias.name
12 invoke_arn = aws_lambda_alias.alias.invoke_arn
13 }
14 jwt_authorizer = {
15 issuer = "https://securetoken.google.com/flask-lambda"
16 audience = [ "flask-lambda" ]
17 }
18 routes_definitions = {
19 health_check = {
20 operation_name = "Service Health Check"
21 route_key = "GET /v1/manage/hc"
22 }
23 token = {
24 operation_name = "Get authorization token"
25 route_key = "POST /v1/auth/token"
26 }
27 whoami = {
28 operation_name = "Get user claims"
29 route_key = "GET /v1/users/whoami"
30 # Authorization
31 api_key_required = false
32 authorization_type = "JWT"
33 authorization_scopes = []
34 }
35 site_map = {
36 operation_name = "Get endpoints list"
37 route_key = "GET /v1/admin/endpoints"
38 # Authorization
39 api_key_required = false
40 authorization_type = "JWT"
41 authorization_scopes = []
42 }
43 swagger_specification = {
44 operation_name = "Swagger Specification"
45 route_key = "GET /v1/swagger.json"
46 }
47 swagger_ui = {
48 operation_name = "Swagger UI"
49 route_key = "GET /v1/docs"
50 }
51 }
52 access_logs_retention_in_days = 3
53}

Expose it!

Source: AWS API Gateway APIs Exposer

Expose

After deploying the Flask Application as a Lambda Function and integrate it with API Gateway, now we need to expose it to the outer world with a beautiful domain name instead of the ugly one generated by AWS.

We need these prerequisites before exposing our API:

  • AWS route53 or cloudflare zone.
  • AWS ACM Certificate for the subdomain that we will use with our API.
  • An A record pointing to APEX domain (the custom domain creation will fail otherwise).

If you already have these requirements let’s create our custom API Gateway domain which will replace the default invoke URL provided by API gateway:

1# modules/core-route53/gateway_domains.tf
2resource "aws_apigatewayv2_domain_name" "stateless" {
3 domain_name = "${var.sub_domains.stateless}.${var.domain_name}"
4
5 domain_name_configuration {
6 certificate_arn = var.cert_arn
7 endpoint_type = "REGIONAL"
8 security_policy = "TLS_1_2"
9 }
10}

After that we will create an API Gateway Mapping to map the deployed API/Stage with the custom domain and the mapping key for our Lambda API will be flask. count is used here to support multiple HTTP APIs in case we have multiple Lambda APIs.

1# modules/core-route53/microservices.tf
2resource "aws_apigatewayv2_api_mapping" "stateless_microservices" {
3 count = length(var.http_apis)
4 stage = var.http_apis[count.index].stage
5 api_id = var.http_apis[count.index].id
6 api_mapping_key = var.http_apis[count.index].key
7 domain_name = aws_apigatewayv2_domain_name.stateless.domain_name
8}

Consider the mapping key as API microservice namespace, for example in a shopping site we can have accounts, products and payments microservices, so we will have three APIs with three different mapping keys (prefixes), this will expose these URLs:

Finally, we create the DNS record pointing to our API Gateway target domain name:

1# modules/core-route53/records.tf
2resource "aws_route53_record" "stateless" {
3 zone_id = var.r53_zone_id
4 name = var.sub_domains.stateless
5 type = "A"
6
7 alias {
8 name = aws_apigatewayv2_domain_name.stateless.domain_name_configuration[0].target_domain_name
9 zone_id = aws_apigatewayv2_domain_name.stateless.domain_name_configuration[0].hosted_zone_id
10 evaluate_target_health = true
11 }
12}

If you prefer Cloudflare, the reusable module can also create Cloudflare records:

1# modules/core-cloudflare/records.tf
2resource "cloudflare_record" "stateless" {
3 zone_id = var.cloudflare_zone_id
4 name = var.sub_domains.stateless
5 type = "CNAME"
6 proxied = true
7 value = aws_apigatewayv2_domain_name.stateless.domain_name_configuration[0].target_domain_name
8}

The reusable AWS API Gateway APIs Exposer module can be called like this:

1module "gato" {
2 source = "git::https://github.com/obytes/terraform-aws-gato//modules/core-route53"
3 prefix = local.prefix
4 common_tags = local.common_tags
5
6 # DNS
7 r53_zone_id = aws_route53_zone.prerequisite.zone_id
8 cert_arn = aws_acm_certificate.prerequisite.arn
9 domain_name = "kodhive.com"
10 sub_domains = {
11 stateless = "api"
12 statefull = "live"
13 }
14
15 # Rest APIS
16 http_apis = [
17 {
18 id = module.flask_api_gw.http_api_id
19 key = "flask"
20 stage = module.flask_api_gw.http_api_stage_name
21 },
22 ]
23
24 ws_apis = []
25}

What’s next?

Share

HTTP APIs operates on stateless protocol and usually the client sends requests, and then the server responds with requested data. There is no generic way for the server to communicate with the client on its own. This is referred to as a one-way communication.

Next we will see how we will build a Serverless WebSocket API that works over persistent TCP communication, and it’s possible for both the server and client to send data independent of each other, This is referred to as bi-directional communication.

Share if you like the article and Stay tuned for the next article. it’s just the beginning!

More articles from Obytes

Getting started with GraphQL in Python with FastAPI and Ariadne

Generate a FullStack playground using FastAPI, GraphQL and Ariadne

October 5th, 2021 · 5 min read

GO Serverless! Part 2 - Terraform and AWS Lambda External CI

If you didn't read the previous part , we highly advise you to do that! In the previous part we have discussed the pros and cons of…

September 29th, 2021 · 8 min read

ABOUT US

Our mission and ambition is to challenge the status quo, by doing things differently we nurture our love for craft and technology allowing us to create the unexpected.