Provisioning a production-ready Amazon EKS Fargate cluster using Terraform
What is Amazon EKS?
Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service that you can use to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane or nodes. Kubernetes is an open-source system for automating the deployment, scaling, and management of containerized application workloads and helps standardize operations across your environments. In this article, we are going to demonstrate the provisioning of an EKS environment using IaC (Terraform).
This blog post will be part of a series of multiple articles to demonstrate the following
- Provisioning and deploying Fargate EKS via Terraform
- Managing AWS auth config to control access to K8S cluster via AWS IAM
- Implementing CI/CD process using AWS Codebuild and Codepipeline
Environment requirements
Tool | Version | Purpose |
---|---|---|
awscli | 2 | AWS command line tool |
terraform | >=1.1.3 | IaC |
helm | 3.0.x | Kubernetes packaging |
kubectl | >=1.21 | Kubernetes command line |
Helm Kubernetes packages
Chart Name | Namespace | Chart Version | Application Version | Docker Version |
---|---|---|---|---|
aws-load-balancer-controller | kube-system | aws-load-balancer-controller-1.3.3 | v2.3.1 | v2.3.1 |
Git repositories
- docker-nginx-sample : A Git repo hosting the K8S YAML config files, provisioning the Nginx application deployment, ALB, Nginx service and Nginx Ingress resources along with the required namespaces and service accounts.
- terraform-aws-eks : A Git repo hosting the Terraform config files provisioning AWS services such as VPC, ACM, EKS, KMS and the CI/CD process using Codebuild and Codepipline.
EKS Architecture
Lets get the party started
After the long introduction and overview, now is the time to get our hands dirty and start our deployment processes. For your convenience, we have hosted all the files used in this article on GitHub, there are two different Git repositories as shown above hosting the infrastructure Terraform files and the K8S config YAML files.
AWS Resource Creation
The Terraform Git repository will create the below resources to host the EKS Fargate Cluster:
VPC
The below module will create the required VPC and all its components based on AWS best practices. The configuration for this design includes a virtual private cloud (VPC) with a public subnet and a private subnet in each availability zone for the selected region along with NAT Gateways, Internet GW, and a custom routing table. All the backend/K8S resources will be created in the private subnets spanning multiple AZs for more scalability. For more information about how to customize this VPC module please check the README.
Note: we are tagging our public subnets with special Tag Keys required by the AWS ALB ingress controller. For more information, please visit the AWS Documentation for ALB Ingress requirements
module "vpc" {
source = "github.com/obytes/terraform-aws-vpc.git?ref=v1.0.6"
environment = var.environment
region = var.region
project_name = var.project_name
cidr_block = var.cidr_block
enable_dns_hostnames = var.enable_dns_hostnames
enable_nat_gateway = var.enable_nat_gateway
enable_internet_gateway = var.enable_internet_gateway
create_public_subnets = var.create_public_subnets
single_nat_gateway = var.single_nat_gateway
map_public_ip_on_lunch = true
additional_public_subnet_tags = {
"kubernetes.io/cluster/${join("-", [local.prefix, "backend"])}" = "shared"
"kubernetes.io/role/elb" = 1
}
additional_private_subnet_tags = {
}
}
ACM
Amazon Certificate is used by the ALB ingress controller to allow our Nginx application to listen on port 443/HTTPS
resource "aws_acm_certificate" "_" {
count = var.create_acm_certificate ? 1 : 0
domain_name = var.domain
subject_alternative_names = [join(".", ["*", var.domain])]
tags = merge(local.common_tags, tomap({ DomainName = var.domain, Name = local.prefix }))
validation_method = "DNS"
}
AWS EKS
There are a lot of Terraform EKS modules out there, especially the official terraform-aws-eks, but since this is my first AWS EKS project, I decided to create a module from scratch that fits our case study where all the pods will be hosted on Fargate. Also, I would like to mention that you can take advantage of eksctl.io, which is a simple CLI tool for creating and managing clusters on EKS - Amazon's managed Kubernetes service for EC2, written in Go, and using CloudFormation APIs.
IAM Roles and policies
The Kubernetes clusters managed by Amazon EKS make calls to other AWS services on your behalf to manage the resources that you use with the service. Before you can create Amazon EKS clusters, you must create an IAM role with the following IAM policies:
- AmazonEKSClusterPolicy: This policy provides Kubernetes the permissions it requires to manage resources on your behalf. Kubernetes requires
EC2:CreateTags
permissions to place identifying information on EC2 resources including but not limited to Instances, Security Groups, and Elastic Network Interfaces. - AmazonEKSVPCResourceController: This policy allows the role to manage network interfaces, their private IP addresses, and their attachment and detachment to and from network instances.
- AmazonEKSFargatePodExecutionRolePolicy: Before you create a Fargate profile you must specify a pod execution role for the Amazon EKS components that run on the Fargate infrastructure using the profile. This role is added to the cluster's Kubernetes Role based access control (RBAC) for authorization. This allows the kubelet that's running on the Fargate infrastructure to register with your Amazon EKS cluster so that it can appear in your cluster as a node.
- AWS ALB Controller IAM Policy and Roles: This is a set of policies required by the ALB Ingress controller to create the needed target groups, security groups and listing the ACM certificates provisioning of
elasticloadbalancing
which is controlled by K8S Ingress resources. For more information, you can refer to aws-load-balancer-controller installation docs
All of the above mentioned roles and policies are created by IAM Terraform file hosted on our repository.
EKS Security Groups
Once the cluster is created, a default security group will be created and associated with the cluster's security policy. However, as an additional security precaution, you can associate an additional security group to limit the open ports between the cluster and the nodes as shown bellow. For more information, please check sec-group-reqs
resource "aws_security_group" "cluster" {
name = join("-", [local.prefix, "sg"])
description = join(" ", [local.prefix, "node ECS service"])
vpc_id = element(module.vpc.vpc_id, 0)
tags = merge(local.common_tags, tomap({ "Name" = join("-", [local.prefix, "sg"]) }))
}
resource "aws_security_group_rule" "cluster" {
for_each = local.cluster_security_group_rules
security_group_id = aws_security_group.cluster.id
protocol = each.value.protocol
from_port = each.value.from_port
to_port = each.value.to_port
type = each.value.type
self = try(each.value.self, null)
ipv6_cidr_blocks = try(each.value.ipv6_cidr_blocks, null)
source_security_group_id = try(each.value.source_node_security_group, null)
cidr_blocks = try(each.value.cidr_blocks, null)
}
EKS Cluster
Below is the cluster resource where we set different parameters such as ClusterName, IAM Roles, VPC Network settings, and Kubernetes Control plain network settings such as CIDR block and public access strategy.
Note:
- Kubernetes Control Plane public accessibility doesn't interfere with pods public accessibility, this is to limit your private access for your Amazon EKS cluster's Kubernetes API server endpoint and limit or completely disable the public access from the internet. As an enhancement, we will improve the EKS module to add a feature to control the behavior of the Amazon EKS cluster's Kubernetes API network access. For more information, please check private-clusters
- Pods that run on Fargate are only supported on private subnets (with NAT gateway access to AWS services, but not a direct route to an Internet Gateway), so your cluster's VPC must have private subnets available.
resource "aws_eks_cluster" "_" {
name = join("-", [local.prefix, "backend"])
role_arn = aws_iam_role._.arn
vpc_config {
subnet_ids = module.vpc.prv_subnet_ids
endpoint_private_access = true
endpoint_public_access = true
security_group_ids = [aws_security_group.cluster.id]
}
kubernetes_network_config {
service_ipv4_cidr = var.kubernetes_cidr
}
encryption_config {
resources = ["secrets"]
provider {
key_arn = aws_kms_key._[0].arn
}
}
enabled_cluster_log_types = ["api", "audit"]
depends_on = [
aws_iam_role_policy_attachment._["eks_cluster"],
aws_iam_role_policy_attachment._["vpc_resource_controller"],
aws_iam_role_policy_attachment.eks_fargate["fargate_execution"],
aws_cloudwatch_log_group._,
aws_iam_role._,
aws_iam_role.eks_fargate_role
]
timeouts {
create = lookup(var.cluster_timeouts, "create", null)
delete = lookup(var.cluster_timeouts, "update", null)
update = lookup(var.cluster_timeouts, "delete", null)
}
version = "1.21"
}
EKS Addons
There are 3 required EKS Addons that need to be created:
- CoreDNS: it's a flexible, extensible DNS server that can serve as the Kubernetes cluster DNS. When you launch an Amazon EKS cluster with at least one node, two replicas of the CoreDNS image are deployed by default, regardless of the number of nodes deployed in your cluster.
- VPC CNI plugin for Kubernetes: Amazon EKS supports native VPC networking with the Amazon VPC Container Network Interface (CNI) plugin for Kubernetes. This plugin allows Kubernetes pods to have the same IP address inside the pod as they do on the VPC network.
- Kube-proxy: kube-proxy maintains network rules on each Amazon EC2 node. It enables network communication to your pods. kube-proxy is not deployed to Fargate nodes, so this is not needed in our deployment.
resource "aws_eks_addon" "this" {
for_each = { for k, v in local.cluster_addons : k => v }
cluster_name = aws_eks_cluster._.name
addon_name = try(each.value.name, each.key)
addon_version = lookup(each.value, "addon_version", null)
resolve_conflicts = lookup(each.value, "resolve_conflicts", null)
service_account_role_arn = lookup(each.value, "service_account_role_arn", null)
lifecycle {
ignore_changes = [
modified_at
]
}
tags = local.common_tags
}
Note: By default, CoreDNS is configured to run on Amazon EC2 infrastructure on Amazon EKS clusters. If you want to only run your pods on Fargate, complete the following steps.
kubectl patch deployment coredns \
-n kube-system \
--type json \
-p='[{"op": "remove", "path": "/spec/template/metadata/annotations/eks.amazonaws.com~1compute-type"}]'
Delete and re-create any existing pods so that they are scheduled on Fargate
kubectl rollout restart -n kube-system deployment coredns
IAM OIDC provider
Amazon EKS supports using OpenID Connect (OIDC) identity providers as a method to authenticate users to your cluster. OIDC identity providers can be used with, or as an alternative to AWS Identity and Access Management (IAM). OIDC will allow configuring authentication to your cluster, you can create Kubernetes roles
and clusterroles
to assign permissions to the roles, and then bind the roles to the identities using Kubernetes rolebindings
and clusterrolebindings
.
We will take advantage of the tls_certificate
Terraform data source to get information about the TLS certificates for the EKS cluster.
data "tls_certificate" "this" {
url = aws_eks_cluster._.identity[0].oidc[0].issuer
}
resource "aws_iam_openid_connect_provider" "oidc_provider" {
client_id_list = ["sts.${data.aws_partition.current.dns_suffix}"]
thumbprint_list = [data.tls_certificate.this.certificates[0].sha1_fingerprint]
url = aws_eks_cluster._.identity[0].oidc[0].issuer
tags = merge(
{
Name = "${aws_eks_cluster._.name}-irsa" },
local.common_tags
)
}
EKS Fargate profiles
The Fargate profile declares the pods to run on Fargate. This declaration is done through the profile’s selectors. Each profile can have up to five selectors that contain a namespace and optional labels. You must define a namespace for every selector. The label field consists of multiple optional key-value pairs. Pods that match a selector (by matching a namespace for the selector and all of the labels specified in the selector) are scheduled on Fargate. During the creation of a Fargate profile, you must specify a pod execution role for the Amazon EKS components that run on the Fargate infrastructure using the profile. This role is added to the cluster's Kubernetes Role-Based Access Control (RBAC)
Below is the Terraform resource that creates the different Fargate profiles, as you will see in a later section the below pod selectors are used within the K8S YAML config files to declare which pod will be mapped to which Fargate profile.
Profile Name | Namespace | Selector |
---|---|---|
CoreDNS | stg-eks-euwest1 | {k8s-app: kube-dns} |
stg-eks-euwest1 | default | {Application: stg-eks-euwest1-core} |
stg-eks-euwest1-core | stg-eks-euwest1 | {app.kubernetes.io/name: core} |
stg-eks-euwest1-kube | kube-system | |
stg-eks-euwest1-kube-system | kube-system | {app.kubernetes.io/name: aws-load-balancer-controller, app.kubernetes.io/instance: aws-load-balancer-controller} |
resource "aws_eks_fargate_profile" "_" {
for_each = { for k, v in local.fargate_profiles : k => v }
cluster_name = aws_eks_cluster._.name
fargate_profile_name = each.value.name
pod_execution_role_arn = aws_iam_role.eks_fargate_role.arn
subnet_ids = module.vpc.prv_subnet_ids
dynamic "selector" {
for_each = each.value.selectors
content {
namespace = selector.value.namespace
labels = lookup(selector.value, "labels", {})
}
}
}
Kubernetes config files
The Kubernetes Nginx YAML config files hosted on docker-nginx-sample is used to create the K8S environment to host the Nginx website, it consists of:
- namespace: creates the required namespaces used in Nginx deployment files
- nginx-deployment: the nginx application deployment file, here we define the required Fargate profile by defining the selector labels, CPU/Memory resources, replicas count and Docker image
apiVersion: apps/v1
kind: Deployment
metadata:
name: "core-deployment"
namespace: "stg-eks-euwest1"
spec:
replicas: 2
selector:
matchLabels:
app.kubernetes.io/name: "core"
template:
metadata:
labels:
app.kubernetes.io/name: "core"
spec:
containers:
- image: nginx
resources:
requests:
memory: "2Gi"
cpu: "1024m"
limits:
memory: "2Gi"
cpu: "1024m"
imagePullPolicy: Always
name: "core"
ports:
- containerPort: 80
- nginx-service: Nginx service to expose an application running on a set of pods as a network service.
apiVersion: v1
kind: Service
metadata:
name: "core-service"
namespace: "stg-eks-euwest1"
annotations:
alb.ingress.kubernetes.io/target-type: ip
spec:
ports:
- port: 80 # exposes the Kubernetes service on the specified port within the cluster
targetPort: 80 # is the port on which the service will send requests to, that your pod will be listening on
protocol: TCP
nodePort: 31254
type: NodePort
selector:
app.kubernetes.io/name: "core"
- nginx-ingress.yaml: Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster, Ingress gives Service externally-reachable URLs, load balanced traffic, terminate SSL / TLS, and offer name-based virtual hosting.
- alb-service-accounts: ALB Service accounts needed by the ALB ingress controller
- helm-installation: A Bash script to install the AWS Ingress controller using Helm Kubernetes packaging system, here we are passing the clusterName, VPCiD, replicas as parameters to Helm, we are using the helm installation method as we are going to deploy in Fargate. For more information, please check add-controller-to-cluster
#!/bin/bash
#================#
# ALB Controller #
#================#
echo "Adding required helm packages ...."
helm repo add eks https://aws.github.io/eks-charts
echo "Install the TargetGroupBinding CRDs if upgrading the chart via helm upgrade."
kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"
echo "Install the helm chart if using IAM roles for service accounts. "
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
-n kube-system \
--set clusterName=stg-eks-euwest1-backend \
--set serviceAccount.create=false \
--set serviceAccount.name=aws-load-balancer-controller \
--set vpcId=vpc-0e1e5b5db323d5764 \
--set replicaCount=1