Deploying Boomi on AWS ECS

Ahmed AL-Haffar
November 12th, 2020 · 4 min read

At Obytes, we recently started using Boomi as Service Integration Platform. Our requirement was to run Boomi Atom in AWS, and we had interesting and fun challenges to overcome, from deployment, CI/CD, monitoring and alerting.

We decided to create this article in series of articles that aims to share our experiences when it comes to deploying Boomi in cloud services. In this this article, we will start with deploying Boomi in AWS ECS using Terraform. It’s worth mentioning that, there are other guides by Boomi on how to deploy Boomi Molecule on AWS that explain how to run Boomi in AWS using EC2 instances and AWS CloudFormation.

Things to take into consideration before starting:

  • You must have a valid Dell Boomi AtomSphere account to use the Boomi docker image. For information about getting a 30-day free trial account, see AtomSphere Editions.
  • Boomi docker image can be run in two modes, privileged and unprivileged mode. In our deployment we are going to work with the unprivileged for security risks associated with the privileged containers.
  • Boomi using systemd daemon and systemctl to start its services, for more information kindly check this article
  • There are some requirements we should meet in order to be able to run systemd in unprivileged mode, such as mounting /sys/fs/cgroup as R/O or R/W, /run and /tmp as tmpfs filesystem, for more information please refer to running-systemd-in-a-non-privileged-container and Boomi Atom DockerHub.
  • Since Boomi requires tmpfs volumes, we should use EC2 launch type as of writing this article the FARGATE container still lacks the support for tmpfs, for more information kindly refer to task_definition_parameters.
  • We are going to use AWS EFS mount points as it is required when working on Boomi Molecule and settings up and AWS CloudWatch logging which we will discuss in other articles.

High Level Diagram of the topology

HLD

Overview of the Infra and Docker Image

The Infrastructure

  • ECS Cluster: EC2 launch type with the required AutoScaling Group and launch configuration.
  • EFS: AWS EFS mount points to be used by the ECS Task definitions and the required assume_roles to access other services such as ECR,CW and ECR
  • AWS Secrets Manager: We are using the secrets manager to store the secrets and parameters please, refer to this repo for more information Boomi Atom
  • ECR: For hosting a docker image which we are going to build in a few seconds!

The Infra Repo

  • Covering VPC creation is not part of the article scope, but if you are looking for managing VPC via terraform, you can check the official terraform-aws-vpc
  • The repo is structured based on Providers, stacks and modules. For more details, kindly refer to repo
  • There are some required variables need to be addressed and it’s documented on the README.md
  • We are following a naming convention to tag our resources based on locals, please check the main.tf for more info.

The Docker Image

  • We are extending the boomi/atom docker image to install the required packages [awscli, jq] to export the AWS Secrets as ENV variables, kindly refer to Dockerfile for more details
  • entrypoint.sh file is used to export the secrets stored by AWS secrets manger on boot time, this is done by defining the SECRETS_ID env variable on aws ecs task definition and with the help of awscli and jq packages we are exporting the secrets stored on aws secrets manager, kindly check the entrypoint file for more details
    1"environment": [
    2 {
    3 "name": "SECRETS_ID",
    4 "value": "${aws_secretsmanager_secret.secrets.id}"
    5 },
    6......

Get Things Done!

After the long introduction and overview of the resources, let’s start to get things done! Our first step is to build the image using the Dockerfile hosted on the repo, we can do this by running the below command, the docker image will be tagged as boomi:latest, please note that the image name which used by our task-definition as shown in the following section are referred as var.env, for more information kindly check the Inputs section in our Boomi-AWS-ECS

1<repo_path>$ docker build -t boomi:latest .

After the image is built, we will switch to our terraform repo and execute terraform plan --target module.common, just be aware before running the terraform plan you should define the required variables mentioned in the README.md, once you are happy with the terraform plan and getting a clean one you can proceed with the terraform apply --target module.common. This module will prepare our AWS infra. with the required ECS Cluster, EC2 nodes, ECR repo, S3 bucket for logging and KMS Key which will be used to publish our local image to. Please refer to the below steps to upload local image to ECR.

1aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <AWS_ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com
2
3docker tag boomi:latest <AWS_ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/prd-boomi-useast1:latest
4
5docker push <AWS_ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/prd-boomi-useast1:latest

You can replace the latest tag to whatever suits you, but please note that we are referring to this image in our task-definition with a provider environment variable - please feel free to edit it based on your needs.

1"image": "${var.repository_url}:${var.common_tags["env"]}",
2 "name": "${var.container_name}",
3 "networkMode": "awsvpc",
4 "entryPoint": [
5 "sh", "/home/boomi/entrypoint.sh"
6 ],
7 "command": [
8 "/sbin/init"
9 ],

Now it’s the time to obtain a Boomi Atom installation token, this can be done by logging with your Boomi account and go to Manager -> Atom Management and clicking on the New button located at the right of the search bar on the left menu. No need to download the installer as we are going to build our docker image in the later section.

Atom Installation Token

By now we have all the required attributes needed by the Atom container to boot up. for more details about which attributes/secrets needed by the docker please refer to Boomi Atom Docker Hub - in our deployment we are going to use:

  • INSTALL_TOKEN - (Alternative to BOOMI_USERNAME and BOOMI_PASSWORD) Specifies a unique installer token. A token is valid only for the account in which it was generated. Tokens expire after a set amount of time ranging from 30 minutes to 24 hours.
  • BOOMI_ACCOUNTID - (Required with BOOMI_USERNAME and BOOMI_PASSWORD; invalid with INSTALL_TOKEN) Specifies your AtomSphere account ID
  • BOOMI_ATOMNAME - (Required) Specifies the name of the Atom that you are installing.
  • BOOMI_CONTAINERNAME - (Optional) Specifies a name for the Atom container that is different from the Atom name. This name will be displayed on the AtomSphere Atom Management page.
  • BOOMI_ENVIRONMENT_NAME
  • INSTALLATIONDIRECTORY - (Optional) Specifies the directory where the Atom will be installed. By default, the Atom is installed in /var/boomi/Atom.
  • URL : URL for the Boomi Platform
  • BOOMI_ENVIRONMENT_CLASS - Whether the environment is production or testing

Most of these attributes will be filled automatically by applying the terraform common module prd-boomi-useast1-params, EXCEPT the INSTALL_TOKEN and BOOMI_ACCOUNTID will be added manually as key/value pairs to the prd-boomi-useast1-secs on the console just not to keep any secrets in the terraform state.

After all the ENVs are set, we can go and run the terraform apply --target module.boomi to create the ECS Service, Task definition and all the required IAM roles. Once the terraform plan run successfully you should be able to see the atom online but un-attached on Atom Management Page in Boomi.

Un-Attached Atoms

Conclusion

I hope you find this article useful, feel free to submit your comments and I will be more than happy to answer your inquiries. I will be working on the Part 2 of this article for Logging and Monitoring Boomi using AWS CloudWatch.

More articles from Obytes

Setting up a Cloudflare Argo Tunnel on AWS Fargate.

A complete guide to set up a Cloudflare Argo Tunnel on AWS Fargate.

November 8th, 2020 · 2 min read

Authentication in React Native, Easy, Secure, and Reusable solution.

A guide to build a generic solution that handles most of the authentication use cases and easy to copy-paste in your next project.

August 17th, 2020 · 3 min read

ABOUT US

Our mission and ambition is to challenge the status quo, by doing things differently we nurture our love for craft and technology allowing us to create the unexpected.