Deploy Wasabi using a remote Cassandra cluster and RDS

Deploy Wasabi using a remote Cassandra cluster and RDS

Jose López - 15 September 2018

13 minutes read

What's Wasabi?

Wasabi is a real-time, 100% API driven, A/B Testing platform. The platform offers companies and individuals the opportunity to perform experiments on the web, mobile and desktop for back-end and front-end, products and marketing.

Do you want to know more? Meet Wasabi, an Open Source A/B Testing Platform

Architecture behind Wasabi

Wasabi is formed by two tiers:

Data tier:

  • Cassandra
  • MySQL

Web/App tier:

  • Wasabi

Wasabi architecture diagram

The Architecture Behind Wasabi, an Open Source A/B Testing Platform

Deploying AWS basic infra

Source code used on this article is available on GitHub. Example variables are provided on

terraform.tvars
files for each provider. Modify that variables according to your needs.

S3 bucket for terraform state

First of all, we are going to create an S3 bucket for terraform state files. This will allow us to store terraform state remotely and use terraform remote state data sources.

Network

We need to deploy a VPC, with two private subnets and two public subnets, along with a NAT gateway and Internet Gateway.

  1. Clone OBytes Wasabi project:

     git clone https://github.com/obytes/terraform-wasabi 

  2. Go to

     terraform_wasabi/providers/aws/network/ 

  3.  terraform init && terraform apply

Deploying Wasabi data tier

Cassandra

Cluster specs

We are going to deploy a Cassandra cluster on AWS, using Terraform. We are going to use this Medium article: First experience deploying Cassandra on AWS as a guide to deploy our Cassandra cluster, and reuse most of the code. Many thanks to Bohdan Kalytka for publishing it along with source code used!

Our Cassandra cluster will be formed of 6 instances, of type

 m4.large 
, with EBS volumes (SSD type GP2) of 200GB each. Resources assignment for Cassandra clusters is a fuzzy topic. Nonetheless, most of the checked sources agreed on SSD volumes should be used, and
 m4.large 
instance types (2 vCPU, 8GB RAM, memory-optimized) are the smallest type suitable for production clusters.

We are going to deploy this cluster on

 us-east-1 
region, in 2 different AZ:
 us-east-1a 
and
 us-east-1c 
. We will have 2 seed nodes, and 4 non-seed nodes, distributed between both AZ.

Deploying Cassandra

First of all, we need to generate Cassandra custom AMI. We are going to use Packer for this, and Ansible for provisioning the AMI.

  1. Set environment variables for Terraform and Packer — AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION.

  2. Edit

     terraform_wasabi/modules/aws/cassandra/packer/cassandra_aws_ami.json 
    and add your public
     vpc_id 
    and
     subnet_id 
    (public subnet). Then run
     packer build cassandra_aws_ami.json 
    to generate your custom AMI with Cassandra installed.

  3. Copy AMI code generated and change the value of

     cassandra_ami 
    on
     terraform_wasabi/providers/aws/cassandra/terraform.tfvars 

  4. Go to

     terraform_wasabi/providers/aws/cassandra/ 

  5.  terraform init && terraform apply 

Now, you should be able to connect to your Cassandra cluster using cqlsh:

 cqlsh <load_balancer_DNS> -u cassandra -p cassandra 

You can change the password using this command:

 ALTER USER cassandra WITH PASSWORD 'new_password'; 

Note: If the load balancer show some Cassandra instances out of service, SSH into the instance and restart Cassandra service:

sudo service cassandra restart

MySQL

  1. Go to

     terraform_wasabi/providers/aws/mysql/ 

  2.  terraform init && terraform apply 

Deploying Wasabi app

Generating a custom AMI

For Wasabi, we have to generate an AMI with custom configuration. We will use Wasabi version

 1.0.20180226051442 
, since there's a bug after this version still unresolved: https://github.com/intuit/wasabi/issues/265.

We will create a

 t2.medium 
instance via AWS console, based on Ubuntu 16.04 AMI (
 ami-759bc50a 
). We create it on the VPC we previously created using terraform and on a public subnet. We will attach an EBS volume to this instance.

Creating SG and role for the instance

We are going to use terraform to create a SG and role/policy/profile for our instance. We are going to target the creation of these resources.

  1. Go to

     terraform_wasabi/providers/aws/wasabi/ 

  2.  terraform init 

  3.  terraform apply -target aws_security_group.wasabi_sg && terraform apply -target module.prod_wasabi_policy && terraform apply -target module.prod_wasabi_role && terraform apply -target module.prod_wasabi_profile 

Bootstrapping Wasabi

SSH to the instance we have just created, clone Wasabi repository and checkout version

 1.0.20180226051442
.

 git clone https://github.com/intuit/wasabi && cd wasabi && git checkout 1.0.20180226051442

Then, bootstrap Wasabi:

 ./bin/wasabi.sh bootstrap

And reboot the instance:

 sudo reboot

Configure pom.xml

Now, let's edit

 pom.xml
file and add remote URL for Cassandra/MySQL and config them.

 cd wasabi && vi pom.xml 

Change the following lines.

  <mysql.host>'rds_endpoint'</mysql.host>
  <mysql.port>3306</mysql.port>
  <mysql.dbName>wasabi</mysql.dbName>
  <mysql.username>wasabi</mysql.username>
  <mysql.password>'password'</mysql.password>
  <mysql.numPartitions>1</mysql.numPartitions>
  <mysql.minConnections>10</mysql.minConnections>
  <mysql.maxConnections>30</mysql.maxConnections>
  <mysql.Args>verifyServerCertificate=false&amp;useSSL=false&amp;requireSSL=false&amp;autoReconnect=true&amp;autoReconnectForPools=true</mysql.Args>

  <cassandra.username>cassandra</cassandra.username>
  <cassandra.password>'password'</cassandra.password>
  <cassandra.experiments.keyspaceName>wasabi_experiments</cassandra.experiments.keyspaceName>
  <cassandra.experiments.nodeHosts>'cassandra_elb_endpoint'</cassandra.experiments.nodeHosts>
  <cassandra.experiments.port>9042</cassandra.experiments.port>
  <cassandra.experiments.useSSL>false</cassandra.experiments.useSSL>
  <cassandra.local.dc>us-east</cassandra.local.dc>
  <cassandra.experiments.defaultReadConsistency>QUORUM</cassandra.experiments.defaultReadConsistency>
  <cassandra.experiments.defaultWriteConsistency>QUORUM</cassandra.experiments.defaultWriteConsistency>
  <cassandra.experiments.replicationFactor>3</cassandra.experiments.replicationFactor>
  <cassandra.events.nodeHosts>'cassandra_elb_endpoint'</cassandra.events.nodeHosts>
  <cassandra.events.port>9042</cassandra.events.port>

We set up

 defaultReadConsistency 
and
 defaultWriteConsistency 
to
 QUORUM 
and
 replicationFactor 
to 3. Explaining these values go beyond the scope of this article.

Apply Cassandra migrations

Now, we need to apply migrations to Cassandra DB. For this, we need to follow these steps.

  # Download Cassandra migration tool
  wget https://oss.sonatype.org/content/repositories/public/com/builtamont/cassandra-migration/0.9/cassandra-migration-0.9-jar-with-dependencies.jar
  export CASSANDRA_MIGRATION=/home/ubuntu/wasabi/cassandra-migration-0.9-jar-with-dependencies.jar
  export MIGRATION_SCRIPT=/home/ubuntu/wasabi/modules/repository-datastax/src/main/resources/com/intuit/wasabi/repository/impl/cassandra/migration

  # Install Cassandra 2.1
  echo "deb http://www.apache.org/dist/cassandra/debian 21x main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list
  curl https://www.apache.org/dist/cassandra/KEYS | sudo apt-key add - && sudo apt-key adv --keyserver pool.sks-keyservers.net --recv-key A278B781FE4B2BDA && sudo apt-get update && sudo apt-get install -y cassandra

  # Apply migrations
  CQLSH_USERNAME=cassandra CQLSH_PASSWORD=<password> CQLSH_HOST=<cassandra_elb_endpoint> bin/docker/migration.sh

Apply MySQL migrations

This step shouldn't be necessary, but Wasabi doesn't generate migrations properly when we are using an external MySQL DB. After finishing the installation, it shows this error: https://github.com/intuit/wasabi/issues/348

As a workaround, we installed Wasabi on Docker containers and created a dump of MySQL database. After that, we restored that dump and it worked. You can find the dump on this gist

Install MySQL client:

  sudo apt-get install mysql-client

Restore the file on your MySQL DB using this command.

  mysql -u wasabi -p --host=<mysql_endpoint> < wasabi_mysql_dump.sql

Build and start Wasabi

After all these steps, everything should be OK. So, the only steps left are build Wasabi app and start it! We need to build & run without unit tests, otherwise, the build fails.

  MYSQL_HOST=<mysql_endpoint> NODE_HOST=<cassandra_elb_endpoint> ./bin/wasabi.sh -t false build
  docker network create wasabi_nw # We need to create docker network prior to start or an error is raised
  MYSQL_HOST=<mysql_endpoint> NODE_HOST=<cassandra_elb_endpoint> ./bin/wasabi.sh -t false start:wasabi
  docker update --restart always wasabi-main # Configure wasabi container to autostart

Creating an AMI based on this instance

Now, we need to create an AMI based on this instance we've just configured. We can do this via AWS console: https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/tkv-create-ami-from-instance.html

Then, we copy the AMI ID and change

 wasabi_ami 
on
 terraform_wasabi/providers/aws/wasabi/terraform.tfvars 

Deploying Wasabi app

We will deploy an ALB for Wasabi and 2x instances on 2x different AZ.

  1. Go to

     terraform_wasabi/providers/aws/wasabi/ 

  2.  terraform apply 

All set! 🎉 You can reach Wasabi UI on the ELB endpoint and start to use it!

Comments