# Deploy Wasabi using a remote Cassandra cluster and RDS.

- URL: https://www.obytes.com/blog/deploy-wasabi-cassandra-mysql
- Published: 2018-09-15
- Author: Jose López
- Topics: DevOps
- Summary: Wasabi is a real-time, 100% API driven, A/B Testing platform. We will see how to deploy it on AWS along with Cassandra and a MySQL RDS.

### What's Wasabi?

Wasabi is a real-time, 100% API driven, A/B Testing platform. The platform offers companies and individuals the opportunity to perform experiments on the web, mobile and desktop for back-end and front-end, products and marketing.

Do you want to know more? [Meet Wasabi, an Open Source A/B Testing Platform](https://medium.com/blueprint-by-intuit/open-sourcing-wasabi-the-a-b-testing-platform-by-intuit-a8d5abc958d)

### Architecture behind Wasabi

[Wasabi](https://github.com/intuit/wasabi) is formed by two tiers:

**Data tier:**

- Cassandra
- MySQL

**Web/App tier:**

- Wasabi

![Wasabi architecture diagram](https://cdn-images-1.medium.com/max/1600/0*yFkvfp2ogr0GeCkP.)

[The Architecture Behind Wasabi, an Open Source A/B Testing Platform](https://medium.com/blueprint-by-intuit/the-architecture-behind-wasabi-an-open-source-a-b-testing-platform-b52430d3fd80)

### Deploying AWS basic infra

[Source code](https://github.com/obytes/wasabi) used on this article is available on GitHub. Example variables are provided on `terraform.tfvars` files for each provider. Modify that variables according to your needs.

#### S3 bucket for terraform state

First of all, we are going to create an S3 bucket for terraform state files. This will allow us to store terraform state remotely and use [terraform remote state data sources](https://www.terraform.io/docs/providers/terraform/d/remote_state.html).

### Network

We need to deploy a VPC, with two private subnets and two public subnets, along with a NAT gateway and Internet Gateway.

- Clone OBytes Wasabi project: `git clone https://github.com/obytes/terraform-wasabi`
- Go to `terraform_wasabi/providers/aws/network/`
- `terraform init && terraform apply`

### Deploying Wasabi data tier

#### Cassandra

##### Cluster specs

We are going to deploy a Cassandra cluster on AWS, using Terraform. We are going to use this Medium article: [First experience deploying Cassandra on AWS](https://medium.com/devoops-and-universe/first-experience-deploying-cassandra-on-aws-4a1b93af311d) as a guide to deploy our Cassandra cluster, and reuse most of the code. Many thanks to Bohdan Kalytka for publishing it along with source code used!

Our Cassandra cluster will be formed of 6 instances, of type `m4.large`, with EBS volumes (SSD type GP2) of 200GB each. Resources assignment for Cassandra clusters is a fuzzy topic. Nonetheless, most of the checked sources agreed on SSD volumes should be used, and `m4.large` instance types (2 vCPU, 8GB RAM, memory-optimized) are the smallest type suitable for production clusters.

We are going to deploy this cluster on `us-east-1` region, in 2 different AZ: `us-east-1a` and `us-east-1c`. We will have 2 [seed nodes](https://www.quora.com/How-many-seed-coordinator-nodes-should-you-have-in-a-10-node-cluster-of-cassandra-Can-you-have-everyone-be-a-coordinator), and 4 non-seed nodes, distributed between both AZ.

##### Deploying Cassandra

First of all, we need to generate Cassandra custom AMI. We are going to use Packer for this, and Ansible for provisioning the AMI.

- Set environment variables for Terraform and Packer — AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION.
- Edit `terraform_wasabi/modules/aws/cassandra/packer/cassandra_aws_ami.json` and add your public `vpc_id` and `subnet_id` (public subnet).
- Then run `packer build cassandra_aws_ami.json` to generate your custom AMI with Cassandra installed.
- Copy AMI code generated and change the value of `cassandra_ami` on `terraform_wasabi/providers/aws/cassandra/terraform.tfvars`.
- Go to `terraform_wasabi/providers/aws/cassandra/`
- `terraform init && terraform apply`

Now, you should be able to connect to your Cassandra cluster using cqlsh:

```bash
cqlsh <load_balancer_DNS/> -u cassandra -p cassandra
```

You can change the password using this command: `ALTER USER cassandra WITH PASSWORD 'new_password';`

**Note: If the load balancer show some Cassandra instances out of service, SSH into the instance and restart Cassandra service:** `sudo service cassandra restart`

#### MySQL

- Go to `terraform_wasabi/providers/aws/mysql/`
- `terraform init && terraform apply`

### Deploying Wasabi app

#### Generating a custom AMI

For Wasabi, we have to generate an AMI with custom configuration. We will use Wasabi version `1.0.20180226051442`, since there's a bug after this version still unresolved: <https://github.com/intuit/wasabi/issues/265>.

We will create a `t2.medium` instance via AWS console, based on Ubuntu 16.04 AMI (`ami-759bc50a`). We create it on the VPC we previously created using terraform and on a public subnet. We will attach an EBS volume to this instance.

#### Creating SG and role for the instance

We are going to use terraform to create a SG and role/policy/profile for our instance. We are going to target the creation of these resources.

- Go to `terraform_wasabi/providers/aws/wasabi/`
- `terraform init`
- `terraform apply -target aws_security_group.wasabi_sg && terraform apply -target module.prod_wasabi_policy && terraform apply -target module.prod_wasabi_role && terraform apply -target module.prod_wasabi_profile`

#### Bootstrapping Wasabi

SSH to the instance we have just created, clone Wasabi repository and checkout version `1.0.20180226051442`.

```bash
git clone https://github.com/intuit/wasabi && cd wasabi && git checkout 1.0.20180226051442
```

Then, bootstrap Wasabi: `./bin/wasabi.sh bootstrap`

And reboot the instance: `sudo reboot`

#### Configure pom.xml

Now, let's edit `pom.xml` file and add remote URL for Cassandra/MySQL and config them.

```bash
cd wasabi && vi pom.xml
```

Change the following lines.

```xml
<mysql.host>'rds_endpoint'</mysql.host>
  <mysql.port>3306</mysql.port>
  <mysql.dbName>wasabi</mysql.dbName>
  <mysql.username>wasabi</mysql.username>
  <mysql.password>'password'</mysql.password>
  <mysql.numPartitions>1</mysql.numPartitions>
  <mysql.minConnections>10</mysql.minConnections>
  <mysql.maxConnections>30</mysql.maxConnections>
  <mysql.Args>verifyServerCertificate=false&amp;useSSL=false&amp;requireSSL=false&amp;autoReconnect=true&amp;autoReconnectForPools=true</mysql.Args>

  <cassandra.username>cassandra</cassandra.username>
  <cassandra.password>'password'</cassandra.password>
  <cassandra.experiments.keyspaceName>wasabi_experiments</cassandra.experiments.keyspaceName>
  <cassandra.experiments.nodeHosts>'cassandra_elb_endpoint'</cassandra.experiments.nodeHosts>
  <cassandra.experiments.port>9042</cassandra.experiments.port>
  <cassandra.experiments.useSSL>false</cassandra.experiments.useSSL>
  <cassandra.local.dc>us-east</cassandra.local.dc>
  <cassandra.experiments.defaultReadConsistency>QUORUM</cassandra.experiments.defaultReadConsistency>
  <cassandra.experiments.defaultWriteConsistency>QUORUM</cassandra.experiments.defaultWriteConsistency>
  <cassandra.experiments.replicationFactor>3</cassandra.experiments.replicationFactor>
  <cassandra.events.nodeHosts>'cassandra_elb_endpoint'</cassandra.events.nodeHosts>
  <cassandra.events.port>9042</cassandra.events.port>
```

We set up `defaultReadConsistency` and `defaultWriteConsistency` to `QUORUM` and `replicationFactor` to 3. Explaining these values go beyond the scope of this article.

#### Apply Cassandra migrations

Now, we need to apply migrations to Cassandra DB. For this, we need to follow these steps.

```bash
 # Download Cassandra migration tool
  wget https://oss.sonatype.org/content/repositories/public/com/builtamont/cassandra-migration/0.9/cassandra-migration-0.9-jar-with-dependencies.jar
  export CASSANDRA_MIGRATION=/home/ubuntu/wasabi/cassandra-migration-0.9-jar-with-dependencies.jar
  export MIGRATION_SCRIPT=/home/ubuntu/wasabi/modules/repository-datastax/src/main/resources/com/intuit/wasabi/repository/impl/cassandra/migration

  # Install Cassandra 2.1
  echo "deb http://www.apache.org/dist/cassandra/debian 21x main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list
  curl https://www.apache.org/dist/cassandra/KEYS | sudo apt-key add - && sudo apt-key adv --keyserver pool.sks-keyservers.net --recv-key A278B781FE4B2BDA && sudo apt-get update && sudo apt-get install -y cassandra

  # Apply migrations
  CQLSH_USERNAME=cassandra CQLSH_PASSWORD=<password> CQLSH_HOST=<cassandra_elb_endpoint> bin/docker/migration.sh
```

#### Apply MySQL migrations

This step shouldn't be necessary, but Wasabi doesn't generate migrations properly when we are using an external MySQL DB. After finishing the installation, it shows this error: <https://github.com/intuit/wasabi/issues/348>

As a workaround, we installed Wasabi on Docker containers and created a dump of MySQL database. After that, we restored that dump and it worked. You can find the dump on [this gist](https://gist.github.com/kstromeiraos/5cc84e07cc166033fd28d53d9e2eb1d0)

Install MySQL client: `sudo apt-get install mysql-client`

Restore the file on your MySQL DB using this command.

```bash
mysql -u wasabi -p --host=<mysql_endpoint> < wasabi_mysql_dump.sql
```

#### Build and start Wasabi

After all these steps, everything should be OK. So, the only steps left are build Wasabi app and start it! We need to build and run without unit tests, otherwise, the build fails.

```bash
MYSQL_HOST=<mysql_endpoint> NODE_HOST=<cassandra_elb_endpoint> ./bin/wasabi.sh -t false build
docker network create wasabi_nw # We need to create docker network prior to start or an error is raised
MYSQL_HOST=<mysql_endpoint> NODE_HOST=<cassandra_elb_endpoint> ./bin/wasabi.sh -t false start:wasabi
docker update --restart always wasabi-main # Configure wasabi container to autostart
```

#### Creating an AMI based on this instance

Now, we need to create an AMI based on this instance we've just configured. We can do this via AWS console: <https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/tkv-create-ami-from-instance.html>

Then, we copy the AMI ID and change `wasabi_ami` on `terraform_wasabi/providers/aws/wasabi/terraform.tfvars`

#### Deploying Wasabi app

We will deploy an ALB for Wasabi and 2x instances on 2x different AZ.

- Go to `terraform_wasabi/providers/aws/wasabi/`
- `terraform apply`

All set! 🎉 You can reach Wasabi UI on the ELB endpoint and start to use it!