Deploying Aerospike to AWS with Ansible
Thinknear's mobile ad buying platform requires a high performance, scalable, and highly available datastore. We use Aerospike on AWS because it fits our system requirements and its SSD performance is more cost effective for us compared to other products. Engineers need tools to help create new infrastructure, provision hosts, and roll out changes. We use Ansible because it’s easy to learn, easy to use, based on SSH, and extensible.
Together, Ansible managed Aerospike database clusters makes our engineers smile. Now we would like to share the same tools we use with the rest of the world.
That is why today we announce the release of three open source Ansible roles to deploy and manage Aerospike clusters on AWS.
The three Ansible Galaxy roles are:
- aerospike - An Ansible role that installs, configures, and runs Aerospike on CentOS and Amazon Linux.
- aerospike-aws - An Ansible role that launches a auto-healing and discovering Aerospike Cloudformation stack on AWS.
- aerospike-collectd - An Ansible role that installs, configures, and runs collectd with the Aerospike plugin on Amazon and CentOS Linux.
With these roles, Thinknear deploys many Aerospike clusters in different release stages, AWS subnets, and configuration versions. Versioned, tested, and reviewed playbooks save huge amounts of time managing Aerospike. We think others will find the roles useful too.
Of course, you could mix and match. Maybe all you need is a role to set up Aerospike. Then aerospike on its own is all you need. Roles are flexible like that.
I created a GitHub repo called ThinkNear/ansible-role-example to demonstrate how to use all three together. In this project you will find a playbook that includes all three roles and inventory variables to get you started.
Run the playbook for the first time and the aerospike-aws role creates a Cloudformation stack with the infrastructure you need to run a multi-node Aerospike cluster. The diagram below shows the resources used in the Cloudformation stack: A DNS record, five ENIs, a security group, a launch configuration, and an auto-scaling group.
Our stack uses AMI's already installed with Aerospike. The launch configuration defines a user data script that downloads the cluster's configuration file from S3, attaches an available ENI to the instances, and restarts Aerospike on instance start. This works great for us -- we don't do anything when an instance is turned off by Amazon and replaced by the ASG in the middle of the night. Aerospike starts configured and joins the cluster by listening for heartbeats from any of the ENI's IPs.
We also have the ability to easily upgrade Aerospike. All you do is update the inventory variables “aerospike_version” and “aerospike_aws_ami” when the latest versions are released. Run the playbook again. The launch configuration updates with the latest AMI. Now any new instances will start with the latest version ready to go.
The playbook run continues and the aerospike role upgrades each host. Ansible's rolling update steps through each host serially to keep the service available for our clients. Because the version changed, Ansible downloads the release from Aerospike, installs it on the host, restarts the service, waits for the service to reconnect with the cluster, and finally finish building indexes. Once the service is ready the playbook continues to rollout the deploy to the rest of the hosts. Perfect.
Finally, we need to monitor and alert on our Aerospike benchmarks like read and write throughput, transactions per second, client connections, and migrations. We like to install collectd on all of our hosts to measure memory, CPU, and more. The Aerospike team must like collectd too because they released an awesome aerospike-collectd plugin. This plugin adds every Aerospike metric available to collectd -- nice. Our Ansible role installs and configures collectd with this plugin enabled on Aerospike hosts. From collectd we forward our metrics to Librato where we set up alerts and charts. Checkout the GitHub repo for aerospike-collectd for more information if you want to integrate Librato with Aerospike too.
Try our roles and let us know what you think!