On Wednesday, I went to see a talk titled Scaling docker microservices w/ Mesos & Kafka ~ Human API. I was super excited by the topic, as I’m also using Docker and Kafka in production. You’re actually using it by reading this blog! I was also excited because I used to work with one of the presenters Rob Gulewich at Joyent.
The last time I was in an OpenDNS office was 10 years ago in San Francisco when I was picking up an old donated server from David Ulevitch for a community co-location service I was involved in. Some of the organizers were Will Doherty and David Weekly. (The site is a zombie -- the colo shut down a long time ago)
The event was hosted at the OpenDNS offices in Vancouver as part of their “OpenLate YVR“ series of tech talks. This was the first time I’d visited their offices in Vancouver, located in the historic Royal Bank Building on Hastings.
Human API’s business is to provide an API which aggregates health data from many third-party sources, and normalizes it so it can be accessed from a single API. Sort of like Apple’s HealthKit, but in the cloud. Their MVP was built to aggregate data from various fitness tracking devices (Fitbit, Jawbone, Withings, etc.), but with their Series A investment, they are now integrating things like medical records data, which is where the money is. The founders are from Vancouver, relocated to San Francisco, but they have opened an engineering office here. It’s really an exciting company.
Chris and Rob talked about how they changed where they were running their microservices (mostly written in Node.js) from Elastic Beanstalk to Docker containers managed by Mesos. They use Jenkins to do automated deployment. The actual EC2 instances are constructed using Ansible, Terraform and Packer. Instead of using Elastic Load Balancer, which doesn’t really work well with Mesos, they wrote their own layer that uses Consul and confd to configure nginx instances.
The slides are available.
To tie everything together, all the microservices output logs to a Kafka cluster. This gives them the ability to scale and deal with large amounts of data. They can replay data if necessary. They can construct data pipelines. And they can use it to store metrics that can be fed into services such as Graphite and Grafana.
I love that they independently came up with a similar set of tools as what we’ve been using for our side project at Backblend. Our setup is a bit newer and simpler - I went with stock AWS CloudFormation and ECS instead of Mesos, but I built a Kafka cluster as well, with the intention of using it as the primary store for all the data we generate. I’ll be blogging more about that in the future.