Talk: Human API - Scaling docker microservices w/ Mesos & Kafka

On Wednesday, I went to see a talk titled Scaling docker microservices w/ Mesos & Kafka ~ Human API. I was super excited by the topic, as I’m also using Docker and Kafka in production. You’re actually using it by reading this blog! I was also excited because I used to work with one of the presenters Rob Gulewich at Joyent.

The last time I was in an OpenDNS office was 10 years ago in San Francisco when I was picking up an old donated server from David Ulevitch for a community co-location service I was involved in. Some of the organizers were Will Doherty and David Weekly. (The site is a zombie -- the colo shut down a long time ago)

The event was hosted at the OpenDNS offices in Vancouver as part of their “OpenLate YVR“ series of tech talks. This was the first time I’d visited their offices in Vancouver, located in the historic Royal Bank Building on Hastings.

Human API’s business is to provide an API which aggregates health data from many third-party sources, and normalizes it so it can be accessed from a single API. Sort of like Apple’s HealthKit, but in the cloud. Their MVP was built to aggregate data from various fitness tracking devices (Fitbit, Jawbone, Withings, etc.), but with their Series A investment, they are now integrating things like medical records data, which is where the money is. The founders are from Vancouver, relocated to San Francisco, but they have opened an engineering office here. It’s really an exciting company.

Chris MacDonald and Rob Gulewich

Presenters: Chris MacDonald and Rob Gulewich

Chris and Rob talked about how they changed where they were running their microservices (mostly written in Node.js) from Elastic Beanstalk to Docker containers managed by Mesos. They use Jenkins to do automated deployment. The actual EC2 instances are constructed using Ansible, Terraform and Packer. Instead of using Elastic Load Balancer, which doesn’t really work well with Mesos, they wrote their own layer that uses Consul and confd to configure nginx instances.

The slides are available.

IMG_3163

Another shot of the crowd. (I’d say the audience was 90%+ male, but I don’t think that’s anybody’s fault)

To tie everything together, all the microservices output logs to a Kafka cluster. This gives them the ability to scale and deal with large amounts of data. They can replay data if necessary. They can construct data pipelines. And they can use it to store metrics that can be fed into services such as Graphite and Grafana.

Slide: Just point it at Kafka

I love that they independently came up with a similar set of tools as what we’ve been using for our side project at Backblend. Our setup is a bit newer and simpler - I went with stock AWS CloudFormation and ECS instead of Mesos, but I built a Kafka cluster as well, with the intention of using it as the primary store for all the data we generate. I’ll be blogging more about that in the future.

IMG_3162

Pretty awesome talk! Human API is hiring in Vancouver and Redwood City. And if you’d like to ask me more about my Kafka setup, hit me up on Twitter.

Flux frameworks: Marty.js => Redux or Alt?

A few months back, I did some research into Flux frameworks for React.js, and I decided to try out Marty.js. I liked the documentation, the fact that it had some nice examples of how to build an Isomorphic/Universal app, and it had some opinions about how to hook in the asynchronous data fetching code. Compared to some of the other Flux frameworks, it was less biased towards ES6. I really like the new ES6 features, but there’s so much tooling involved.

So I added a Marty.js version to my todos-offline project on GitHub, which is a proving ground of sorts where I’ve conducted several experiments. I was able to successfully integrate PouchDB, and sync to CouchDB on Cloudant, as well as to the previous Backbone.js version.

But Marty.js is no longer the new hotness, there is a new JavaScript framework every 16 minutes, and James Hollingworth, the developer of Marty.js just announced that there will be no new major releases.

James recommends migrating to Alt, which was another alternative I looked at a few months back. Brian Leroux has been playing with it as well:

I just bumped into Brian at the Style and Class meetup here in Vancouver. He says he’s pretty happy with Alt and he’s been able to get it to do the hot reloading tricks.

James Hollingsworth also mentions that he’s been using Dan Abramov’s Redux, and that another Flux framework, flummox, had recently been abandoned in favour of it. I’d even seen it on Hacker News recently, so it seems to have a small amount of momentum. Confusingly, there is also a WordPress framework named Redux.

The docs for Redux have been recently improved, which is super important to me:

http://gaearon.github.io/redux/

With all that, I decided to invest the time and watch Dan’s React Europe talk:

Wow! That’s got me excited. He’s doing live coding inside the app running his slides. Crazy. I love it!

So I think I’ll try out Redux first, and see how that goes. My feeling is that there’s a bit of a community around it that cares about hot reloading.

I’ll update my blog as I make progress.

Questions? Comments? Hit me up on Twitter.

Rebooting my blog

I’ve had a vanity domain name for a long time. The whois record for jimpick.com says I registered my domain way back in May 1996.

Domains were pretty expensive back then – I think it was $75 US for 2 years. That was super expensive for me at the time. I had just turned 26, and I had just quit a steady job in a big IT department to start freelancing because I was so very excited about the Internet.

Let’s laugh at the first version of this site, thanks to the Wayback Machine:

Screenshot of jimpick.com from 1996 via Archive.org

Do you see?

  • Frames! The frame on the right had a menu, but it doesn’t display in modern browsers. :-(
  • A Java applet! When it worked, it displayed some spinning 3D Text that said “Java” (maybe I’m a WebGL hipster?)
  • Linux 2.0 on a 386DX/33. I started with Slackware before I switched to Debian. I was paying $200/month to co-locate my server at a small ISP in Penticton, BC, Canada. I had set up my own DNS, web server, email server, and was renting space out to friends and clients.
  • Comic Sans!

In a lot of ways, I had a better online web presence 19 years ago than I do today. With this new staticly built blog, I hope to build a foundation for the next 19 years.

I have plans to add comments to this new blog. In the meantime, you can share your thoughts with me on Twitter.