C for Celerity
Speed is not just a trait: on the web it is a feature. An important feature. All too often do we find that the web application we are depending on in order to finish that document or meet that deadline is too sluggish to be workable.
At Wakoopa, we strive to make our products a joy and a breeze to use. This encompasses all our software (including the trackers to be installed on panelists’ computers), but most affected is our research dashboard, which is used by our clients.
In this article, I would like to shed some light on a development process that we went through earlier this year and that allowed us to greatly increase the speed of certain operations in the dashboard (as you may, hopefully, have noticed). Bear with me, as we will dive straight into some technical aspects (with accompanying mumbo-jumbo).
Our backend software, hosted on Amazon’s AWS, is mostly written in Ruby and uses the excellent Ruby on Rails framework. This makes our developers, and ultimately you too, very happy, as they can focus on the features they want to implement for you instead of building every minute detail from the ground up.
However, it turns out that in some cases we do want to obsess over details. Early this year we noticed that the real-time processing of data to be shown and graphed in the dashboard was growing out of hand. Our Ruby software, while elegant and functional, was struggling with the vast torrents of data that were thrown at it when calculating channel- or domain-metrics. Under the hood, we used standard JSON libraries to fetch data out of a database and perform calculations on the resulting data. At a certain point, the data will just not fit anymore. Furthermore, the JSON data was only using a tiny subset of JSON syntax, so using a full-blown parser is like using a monster truck to go grocery shopping. An Aygo will do (better).
Enter Go. This new language by Google caught our eye as being a promising new low-level language for developing high-performance server-based software in. It is a statically typed language that should appeal to programmers who love C and Python and would like to work with something in between. Essentially, it is poised to be a modern C.
We implemented all necessary features: fetching data from the database, decompressing it, parsing it using our custom linear JSON-parser and calculating metrics. And, of course, a handful of unit tests. Already, we were seeing huge performance gains over the previous solution. Unfortunately, however, the garbage collector of Go proved to be too immature at the time and we still kept running into memory problems.
After some consideration we decided to reimplement the Go code in C. As many batteries are not included with C this required us to write some more code (for strings and other data structures), but the fine-grained control over memory allocation and code execution were completely worth the investment. We now see performance of at least an order of magnitude over our initial solution and memory use is minimal. In other words, we went from calculations taking 20 seconds to 2.
Marten Klencke - Desktop Engineer
