Time, Tools & Technical Debt
How not to modernise a codebase
We tried something and it didn’t work. That’s a common part of programming but we don’t often write about it. Teams are inspired to recount ambitious successes and postmortems of technical disasters are a sign of integrity, but these too, are usually success stories. “Something broke—but—we identified and fixed it”.
We spent much of the last two years trying to modernise the frontend of Kumu’s codebase, and ultimately, our slow progress caused us to abandon those efforts and contributed to the decision to rewrite Kumu instead. One of the inhibitors was CoffeeScript, and indirectly, the whole suite of tools that we use to turn code that isn’t-quite-JavaScript into JavaScript.
CoffeeScript
CoffeeScript (for anyone lucky enough not to remember) is a little language that provides some subjective syntactic sugar for JavaScript. It was created during a time when JavaScript had no arrow functions, no classes, no destructuring, no default parameters, no rest parameters, no spreading, no nullish coalescing, and no string interpolation. CoffeeScript provided all of the above and more. If you were going to write significant quantities of code, it was a decent quality of life improvement.
CoffeeScript often went hand-in-hand with Ruby on Rails, and that’s how it ended up becoming the first language in Kumu’s frontend codebase.
Most languages for frontend development need to be compiled into JavaScript, which means you’ll need a way to run a compiler. In those early days, our tools were Sprockets and the Rails Asset Pipeline. CoffeeScript goes in, JavaScript comes out. This was way before CommonJS modules and before bundlers, but you could still sort of stitch files together with some funky #= require comments. Still, the only way for code in one file to interact with code in another file was through global variables, and this was one of the defining features of the older parts of our codebase.
Webpack
Then along came Webpack, an imperfect harbinger of better days for modules and tools. It was an alternative to the Rails Asset Pipeline, creating non-global module boundaries with CommonJS (CJS) modules. Webpack also supported the concept of “loaders”—ways to transform arbitrary code into bundleable JavaScript. Armed with the “Coffee Loader” we added Webpack into our tooling and began a new chunk of caffeinated codebase.
Webpack was still a relatively new tool when we adopted it and a full migration from Sprockets would have been a gamble. Instead, Webpacked code lived in its own corner, and that’s where all the new code went. New code could talk to new code through requires and exports. New code could talk to old code through globals. Old code talking to new code? Well, that is a bit messy, the new code also needs to expose some globals. Not ideal, but we’d eventually move everything over, right?
Wrong. This setup was good enough that we ended up coasting for long enough that JavaScript had time to significantly improve as a language, implementing many of the syntactic niceties from CoffeeScript. There were fewer reasons to use CoffeeScript, and for new code we began using modern JavaScript features (along with some non-standard ones like JSX) adding Babel into our now burgeoning tool chain.
Our Webpack configuration grew to patch over some of the differences between the module formats we were using, and during this time we migrated all of our products to live in the same repository, in order to share Kumu’s core visualisation code. Monorepo-wide Webpack upgrades became a dreaded part of housekeeping, due to the complexity of the setup.
TypeScript
Ever turneth the wheel of frontend development, and we gradually began to adopt TypeScript in new projects. There was more contention around static types at the time, but these days it’s widely accepted that 20% more typing for 80% more confidence is an objectively good engineering tradeoff. Kumu is our primary tool, but it’s not our only tool. Of the other products we make, Sticky Studio, Undercurrent, and Weavr are all mature TypeScript codebases.
We’re sold on TypeScript, we have plenty of experience using it in production. Getting it into an existing codebase as big as Kumu, however, was a different challenge.
The story for incremental adoption is great. TypeScript is just JavaScript with some type annotations, so we began by sprinkling some types around the more modern parts of the Webpack side of our codebase. There was still friction where TS files imported completely untyped CoffeeScript files, but we were able to patch over that in many common cases by writing type declarations for CoffeeScript files.
It was a breath of fresh air to get some type safety into the codebase, but it wasn’t quite the strict TypeScript experience we were used to from our other codebases either. First, we couldn’t run the compiler in strict1 mode without creating lots of difficult work around the migration boundaries where TS and JS files interact. Second, we were still missing types for many of the bits of data that flow around Kumu’s frontend. Trying to fill in types for rabbit holes of object references was a trap we discovered in our earliest migration branches.
When types don’t line up, any is the easiest tool to reach for. In my experience, it’s also one of the few really poor design choices in the language. It’s easy to type and it looks harmless. It should instead be screaming “ALL BETS ARE OFF” in neon red. We opted to alias any as $FixMe which would serve as a clear indicator that we identified a missing or broken type, but fixing it would have been out of scope for whatever we were writing.
Decaffeination
Inspired by some wonderful write-ups on how Dropbox, Bugsnag, Benchling, and others, wrangled ambitious CoffeeScript migrations, we tried our hand at decaffeinate, a tool that turns CoffeeScript into readable JavaScript. Despite the battle-tested success of the tool, some of the quirks of our codebase and our Webpack setup meant that the automatic migration process didn’t go off without some significant hitches, generating code that lost a lot of clarity when translated from our idiosyncratic style of CoffeeScript.
With no way to confidently translate the whole codebase automatically, we began a painstaking file by file migration strategy.
We’d each take a CoffeeScript file, ensure it had unit tests, run decaffeinate to get JavaScript, clean up warnings and oddities by hand, convert it to loose TypeScript, check that the tests still pass, then apply more rigorous types. 50 lines of CoffeeScript would routinely turn into 200 or 300 lines of TypeScript, often due to the code in question being resistant to testing and requiring complex stubs for globals.
The line count inflation would have been reasonable if the code coming out the other side was rock solid, but time and time again, we introduced bugs during translation, because adding types steered us into adding safer checking branches, which conflicted with subtle edge case behaviours that would have been hard to infer and test from the original code. Many of these bugs made it through our end-to-end tests and we broke production in some fairly spectacular ways during this period. Sorry!
The direction of travel was good, but progress was Sisyphean and Sprockets was still doing its best to throw wrenches into many of our plans. Our codebase was still split (by this time, fairly evenly) between code on the Rails side, packaged by Sprockets, and code on the Webpack side, that could use modules and loaders. The only way to translate a CoffeeScript file under Sprockets, was to pull it across to the new side of the codebase, a process which invariably involved some hairy problems around the order in which global variables were declared.
We decided it would be good for our sanity to move everything from Sprockets to Webpack. We left the global variables alone, moved the asset directory over and replaced all of the Sprockets require comments with CJS require calls. There was a minor adventure in topological sorting to resolve some of the dependency chains that Webpack wouldn’t jive with, but somehow, miraculously, everything started working.
Major version upgrades to Webpack and Babel were surprisingly significant challenges during this process (including a particularly painful transition away from a plugin that smoothed over the Babel 5 changes to default exports at the CJS/ESM boundary). Normally a bundler would give you some kinds of static guarantees around module shape mismatch, but enough of our codebase still relied on globals that we sometimes didn’t notice errors until development runtime at best, and CI failures or bug reports at worst.
Then there’s also the inevitable, incidental firefighting which will always demand urgent attention:
Oh, hey AWS, what’s that? Our main database server is running outside a VPC and we need to move it into the VPC by June? Ok. I’ll get right on that.
Oh, not quite? the new database server was provisioned with a much smaller disk? That shouldn’t matter because our data is mounted onto a separate EBS volume with plenty of capacity. Then why is production down? Because our database stores gigabytes of indexes outside the mount point. Oh 🤦.
Oh, why is search down? Because a rogue crawler ignored our robots.txt and decided to start crawling paginated search results for generic terms, overloading our Elasticsearch cluster.
Oh, why are we getting huge bills from Twilio? Looks like our multi-factor authentication is being abused to send tens of thousands of SMS messages. Time to learn about “Toll Fraud” 🤷.
Oh, why is there huge latency when creating new projects? Ah, looks like our database is overloaded due to spammers automating new project creation so that they can share suspicious links into social media platforms, using Kumu as a proxy. Time to deal with them. Again.
Energy is a precious resource in small engineering teams. If someone burns out, you notice their absence in ways that you can’t paper over, and this kind of tedious but well-intentioned work saps energy. No one enjoys sitting down to translate CoffeeScript, or to debug stack traces from tools by diving deep into the source for dependencies of dependencies of dependencies in the node modules directory. We were working harder than ever, and yet Kumu looked and felt the same.
Ultimately, we paused our migration process due to a lack of energy, to focus on some user facing improvements. That catalysed our decision to start from scratch, but we learned some important lessons from trying and failing to modernise a large codebase.
Bet cautiously but move confidently. Incremental approaches are good for exploration, but once you make a decision see it through, don’t get stuck straddling two approaches.
Static types are here to stay. We are betting on TypeScript for the next ten years of Kumu. It’s not a perfect language, but it is good enough.
Stay close to standards. Every deviation from the languages that actually run in browsers is an explicit decision to give yourself more work in the future when the landscape changes again.
Unopinionated tools create work. Our enormous Webpack configuration is effectively a new tool that no one else uses. This kind of complexity encourages abstractions and each layer of disconnect is a fragmentation that makes it harder to reason about the overall behaviour.
Be pragmatic around how you spend energy. Teams need energy in reserve to work effectively. Burned out engineers can’t fight fires, and the drip drip drip of slow CoffeeScript translations is not restorative.
There won’t be any CoffeeScript in the next version of Kumu. A rewrite is a chance to wipe the slate clean, but that doesn’t count for much if we make the same kinds of mistakes again.
This should absolutely be the default and you should have to opt-out of strict type checks, not opt-in.