It’s not rocket science

In 2014, Graydon Hoare, the creator of the Rust programming language, wrote a blog post entitled, “The Not Rocket Science Rule Of Software Engineering.” In the post, he desribes a CI system his team cobbled together circa 2001. The purpose of the system was to enforce one simple rule:

The Not Rocket Science Rule Of Software Engineering:

automatically maintain a repository of code that always passes all the tests

Lurking under that placid description is a simple concept, but it’s challenging to execute.

Why is this so hard

Most CI systems only guarantee that the tests pass in your branch. But that’s different than saying your tests will pass when your branch merges.

One reason for this mismatch may be integration friction: your branch diverged from the main branch over time.

It’s unsafe to merge changes in parallel in a conventional CI system. Semantic conflicts between patches can break the build post-merge. Imagine renaming a method in one branch while merging a patch that depends on that method in another.

The queue

Adopting a rebase-and-merge policy is a possible solution — where you rebase and merge all patches one at a time.

But processing a queue in serial is slow and won’t scale up as you add more developers.

Speculative future state

By enqueuing patches to rebase-and-merge, you know what the codebase will look like after each patch is merged. That work can be done before you test and before you merge.

At the end of last year, GitHub introduced the Merge Queue, a mirror of GitLab’s (premium-tier only) Merge Trains feature to do exactly this.

You can put your merge requests into a queue to be merged. Each merge request in the queue is combined with the merge requests in front of it. It then runs its tests as if it had already merged.

We can run the tests for all patches in the queue in parallel. Waiting to merge each patch until all the patches in front of it pass their tests.

If a patch fails, we remove it and re-run the tests for all the patches behind it in the queue—it’s not rocket science!

This is how the Merge Queue and the Merge Train work today.

Multiply by 𝑛

One of my former colleagues was fond of saying:

The only numbers that matter in computer science are 0, 1, and 𝑛.

The “not rocket science” problem gets more complicated when a production build is composed of 𝑛 repos.

At my work, we’ve yet to embrace the monorepo trend 😬. The result is a release to production composed of roughly 200 different repositories – some of which depend on one another in an ever-spidering graph of maddening dependencies.

A merge queue of merge queues

In 2012, the Openstack Infrastructure team started work on Zuul. What’s unique about Zuul is that it can assure that a build works, even when it’s composed of several repositories.

Zuul maintains what it calls “dependent pipelines.” Dependent pipelines can ensure merge requests across multiple repos get tested and merged together as if they were part of a single Merge Queue – a merge queue of merge queues.

Today it’s a decade after development began on Zuul — more than two decades after Graydon Hoare built his CI system. And the most prominent and best-funded software forges in the history of humanity are pretty gosh darn close to catching up.

Writing is thinking

If you want strength, you lift weights; if you want clear thinking, you write.

I don’t like writing. It was an epiphany to me that almost no one does. Wrestling the jumble of emotions, facts, and opinions floating in my mind into a blog post can be an exhausting and demoralizing process.

Writing is a harsh medium for your ideas. As William Zinsser remarked, “The ear makes allowances for grammar, syntax and transitions that the eye wouldn’t tolerate in print.” Writing is hard-mode for thinking.

But matching words with ideas is addictive and electrifying. You have to hunt for the perfect combination of words to describe and clarify your thinking. As you hold up new words against your existing ideas, you see your ideas in new ways.

This process repeats itself for as long as you have the patience to write and rewrite a sentence. And when I have that patience, I’m shocked at how neat and tidy my thoughts look on the page and in my mind.

Learning is teaching

Writing is a test of your understanding.

The Nobel Prize winning physicist Richard Feynman developed a technique for learning any topic. If you want to deeply learn something, try teaching it to someone else. It seems to me the next level of this technique is to try to teach it to an endless number of people via your writing.

Another goal of mine is to read more, and the best test of whether I understand what I’ve read is to write about it.

Mortimer J. Adler in How to Read a Book uses the term “verbalizing” to  describe the act of mindlessly quoting books without attempting to put what you’ve read into your own words. It’s liberating to recall and use an old idea in a new way – indeed, this is the origin of the term “liberal arts.”

Blogs are conversations

When you blog, you get to talk to brilliant people and (if you’re lucky) they volunteer to refine your ideas.

Benjamin Franklin’s autobiography (which reads like a series of blog posts) attributes much of his early success to his writing. The example that struck me was how he brought street sweeping to Philadelphia.

Franklin wished the street in front of his printing shop was cleaner, so he wrote a paper detailing the myriad advantages of clean streets. He sent a copy of his paper to all the shop owners in his neighborhood which convinced them to pool their money to pay for a street sweeper.

Using the feedback from this experience, Franklin wrote a bill for the Assembly to pave and clean the streets for the whole of Philadelphia.

As I read this anecdote, I realized how analogous it is to the tech industry. Beyond building and open-sourcing Kubernetes, Google evangelized it. They experimented with the ideas behind Kubernetes,  wrote about the experience in the Borg paper, encouraged others to do the same and over time changed everything about data centers.

Ideas that change everything are refinements. Experiment, write, get feedback, repeat. The most tried and proven mechanism of refining your ideas is to write about them.

Why I blog

Blogging is a virtuous cycle. First, you mediate the argument between your mind and the page to fully form your ideas and write them down for others. Then you release your ideas into the world where strangers will combine them with their own ideas and morph and refine them into new thoughts you never imagined.

The gift for your efforts might be new ideas to blog about. Or it might be the warm-fuzzies of perfectly capturing the zeitgeist. Or you may get nothing but the harsh and telling feedback of silence.

I find the endless struggle of zooming and enhancing the details of my ideas in exchange for these gifts to be worth the trade-off. And that’s why I blog.