Don't fork the ecosystem
WE MUST BUILD ON A CLEAR SITE. The city of to-day is dying because it is not constructed geometrically. To build on a clear site is to replace the 'accidental' layout of the ground, the only one that exists today, by a formal layout. Otherwise nothing can save us.
(Le Corbusier, 1929, "The City of Tomorrow")
Why did the web centralize? Why is it slow? Why can’t CPython be parallelized? Why is null both an object and not an object? Why is null at all? Most of our software has been shaped by chance decisions made in haste by people who could not have predicted how the system would end up being used today. Some chance decisions become structural problems. They constrain the system, make it inefficient, hacky, slow.
If we could start over, and rebuild the system, knowing what we know now? We could fix these mistakes, swap out the security model, alter that layout algorithm, make the object threadsafe, avoid the billion dollar mistake. It would be fast, beautiful, elegant. Except…
You can’t rebuild an ecosystem
You can rebuild software. Software is just a machine. Swapping out modules and replacing code is no problem. We do it all the time. The impulse, then, is to use this engineering toolkit to fix the ecosystem.
We could scrape away the old system, the old city, with its layers of ugly history and compromised infrastructure, and replace it with clean geometric lines and rational choices. The city will continue, of course, just as before, only better, more efficient. People will come and go, work, sleep, shop, except these people will be happier, healthier, more beautiful. Like in this architectural model we made. See the little plastic people walking around the plaza?
Somehow this fantasy never quite works out, in cities or in software ecosystems.
Software can be rebuilt, because software is a machine. But a software ecosystem is not a machine. It is a living system. When we attempt to rebuild the ecosystem, we’re making a category error. We’re confusing the software for the ecological process unfolding around it.
In Ersilia, to establish the relationships that sustain the city's life, the inhabitants stretch strings from the corners of the houses, white or black or gray or black-and-white according to whether they mark a relationship of blood, of trade, authority, agency. When the strings become so numerous that you can no longer pass among them, the inhabitants leave: the houses are dismantled; only the strings and their supports remain. From a mountainside, camping with their household goods, Ersilia's refugees look at the labyrinth of taut strings and poles that rise in the plain. That is the city of Ersilia still, and they are nothing.
(Italo Calvino, 1972, Invisible Cities)
You can’t rebuild an ecosystem, just like you can’t rebuild the Amazon rainforest. You can only grow with it, or bulldoze it and start over from zero.
Forking Like a State
And that’s exactly what happens when you try to rebuild an ecosystem. Fixing structural issues means breaking changes. Breaking changes mean forking, and forking has costs:
When you fork an ecosystem, you kill the existing ecosystem's momentum, and start from zero on a new ecosystem.
This doesn’t stop us from trying. Software history is littered with attempts to fix past mistakes in our ecosystems:
Python3 forked the ecosystem by making breaking changes to fix a number of longstanding mistakes. Strings became unicode by default, print went from being a statement to a function, functions switched from using lists to generators. Good stuff! Yet these longstanding mistakes turned out to be structural, and the fixes difficult to adopt. Keystone frameworks like Django and NumPy lagged several years behind in adoption. It took even longer for adoption to propagate through the long tail of the ecosystem. Many developers remained stuck on Python2 due to dependency lock-in. Things remained bifurcated until Python2 was EOL’d in 2020, forcing a migration. The ecosystem finally caught up 12 years after Python3 was released.
JavaScript forked the ecosystem by introducing a syntax-breaking module system, ES modules (import/export). ES modules made significant improvements on Node’s grassroots module system, CommonJS modules (require()). Bindings in the new module system became static, making tree shaking possible. The breaking syntax change was also used as a wedge to upgrade the language in other ways. Nevertheless, it took years for browsers to widely support ES module syntax. Node.js took even longer. Build tooling became necessary for shipping code to the browser. Keystone NPM packages continue to use CommonJS, creating practical difficulties with code re-use. ES Modules are widely supported in browsers today, and Node has patched up most of the compatibility issues, but has taken about 10 years, and build tools have become entrenched as a permanent feature of using the language.
Deno forked the ecosystem by ditching node_modules and package.json. Deno aimed to simplify by shedding Node’s increasingly complex legacy baggage. They ditched build tooling and config by supporting TypeScript, ES Modules, URL-based imports, web standard APIs, and a sandboxed runtime. However, the lack of support for Node’s massive NPM ecosystem hindered adoption. Then Bun launched with full NPM compatibility, and focused on selling performance. It rapidly gained traction. Now Deno finds itself playing catch-up. Deno 2 introduced full NPM compatibility. It also launched JSR, a new open source and decentralizable module registry. This is wonderful, but it forks Deno’s ecosystem yet again, since developers now face a choice between NPM and JSR.
I’ve also attempted to fork, and for sillier reasons!
Subtext rationalized Markdown as a simple block-based syntax, side-stepping syntax ambiguities, and making it easier to parse into useful data structures. Then, a few months later, ChatGPT happened, and Markdown was their lingua franca. Rationalizing Markdown became a pointless exercise. LLMs are trained on and will generate Markdown for eternity. It’s immortal. I should have just gone with Markdown.
What can we learn from all this? Python, ES Modules, Deno, JSR all feel like justified forks to me. They attempt to move the status quo forward in significant ways. And yet every fork is justified. They all happen for good reasons. This doesn’t change the fact that the costs of forking are high, very high.
It seems to take about a decade for an ecosystem to recover from a fork. Even so, a lot of shrapnel remains. When the ecosystem finally pieces itself back together, it seldom looks the way we envisioned.
These examples are comparative successes. More often, our schemes to fix ecosystems simply fail. Forks turn ecosystem tailwinds into headwinds. You find yourself pushing against the ecosystem, trying to halt its natural momentum so you can point it in a new direction. Even if you you succeed in slowing it down, you’ll have to keep pushing to get it going again. We may rationalize the ecosystem, and kill it in the process.
Living systems layer
So, if not fix the ecosystem, then what? Add more layers!
Layer stuff on top
Let things live side-by-side
Allow opt-in
Rich Hickey has a nice bit about this: “Don’t be afraid of foo2()”.
foo2()?! Preposterous! But then again… is it? foo2() means foo() doesn’t have to break. They can live side-by-side. You can opt-in.
We have to resist the urge to remove the legacy foo(). Carrying forward old baggage offends our sense of aesthetics, but hey, that’s how evolved systems work. Chickens still carry around the gene for dinosaur teeth. This is because living systems have to be viable at every evolutionary stage. They can never pause, reset, or make breaking changes. The path of evolution is always through the adjacent possible.
Maintaining viability is another reason that systems tend to evolve into pace layers. Newer layers can be stacked on top of older ones without breaking them. Promises on top of callbacks, async/await on top of promises. Older, slow-moving code can remain in its layer, unbothered. Lizard brain, limbic system, neocortex. Layer on layer.
For a long time I worked on browsers—Firefox and Chrome—and people who work on these kinds of things have a saying: Don’t Break The Web.
Don’t Break The Web is the reason we have <marquee> and quirks mode. It is also why you can visit spacejam.com/1996, and it still works today.
We’ve been playing Don’t Break The Web for 30 years. We based it on another game called Don’t Break The Internet, which has been going on even longer.
Don’t Break is an infinite game. We win by keeping the game going, and we’re still going. Let's build more things like that.
Don’t fork the ecosystem
An ecosystem is the most valuable thing in the world. If you have an ecosystem, you have tailwinds behind everything you do. That’s because ecosystems are nonlinear systems. They refuse to fall into equilibrium. Living systems want to spiral upwards.
Life improves the closed system's capacity to sustain life. Life—all life—is in the service of life. Necessary nutrients are made available to life by life in greater and greater richness as the diversity of life increases.
(Frank Herbert, Dune)
Ecosystems are rare. They don’t emerge very often. Our ecosystem, the one on this planet—life—arose just once, as far as we know.
Evolution is no linear family tree, but change in the single multidimensional being that has grown to cover the entire surface of Earth. (Lynn Margulis)
So, when a living ecosystem does emerge, we have to cherish it, foster it, despite shortcomings, inelegance, ugliness. All living systems are ugly and a little bit eldritch… squishy. That’s how it is.