The internet is modular
Sometimes you can gain fresh perspective walking back through familiar first principles. I had this experience recently after taking a pilgrimage to MIT Press Bookstore and discovering Barbara van Schewick’s Internet Architecture and Innovation.
The gist of the book is that there are specific reasons why the internet is so generative, and they can be mapped back to architectural choices made early in its history. It’s an important book if you want to build new weblike things.
One of the key architectural principles of the internet is modularity. van Schewick devotes the opening chapters to it. What is modularity? How does it work? Why does it matter?
If you work in software, modularity is one of those concepts that is so familiar it might seem banal. But this book has been prompting me to slow down, reflect on modularity, and try to see it again with beginner’s mind.
What is modularity?
Modularity is a design principle that intentionally makes components highly independent (“loosely coupled”).
(Barbara van Schewick, 2010. Internet Architecture and Innovation.)
A system is modular when it is made up of smaller independent parts with identifiable boundaries.
When designing a modular architecture, system architects decompose the system in a way that minimizes dependencies among components.
(Barbara van Schewick, 2010. Internet Architecture and Innovation.)
A modular system can be decomposed and recomposed from component parts. In fact, the mathematical term for formally-pure modularity is compositionality — the property of being composable.
How do you tell if a system is modularized? The key test is if pieces can be added, removed, modified, or swapped without changing other pieces.
If a product is modular in design, its components can be designed independently. If it is modular in production, its components can be produced independently. If it is modular in use, users of the product can replace or “mix and match” components at a later stage.
(Barbara van Schewick, 2010. Internet Architecture and Innovation.)
Modularity is about interfaces
So a modular system is made up of independent parts. But these parts need some way to talk if we’re going to compose them into a system.
To make components independent, and to maintain their independence, modularity employs abstraction, information hiding, and a strict separation of concerns.
A modular approach to design distinguishes between two types of information. Information that is relevant to more than one module is completely specified as part of the architectural design process and is not allowed to change during detailed design. This information is available or “visible” to all designers, and is therefore called the architecture’s visible information. In particular, this includes all the information modules need to work together, such as the information necessary to use a module or otherwise interact with it. For each module, this information is completely defined in detailed interface specifications that describe all the interdependencies or points of interaction between that module and other modules and specify how they will be resolved. Thus, a module’s visible information provides an external view of the module that lets other designers treat the module as a black box, letting them abstract from the complexity of the module.
(Barbara van Schewick, 2010. Internet Architecture and Innovation.)
How modules talk to each other must be carefully specified to maintain independence. For example, lego rigorously defines the interface between pieces (“the dot”). If they didn’t, each piece would have to make up its own mind about the kind of connector it should present. Many pieces wouldn’t click together. Those that did would have to know about the shapes of other pieces, and design their connectors around those specific pieces. The pieces would no longer be independent.
Clearly interfaces are a crucial aspect of compositionality, and I suspect that interfaces are in fact synonymous with compositionality. That is, compositionality is not just the ability to compose objects, but the ability to work with an object after intentionally forgetting how it was built. The part that is remembered is the “interface”, which may be a type, or a contract, or some other high-level description.
(Jules Hedges, On Compositionality)
Interfaces go by many names: APIs, protocols, function signatures, contracts, etiquette, lego dots. Regardless, the design of the interface is pivotal to the success or failure of a modular system. The modular interface is the grammar that defines the expressive potential of the system alphabet.
Modules can evolve quickly
…information that affects only one module—the inside of the black box—is hidden from everyone except its designers; it is called the architecture’s hidden information. This information is not specified during the design of the architecture, but is determined during the detailed design phase; it is free to evolve within the framework provided by the architecture’s visible information until the detailed designing ends.
(Barbara van Schewick, 2010. Internet Architecture and Innovation.)
Because modules are black boxes, you can change anything within the module, provided it doesn’t change the interface. Modules give you the freedom to rapidly evolve the internals of a system without breaking any functionality.
Interfaces must evolve slowly
…On the flip side, the module interface cannot evolve quickly, or the system would break.
Modularity is not without costs. An architecture’s visible information is not allowed to change during detailed design and is difficult to change later. Thus, modularity makes it more difficult to experiment on the architecture’s visible information (such as a module’s interfaces) in order to facilitate experimentation on the architecture’s hidden information (such as a module’s internals). In a way, the inflexibility of the visible information is the price for the flexibility with respect to the architecture’s hidden information.
(Barbara van Schewick, 2010. Internet Architecture and Innovation.)
A protocol moves much more slowly than a platform. After 30+ years, email is still unencrypted; meanwhile WhatsApp went from unencrypted to full e2ee in a year. People are still trying to standardize sharing a video reliably over IRC; meanwhile, Slack lets you create custom reaction emoji based on your face. This isn’t a funding issue. If something is truly decentralized, it becomes very difficult to change, and often remains stuck in time.
(Moxie Marlinspike, 2022. Web3 First Impressions.)
This presents us with a dilemma: it is crucial to draw the boundaries of modularity in the right place. The boundaries of modularity determine which parts of the system can evolve rapidly, and which will be frozen in time. Draw the lines in the wrong place, and you’ll be stuck!
To some degree, we might understand the web’s performance problems through this lens. The modular boundaries of the web were drawn for networked documents, with content split into “pages”, and browser engines designed to lay out and scroll text.
As the basis of competition shifted appward, we found that the modular boundaries which had been drawn for docs were in the wrong place for apps! There are hard modular boundaries between pages, so we write single-page apps to hide them. The layout model of the web can reflow massive amounts of text, but this becomes a liability for animation, so we avoid it. Particular layout modules and and APIs make the web near impossible to parallelize.
But the interface boundaries of the web have been drawn. We’re stuck with them now. The only way forward is to muddle through, with occasional heroic (or dastardly) feats of engineering.
You could see something like Mighty as an attempt to redraw modular boundaries by putting all of the modules of the browser engine into a black box, and throwing computing power at that black box until the only module boundary that matters is the network boundary between client and server.
The cost of interfaces is also why “modularize all the things” can be a mistake. It proliferates interfaces, and limits your ability to rapidly evolve a system. Big black boxes can create space for innovation.
What to modularize is a design problem with no easy answers, and the right boundaries depend upon goals, values, and co-evolutionary fit within the larger environment.
Modularity is how the system itself evolves
I know you think it's idealistic but evolution only builds on open formats and protocols. That's how technology layers.
(Dave Winer, 2014. Young Technologists Love Lockin?)
Modules allow us to cordon off parts of a system so that they can evolve quickly, but they are also the mechanism by which the system itself evolves.
Long before DNA was discovered, it was known that the mechanism of heredity behind biological evolution must be digital, countable, modular. Why? Evolution happens in any system with mutation, heredity, selection, and you can’t have mutation unless it is possible to add and remove parts—modularity! So, modularity is a key attribute of evolving systems.
In The Nature of Technology, W. Brian Arthur notes that modular technology has all of these properties too!
Mutation ← Changing or replacing modules
Heredity ← Composing new modules from old
Selection ← Usefulness
Technology evolves. New technologies are composed recursively from old modules, then become modules in turn. Modules are like genes for invention.
Technology, once a means of production, is becoming a chemistry. We are shifting from technologies that produce fixed physical outputs to technologies whose main character is that they can be combined endlessly for fresh purposes.
(W. Brian Arthur, 2009. The Nature of Technology)
If you want to create open-ended systems, you need modularity.
Modularity in tools for thought
Here are some questions I am asking myself as I reflect on modularity…
What aspects of Subconscious should be black boxes? What aspects of the tools for thought space is undergoing rapid evolution? What parts of the system need the freedom to experiment? In what ways might this change as the space matures?
Where should we present protocols? What parts of the tools for thought space are well-understood? In what areas is it ok to evolve slowly? In what ways might this change as the space matures?
Markup is a protocol, whether specified or not. This is unavoidable. Even if your app doesn’t interface with other apps, it still talks with the rest of the world through copy/paste.
We want markup to present an interface that is useful for many different modules. This is my motivation behind Subtext, a plain text format with markdownish flavors that easily parses to a flat list of blocks.
Data formats are a protocol, whether specified or not. This is one reason we’ve been prioritizing Lindy formats like plain text. They compose well with other tools.