Layered protocols
Following up on The Internet is Modular, and Compositional Components, Emergent Protocols, a few more thoughts on modularity, this time on layering:
Layering is a special form of modularity that restricts interactions between modules. A module assigned to a layer can use any of the modules in the same or in a lower layer; however, it cannot use a module belonging to a higher layer.
(Barbara van Schewick, 2010. Internet Architecture and Innovation.)
A modular design breaks a system down into units with boundaries that communicate over protocols, but does not specify how these modules should be arranged structurally. Layered design goes a step further, organizing modules into levels with distinct responsibilities.
Each layer solves a problem. Low-level layers address general fundamentals. Layers above construct abstractions over layers below, composing concepts at higher and higher levels, until we climb the ladder of abstraction all the way to end-user applications.
By basing design on increasing levels of abstraction, layering greatly reduces complexity… Since each layer only uses the services of layers below, a layered system can be implemented and tested incrementally.
(Barbara van Schewick, 2010. Internet Architecture and Innovation.)
The internet is layered
Layering is one of the key design feature of the internet. The internet protocol stack is layered, with low-level networking primitives below, and high-level features like HTTP and domain names at the top.
The web itself adds a few more layers on top: URLs, HTTP, HTML.
For the web, [the fundamental] elements were, in decreasing order of importance, universal resource identifiers (URIs), the Hypertext Transfer Protocol (HTTP), and the Hypertext Markup Language (HTML).
(Tim Berners-Lee, 1999, “Weaving the Web”)
Ok, so the internet is layered, but what’s so great about layering, anyway?
Layering allows evolution to proceed at multiple speeds
Ecologies work on a much longer timescale than proteins, and world economies change much more slowly than departments in a company.
(John H. Holland, 1987. Complex Adaptive Systems in SFI Bulletin.)
Evolving systems tend to separate into chronological layers, where lower is slower. Pace layering offers us a mental model through which to see this relationship.
Consider the differently paced components to be layers. Each layer is functionally different from the others and operates somewhat independently, but each layer influences and responds to the layers closest to it in a way that makes the whole system resilient.
From the fastest layers to the slowest layers in the system, the relationship can be described as follows:
Fast learns, slow remembers. Fast proposes, slow disposes. Fast is discontinuous, slow is continuous. Fast and small instructs slow and big by accrued innovation and by occasional revolution. Slow and big controls small and fast by constraint and constancy. Fast gets all our attention, slow has all the power.
All durable dynamic systems have this sort of structure. It is what makes them adaptable and robust.
(Stewart Brand, 2018. Pace Layering.)
Pace layering applies to the tech stack, too. Hardware might be seen as analogous to the nature layer, with bytecode, programming languages, APIs, frameworks, apps, each layering, one on top of the other.
The pace layering is partial—evolved layering is relaxed layering—but we can definitely see chronological shearing reflected by the cycle times of different layers of the stack. Hardware and programming languages tend to evolve on the decades timescale. Operating systems tend to release every few years. Apps, every month. Content is instant. Low-level protocols, like text encodings or WASM seem likely to exist on the order of centuries, and this is one reason we’ve been prioritizing Lindy formats like plain text in Subconscious.
But why is it that systems tend to evolve these distinct pace layers?
The stability of the network as a platform allows innovation on top of that platform…
(David D. Clark, 2018. Designing an Internet)
Lower layers act as stable platforms on which to build layers above. For a thing to act as a platform, it must necessarily move more slowly than the things above it. Otherwise you get earthquakes.
You don't want to change your credit card every month because of "security upgrades" of the underlying payment infrastructure, you don't want to change your car every year because it relies on a network technology that is moving too fast, and you don't want to change all your cables, adapters and chargers every week because the USB standard is evolving too fast.
Thus, products need to move fast and can sometimes break things, while platforms need to move slowly and keep backward compatibility.
(Sylvain Kerkour, 2022. Programming languages are platforms, not products.)
Layering allows for shearing
Layering also lets us evolve systems by shearing off layers and replacing them. Take a house. Some things change quickly—people, furniture, popular paint colors. Other things change more slowly—the foundation, studs, insulation, plumbing.
Long-lasting architecture is built so that layers which need to change quickly can be sheared off and replaced as needed. You don’t want to tear down the whole house to replace a lightbulb. Fast layers need to evolve more quickly, and a good design makes it correspondingly easy to peel off and replace.
Layering increases the modifiability of the system by shielding lower layers from changes in higher layers.
(Barbara van Schewick, 2010. Internet Architecture and Innovation.)
In protocol design, these shearing points are sometimes called “portability layers”. They let us upgrade a system piecemeal, instead of tearing it down and starting over again from scratch.
Layering also lets us remix layers of a system to build new systems. From Alphabets of Emergence:
The web was designed to be an alphabet… and HTML makes up an alphabet of its own. The pieces of these alphabets have been recombined to create new things TBL could not have anticipated or imagined. For example:
URLs + HTTP + HTML = web
URLs + HTTP + RSS = Podcasts
URLs + HTTP + JSON = REST APIs
URLs + P2P + HTML = Web3
Note we can construct radically different systems by swapping just the top layer or two.
Layering in Subconscious
Bringing it back to Subconscious, I wanted to share a few notes. This is early prototype stage stuff, so things may change, but here’s how we’re thinking about layering the stack for Subconscious multiplayer and sync.
Layer 1. IPFS, a hypertext protocol that is content-addressed, and peer-to-peer. IPFS sits at the same place in the stack as HTTP, but with a key difference. When you request a file over HTTP, you are routed to the location of a computer that serves you the file. On IPFS, by contrast, you ask for a file using the hash of its contents. It doesn't matter where that file is stored. Any computer on the network can serve the request.
Content addressing gives us the basic building block we need to transcend SaaS walled gardens and build a distributed world-wide knowledge graph. Which brings us to…
Layer 2. Noosphere, our WIP “protocol for thought”. It’s sort of like RSS on top of IPFS. A pragmatic protocol for syndicating notes, knowledge graphs, and other docs.
Distributed over IPFS
Secured using UCAN authorized signing/encryption
Addressable by meaningful names
Versioned like a lightweight Git
(All credit goes to cdata who has been leading design of the Noosphere protocol, and prototyping an implementation in Rust.)
Layer 3. Subtext, our simple markup for note-taking. Markup is a protocol—this is unavoidable—so we’re designing Subtext to be easy to write, easy to read, and easy to parse, so you can easily integrate it into other tools.
Layer 4. Subconscious ties it all together into a world-wide networked note-taking app.
This is all pretty early-stage work, but we’re excited to share more soon!