If headers did not exist, it would be necessary to invent them
I’ve been studying the history of the internet, and reading through internet RFCs to better understand the series of design decisions that lead up to our open-ended and evolvable internet.
Sometimes you stumble across a flash of pure insight buried in these dusty spec documents. RFC 822 Standard for the format of ARPA Internet text messages is one of those. It’s an old RFC, standardizing informal design patterns for email that emerged organically at ARPA. From the intro:
A general "memo" framework is used. That is, a message consists of some information in a rigid format, followed by the main part of the message, with a format that is not specified in this document.
The email memo format was an imitation of paper memos, which were once sent around offices the way we send email now.
And here is what an email looks like when you peek under the hood.
From: subconscious@substack.com
To: you@example.com
CC: them@example.com
Subject: The memo format
Date: 26 Aug 21 1429 EDT
This is the body. It can contain free-form text and data.No magic here, just plain text. A series of key-value metadata pairs, followed by an empty line, and then free-form text. Because these key-value pairs show up at the top, they’re often called headers.
Tim Berners-Lee used this same header pattern for the design of HTTP:
A client sends an HTTP request to a server in the form of a request message, beginning with a request-line that includes a method, URI, and protocol version, followed by header fields containing request modifiers, client information, and representation metadata, an empty line to indicate the end of the header section, and finally a message body containing the payload body (if any).
RFC 7230 Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing
So that’s email and HTTP. Just plain text keys and values, followed by some content. Two of the most important open-ended systems have been built on top of this simple metadata mechanism.
The memo format is Lindy
One thing I’ve been thinking about is how to introduce an open-ended metadata primitive to Subconscious. Key-value metadata is a fundamental pattern that we see emerge repeatedly. Where it doesn’t exist, we often see it bolted on after the fact.
If you’ve ever worked with a static site generator, like Jekyll, then headers probably look familiar. Markdown didn’t have the concept of headers, so Jekyll found it necessary to invent them.
---
layout: post
title: Blogging Like a Hacker
---We also see customizable metadata bolted on after the fact in many knowledge work applications. Ever notice that tools like Zotero, or Mendeley are basically fancy Finder windows with extra metadata fields for author, DOI, etc?
We really do build entire apps just to tag files with a bit of metadata.
Way back in 1990, BeOS took this insight and ran with it. Files in BeOS could be tagged with any key-value metadata you liked. The metadata could be displayed as columns in the file finder.
In fact, many BeOS “apps” were really just finder windows with specific metadata columns. You could customize or create your own “apps” by modifying the columns in a window.
It seems worthwhile to design a customizable metadata mechanism for notes from the get-go, rather than bolting it on later. A simple, standardized metadata primitive means:
You don't have to create bespoke apps for most things
Metadata is saved in standard files, not trapped in some app database
Many different apps and tools can collaborate over same files
With a bit of metadata sprinkled on top, your notes can gain superpowers:
Use your notes as a CRM
Manage citations for your research
Use it for quantified self tracking
Publish your notes to the web with a static site generator
Headers as tabular data
Here’s another BeOS insight: any key-value metadata can be treated as tabular data. Each file is a row, each key is a column, each value is a value for that column.
Introducing metadata to notes opens up the possibility of treating notes-as-data, and collections of notes as tabular data. Imagine marking up your notes with headers, then being able to…
Bulk edit them in Excel
Query over them with SQL
Generate new notes from queries over old notes.
Tabular data is a composable alphabet with a massive toolchain, powerful mathematical properties, a deep body of computer science theory. If you can turn it into a table, you can bring it into a spreadsheet, dump it into a database, index it by column, query it with SQL.
Headers are what you would write anyway
One of the key things to appreciate about email and HTTP headers is that they are just ordinary text, written at the top of the document. The syntax of headers is easy to understand.
Headers are one-per-line. They are made up of a key, followed by a colon, followed by some value.
The header area ends at the first empty line.
Everything under that is free-range text.
Keys are case-insensitive, so you can have nice capitalized key names, without worrying about splitting keys.
The header format is designed to be read and written by hand if you like. It’s probably what you would write anyway. This is even easier to appreciate when we compare headers to the other obvious candidates for structured data:
Unlike HTML, XML, or JSON, headers don’t feel like code.
Unlike YAML, headers are simple, and difficult to screw up.
Headers are extensible
In addition to the fields specified in this document, it is expected that other fields will gain common use. As necessary, the specifications for these "extension-fields" will be published through the same mechanism used to publish this document. Users may also wish to extend the set of fields that they use privately. Such "user-defined fields" are permitted.
RFC 822 Standard for the format of ARPA Internet text messages
You can add any headers you like. This makes the set of headers an open set, enabling new use-cases to evolve from the bottom-up. Want to record the title, author, and publish date of your books? Add some headers. Want to include a location? Add a header.
Header values can also be anything. Header keys can choose whatever format is most appropriate for their values.
The syntax that distinguishes between header fields is specified separately from the internal syntax for particular fields. This separation is intended to allow simple parsers to operate on the general structure of messages, without concern for the detailed structure of individual header fields.
RFC 822 Standard for the format of ARPA Internet text messages
This might cause a bit of inconvenience when parsing exotic headers. On the other hand, the syntax of simple metadata values can be written in whatever way is most natural for the type of data. Text can just be text, lists can be comma-separated values. If you need complex data, you can include JSON. Simple things can be simple. Complex things are possible. Use whatever is fit for purpose. Headers don’t care.
Allow all, bless some
The syntax of several fields of the rigidly-formated ("headers") section is defined in this specification; some of these fields must be included in all messages.
RFC 822 Standard for the format of ARPA Internet text messages
While you can add any headers you like, and they can have any format you like, there are some headers that are standardized, such as To, From, CC, Date. This defines a minimal stable API surface for clients to work with and enables federation between multiple clients and services.
Allow all, bless some is an important general principle for building evolvable systems. Allowing all means new use-cases can evolve from the bottom-up, while blessed fields provide a stable API which multiple clients can count on.
Headers enable open-ended evolution
Always provide an et cetera escape hatch in your ontology. It lets you evolve that ontology later on.
Email initially assumed plain text. Eventually people wanted more complex features. Headers enabled email to evolve the format without breaking backwards compatibility.
The purpose of the Content-Type field is to describe the data contained in the body fully enough that the receiving user agent can pick an appropriate agent or mechanism to present the data to the user, or otherwise deal with the data in an appropriate manner. The value in this field is called a media type.
Because you can freely add new headers, standards bodies were able to introduce the Content-Type header after the fact. It is used as a flag to tell the client what kind of data is in the free-range content area beneath the headers. With the Content-Type flag, email could suddenly be used to embed HTML, photos, videos, binary data, even applications, if a client supported it. If you don’t include a content type, the client just assumes plain text.
RFC 2045 even specifies a “multipart” content type, which is a way of including multiple content sections of different types in the same email. The client can choose whichever one it supports.
My takeaway is when you build your system on top of an open-ended metadata primitive, you can evolve that system in new directions. It turns out email and the web thought this one through pretty well. The syntax is nice, the semantics are open-ended. You could do worse than to copy a pattern that has succeeded in two of our largest open-ended ecosystems.