Something More Profound:
The Political Economy Of Network Protocols
For Remolino and Koochooloo
Table of Contents
- Prologue
- Chapter 1: The Greatest Challenge
- Chapter 2: Standard Wars - What Are They Good For?
- Chapter 3: If You Can’t Beat Them, Co-Opt Them: CLNP vs IPv6 As The Future Of TCP/IP
- Chapter 4: DNS In Anger - From Centralized Directories to Distributed Names to Challenging Westphalian Sovereignty
- Chapter 5: Internet Routing: You Can’t Get There From Here
- Chapter 6: Asleep, Tired, Or It Malingers: How Protocols Wither and Die
- Chapter 7: The Street Finds Its Own Use For Things: The World Wide Web From CERN to Snowden
- Chapter 8: Here Comes Everybody: Overlay Networks
- Chapter 9: Bitcoin: A Fork In The Road
- Chapter 10: The End Of The Beginning (Of The Internet)
- Epilogue
- Bibliography
Prologue
PAXMAN: “You’ve got to think that some of the claims being made for it are hugely exaggerated, I mean, when the telephone was invented…”
BOWIE: “Of course they are…”
PAXMAN: “…people made amazing claims…”
BOWIE: “I know, the president at the time, when it was first invented, it was outrageous… He said he foresaw the time when every town in America would have a telephone. How dare he claim like that. Absolute bullshit. (Chuckle). No, you see, I don’t agree. I don’t think we’ve even seen the tip of the iceberg. I think the potential of what the internet is going to do to society, both good and bad, is unimaginable. I think we’re actually on the cusp of something exhilarating and terrifying.”
PAXMAN: “It’s just a tool though, isn’t it?”
BOWIE: “No, no, it’s an alien life form!”
PAXMAN: “What do you think then…”
BOWIE: “Is there life on Mars? Yes, it’s just landed here.”
PAXMAN: “But it’s simply a different delivery system there. You’re arguing about something more profound.”
BOWIE: “Yeah, I’m talking about the actual context and the state of content is going to be so different from anything we can envisage at the moment. With the interplay between the user and the provider will be so in sympatico it’s going to crush our ideas of what mediums are all about.”
— David Bowie, in an interview with Jeremy Paxman for BBC Newsnight in 1999 [1]
“There is an analogous problem, and probably a more difficult one, in the matter of language for the control of a network of computers. Consider the situation in which several different centers are netted together, each center being highly individualistic and having its own special language and its own special way of doing things. Is it not desirable, or even necessary for all the centers to agree upon some language or, at least, upon some conventions for asking such questions as “What language do you speak?” At this extreme, the problem is essentially the one discussed by science fiction writers: “how do you get communications started among totally uncorrelated “sapient” beings?””
— J.R.C ‘Lick’ Licklider, in a memo dated April 23, 1963, addressed to ‘Members and Affiliates of the Intergalactic Computer Network’ [2]
“The history of the Net is the history of protocols.”
— Vint Cerf [3]
This text is, primarily, a history of protocols and their push-and-pull relationship with the humans that agree to make use of them. As Vint Cerf noted, this makes it a history of the Internet as well, if only by proxy.
The word ‘protocol’ by itself is a bit overloaded. Even limiting ourselves strictly to the context of information technology, there are intranetworking protocols, for example, that are used to communicate between hosts on a local network, as well as cryptographic protocols that are, at their core, a sort of recipe for a hypothetical Alice and an imaginary Bob to share messages with each other by means of exchanging messages signed with their secret keys.
The focus of this text is specifically on internetworking protocols — the shared language used to communicate between “totally uncorrelated sapient beings”, to use Licklider’s phrasing, across the different networks that make up our present-day Internet. By way of analogy, it is a history of language and linguistics, rather than a history of the rise of the printing press or a grammar textbook.
What I hope you take from this book:
Internetworking protocols — from now on in this text, simply “protocols”— are the rules that the software you depend on must implement and follow in order to connect and interact with the software that other users depend on over the Internet. Because they are a foundational component of the software applications that billions of people use to browse the Web or check their email on a daily basis, they play a silent role in shaping human behavior, literally defining the boundaries of our online world. Most users of the Internet are blissfully unaware of the role that these protocols play in their daily lives, and the engineers that are familiar with them are usually too busy dealing with the technical minutiae — the nitty-gritty details of how bits and bytes are swapped back and forth — to step back and appreciate the impact they have on the individual.
The aim of this text is to help the reader understand internetworking protocols by examining their history, to unravel the layers of complexity that have accrued in them over the decades, and to examine their impact on us as users of the Internet. It is an attempt to explore the foundations of the networked world that we inhabit, a world that is shaped not so much by traditional laws, customs, and social mores as it is by half-remembered blueprints and extant lines of code etched into the fabric of networked interactions.
The reach that these protocols have, by definition, extends beyond the individuals that use them. They play a pivotal role in the functioning of our society, from facilitating our financial transactions to controlling and securing our public and private speech. They underpin the backbone of the global economy, enabling trade, communications, and interpersonal relations in ways that were unimaginable scarcely a generation ago. Accordingly, they represent a de facto threat to the pre-existing shared understanding of rules that govern the spread of information, of communications, and of commerce – commonly referred to as the ‘rules-based international order’. This threat has emerged silently and unexpectedly; network protocols are drafted as technical specifications, implemented as simple lines of code, and shipped as part and parcel of the operating systems that power every smartphone and laptop. The threat that has emerged is not from the existence of these programming artifacts, but from the power represented by the masses of users that collectively agreed to use this software to communicate and connect to each other.
In ‘States And Markets’ (1988), Dr. Susan Strange frames the analysis of a sovereign territory’s political economy as an attempt to answer the question of ‘who-gets-what-and-why’— exactly who is granted access to what rights and resources, how are these rights granted, and to what end? If the Internet is the territory we have chosen to examine, then the “how” must be the protocols that serve as the foundation of internetworking; this text will examine the who, what, and why.
As you make your way through this text, I ask that you consider the knowledge you will find within these pages as more than just a historical review; it is a map of sorts — a guide to how we collectively came to agree on the rules that have shaped our present-day Internet and how they define the space that we as users occupy in it. I hope that it helps you to better understand how we arrived at our modern-day Internet, and how we might make our way forward.
Chapter 1
The Greatest Challenge
In the early 1960s, before the Department of Defense’s Advanced Research Project Agency had been tasked with building or even designing the ARPANET — the network commonly cited as being the precursor to today’s Internet — computer scientists were beginning to explore the potential for the sharing of computing resources over telecommunications links, in the hopes of aggregating resources in order to build computers that were more powerful than the sum of their parts. Paul Baran’s September 1962 paper ‘On Distributed Communication Networks’ [1] and later J.C.R Licklider’s April 1963 memo describing an ‘InterGalactic Computer Network’ [2] were the first documents to imagine how disparate communities of researchers might go about connecting, communicating, and sharing these computing resources with one another. These were embryonic thought experiments put to paper, with no real-world implementations of the ideas behind them. It took roughly four years to progress from the first batch of academic papers to something that could be considered a proof-of-concept. That first spark of life would have to wait until October of 1965, when Lawrence Roberts and Thomas Marill built one of the first rudimentary computer networks, akin to cans connected with string, using only:
“Two experimental computers: the TX-2 at the Lincoln Lab and the Q-32 at the System Development Corporation in Santa Monica. A line leased from Western Union provided the communications link, and Marill and Roberts wrote their own software to manage the connection. They published their results in the fall of 1966, just before Roberts left Lincoln Lab for ARPA. In describing their experiment, Marill and Roberts articulated some important concepts. In their view, the “elementary approach” to connecting two computers was for each computer to treat the other as a terminal. Such a connection required little modification of the computers, but it had severe limitations. The connection was slow, since terminals operate at much lower data rates than computers, and there was no general-purpose way to access a remote system, since each separate application program had to manage its own connections rather than having the operating system handle the connections for all applications. Marill and Roberts thought that forgoing the elementary approach and taking on the harder task of modifying the computers’ operating systems would make it possible to create a higher-speed computer-to-computer interface instead of relying on the ordinary terminal-to-computer interface. They proposed that each host computer implement a general-purpose set of rules for handling a network connection, which they called the “message protocol…”
— Marill and Roberts 1966, p. 428, as quoted in Abbate’s Inventing The Internet [3]
Lawrence Roberts’s usage of the word protocol here to describe the “general purpose set of rules for handling network connections” predates even the design of the ARPANET. Roberts correctly anticipated that the task of connecting a diverse collection of far-flung computer terminals would require some sort of mutually agreed lingua franca, without which it would be impossible to realize his vision, one where:
“Almost every conceivable item of computer hardware and software will be in the network… This is the greatest challenge of the system, as well as its greatest ultimate value.”
— Dickson, Paul A. 1968. “ARPA Network Will Represent Integration on a Large Scale.” Electronics, 30 September: 131–134. [4]).
Simultaneously, there were other researchers and engineers working along parallel tracks trying to connect networks of computers to each other; Donald Davies and Paul Baran were separately working on their own designs for packet-switched digital communications networks at the National Physical Laboratory (NPL) in London but had not yet gotten to the point of deciding how hosts on these networks would ever communicate. That would have to wait until August of 1967, when two researchers working under Davies at the NPL, Roger Scantlebury and Keith Bartlett, published a memorandum titled ‘A Protocol for Use in the NPL Data Communications System’ [5]. This appears to have been the first formal use of the word ‘protocol’ in a technical specification.
As luck would have it, Davies and Roberts would meet up shortly after Roberts’s experiment at Lincoln Lab at the Defense Advanced Research Projects Agency (DARPA), where Roberts, with input from physicist Wesley Clark and drawing from the NPL’s proposals on data networking, wrote the paper that laid out the tentative concept of what would become the ARPANET to the wider research community. He presented the paper at the Association for Computing Machinery’s Symposium in Gatlinburg, Tennessee, in October of 1967.
“ARPA supports a number of computer research groups throughout the country, most of which have their own time-shared computer facilities. These researchers have agreed to accept a single network protocol so that they may all participate in an experimental network. The communication protocol is currently being developed. It will conform to ASCII conventions as a basic format and include provisions for specifying the conventions as a basic format and include provisions for specifying the origin, destination, routing, block size, and sum check of a message. There are 35 computers […] at 16 locations; there are several computers at most locations.”
— Lawrence Roberts, Multiple Computer Networks and Intercomputer Communications, January 1967 (https://dl.acm.org/doi/10.1145/800001.811680) [6]
Roberts’s presentation at Gatlinburg caught the attention of several other researchers in attendance. Two former colleagues of Roberts who had moved on to the telecommunications research company Bolt, Beranek, and Newman had been keeping tabs on his work; when DARPA decided to provided funding for the actual implementation of their first network — the ARPANET — BBN submitted a successful bid for the contract starting in January of 1969, based largely on fleshing out the ideas in Roberts’ paper and bringing it to life. The DARPA group tasked with working alongside BBN to implement the communications protocol described in Roberts’s paper became known as the Network Working Group, or NWG.
The initial challenges for the NWG were significant. Beyond the Host-to-IMP protocol (later formalized in BBN Report 1822, which defined how host computers connected to their local IMPs within the BBN-provided infrastructure), the NWG was tasked with defining how applications on different host computers could communicate through this IMP network. This involved defining a “Host-to-Host protocol.” Early ideas were explored in documents like RFC 1 and RFC 2, with more concrete proposals for a “Host-Host Protocol” emerging in RFCs such as RFC 36 and RFC 54. While Roberts appears to have been the first to come up with the concept and terminology of a network ‘protocol’, as well as the first proof-of-concept implementation of one, it was the NWG that would take on the challenge of designing and implementing the first widely adopted host-to-host communications protocols that would be expected to work in real-world conditions, moving beyond the thought experiments that had come before. In doing so, their work put them in the unenviable position of being the first to tease out the differences between protocol specifications in theory and protocol implementations in practice.
As the meetings of the NWG began to take shape, one of founding members, Steve Crocker, realized that it would be in the collective best interests of the group to begin recording their discussions on protocol development and implementation for posterity.
The first few meetings were quite tenuous. We had no official charter. Most of us were graduate students, and we expected that a professional crew would show up eventually to take over the problems we were dealing with…
In February of 1969, we met for the first time with BBN. I don’t think any of us were prepared for that meeting. The BBN folks, led by Frank Heart, Bob Kahn, Severo Ornstein, and Will Crowther, found themselves talking to a crew of graduate students they hadn’t anticipated. And we found ourselves talking to people whose first concern was how to get bits to flow quickly and reliably but hadn’t — of course — spent any time considering the thirty or forty layers of protocol above the link level. And while BBN didn’t take over the protocol design process, we kept expecting that an official protocol design team would announce itself…
A month later, after a particularly delightful meeting in Utah, it became clear to us that we had better start writing down our discussions. We had accumulated a few notes on the design of DEL and other matters, and we decided to put them together in a set of notes. I remember having great fear that we would offend whomever the official protocol designers were, and I spent a sleepless night composing humble words for our notes. The basic ground rules were that anyone could say anything and that nothing was official. And to emphasize the point, I labeled the notes “Request for Comments.” I never dreamed these notes would be distributed through the very medium we were discussing in these notes. Talk about Sorcerer’s Apprentice!
— RFC 1000, “The Requests For Comments Reference Guide” [7], https://datatracker.ietf.org/doc/html/rfc1000
The first functional protocol that came from the NWG was the NCP — Network Control Protocol (initially conceptualized in part as the “Network Control Program” software residing on hosts, the term Network Control Protocol came to define the communication rules themselves)— which was functionally a precursor to today’s modern TCP protocol. The NCP was the protocol that brought the ARPANET to life, serving as the lingua franca that Roberts had initially envisioned; as the first fruit of their efforts, it was to serve as a blueprint not only for how protocols were developed but also for what it takes to replace them.
Why ‘protocol’? It seems an unlikely term to have figured in the vocabularies of engineering students, even bright ones, at the time. Vint Cerf, one of the students involved in the NWG, told Peter Salus that it had originated in a conversation between Steve Crocker and Jon Postel, who had thought of it in terms of diplomats exchanging handshakes and information. It was only later that they discovered its etymology (the word comes from the Greek, and means the first page of a document bearing the title, name of the author, scribe, and so on).
— Naughton’s A Brief History Of The Future [8]
The development process for the NCP was open to all and documented extensively, with dozens of engineers communicating over the course of two years and over twenty Requests For Comments (RFCs). The process of drafting RFCs happened organically and indeed almost accidentally. The NWG never meant to propose a new means of collaborating remotely with their peers; they needed a way to cross the organizational boundaries between private industry, academia, and government to get things done in time to meet the deadlines proposed in their contract, and a voluntary ad hoc process seemed the easiest way to accomplish that. The process of protocol design and development that they came up with turned out to be so effective that it persists to this day: new Internet standards, procedures, protocols, and processes are proposed for adoption by the broader Internet community and codified into RFC documents. When rough consensus has been reached on their suitability, they are adopted as Internet Standards by the Internet Engineering Task Force (IETF). Finally, it is left to software developers to translate the protocols as specified in RFCs into software implementations that computers can use to interoperate with each other.
The implementation of the NCP and its role in bringing the ARPANET to life gave the NWG and the ARPANET participants insight into what a global computer network would look like in practice — and with it, the almost immediate realization that the NCP would be inadequate to realize their vision of what the ARPANET was meant to become. To that end, in 1973, work began to design the first successor protocol to the NCP, the Transmission Control Protocol; the second, InterNet Protocol, was started in 1978. The two deployed together came to be known as TCP/IP.
TCP/IP is perhaps the most successful set of network communications protocols ever designed; any device that connects to the Internet, from mobile phones to iPads to smart TVs to computers, needs to implement support for the TCP/IP protocol suite into its operating system or the silicon that runs it.
It took several revisions and nearly a decade until the Internet Protocol — Internet Protocol Version 4, formally, or more commonly IPv4 — was ready to deploy in the early 1980s. Deployment itself was done by setting a flag day on January 1, 1983, for all nodes on the ARPANET to support the new protocol suite alongside NCP [7]. With only about 400 nodes, coordination was fairly painless, but since TCP/IP enabled the ARPANET to move on from being a single global network to an interconnected network of networks (more commonly, an InterNet), future flag day transitions were rendered impossible due to the resulting explosion of growth. Coordinating changes to one network was straightforward enough, as the administrators of a single network were able to operate as a sort of benevolent dictator; coordinating changes across dozens of networks with different administrators was something more akin to trying to pass a resolution at a meeting of the United Nations.
As luck would have it — or perhaps it was the serendipity of accidental design — the opt-in nature of the RFC process for future protocol adoption would mean that flag day transitions were no longer necessary. Anyone developing software, from hobbyists to corporations, who wishes to take advantage of newly proposed TCP/IP protocol features specified in an RFC is free to do so at their own pace. Implementors only need to concern themselves with ensuring that no changes are made that would be somehow incompatible with other TCP/IP implementations. This focus on backwards compatibility means that different implementations can incorporate new protocol functionality at the pace and tempo of their own choosing while still maintaining the ability to connect and interact with each other. The principle of maintaining backwards compatibility has served the software developers tasked with writing protocol implementations well, allowing the adoption of protocol changes and updates to be driven by bottom-up demand from the Internet community as opposed to top-down hierarchical diktat.
Layering and flexibility
Solutions to the technical problem of connecting a disparate group of hosts into a single network were relatively straightforward to agree upon when compared to the social problem that was on the horizon. Researchers who heard of the initial ARPANET design were impressed by the vision described and eager to suggest ideas for additional features to be implemented even before the specifications had been drafted by the NWG. But how to get the network actually up and running while new features and functionality were still actively being proposed?
With plenty of ideas but no concrete definitions for what services the nascent ARPANET would be providing,
…there wasn’t any clear idea of what work the hosts had to do. Only later did we articulate the notion of building a layered set of protocols with general transport services on the bottom and multiple application-specific protocols on the top. More precisely, we understood quite early that we wanted quite a bit of generality, but we didn’t have a clear idea how to achieve it. We struggled between a grand design and getting something working quickly.
— Hauben and Hauben, Netizens: On the History and Impact of Usenet and the Internet (https://firstmonday.org/ojs/index.php/fm/article/view/612/533) [9]
The model of protocols being layered on top of one another, as opposed to one omnibus protocol being forced to specify every proposed use case, was perhaps a natural conclusion to arrive at. The ARPANET design specified that one type of node in the network would serve as ‘interface message processors’ (IMPs), switching packets back and forth between another type of node in the network, dubbed ‘hosts’; once this logical separation between types of nodes existed, a sort of modular design was made implicit, and the stacking of abstractions on top of abstractions that we have come to know as ‘layering’ fell into place as if it had always been intended. The roles and responsibilities of IMPs would exist on one layer, hosts another, and so on, with each layer building on top of the one underneath it. As a first example, NCP was built to set up connections between hosts on the network; the Telnet protocol and File Transfer Protocol (FTP) were layered atop NCP to enable logging in to remote machines and transferring files between them.
This model of stacked abstractions is ultimately what gave researchers the time and space to work on their core vision of implementing a protocol to interconnect hosts —‘getting something quickly working’— by allowing them to defer the harder problems of exactly what sorts of services would end up powering the ‘grand design’. Without it, it’s quite likely that NCP would have been delayed or even dead on arrival, as the NWG would have been forced to sort out not only the features provided by Telnet and FTP, but also every other desired bit of functionality into NCP. Imagine trying to take our current 2024 understanding of the Internet — web browsing, chat, social media, streaming movies, Zoom, FaceTime and all the rest — and convening a group of researchers to write specifications for it all in the late 1960s, using only the technology available at the time, and you’ll get a sense of the challenge this would pose.
Now try to imagine what an upgrade to those same specifications in the late 1970s (when TCP/IP was first being specified) might look like, and the problem becomes even clearer. With so much ground to cover and so many disparate viewpoints that would need to be accommodated, it’s entirely possible that the NWG would still be working on writing specifications for the NCP by the time TCP/IP was first proposed. Ted Nelson’s Project Xanadu is a good example of what this might have looked like in practice; described as an all-encompassing “digital repository scheme for world-wide electronic publishing”, it was first envisioned in the 1960s, with an initial working proof-of-concept delivered in 2014. Xanadu’s design shoved everything from addressing to connection management to document referencing into one piece of monolithic software, so none of it was ready to use until all of it was ready.
The impossibility of being able to anticipate future users and use cases decades out, with only present-day technology to guide the way, would almost certainly have doomed the enterprise to failure — or perhaps even worse, a slow death of constant revision and re-invention, as future researchers and engineers would go through a cycle of implementing their ideas using building blocks (in protocol speak, ‘primitives’) that had been specified years or even decades before, find that they weren’t fit for purpose, forcing them to propose new primitives, starting the cycle over again. Modern day software developers may recognize this as a symptom of the ‘waterfall’ development methodology, where progress flows from design through to development and ultimately deployment, and it is assumed that the finished software artifact will only require maintenance.
In contrast, layering protocols atop one another allows each supporting protocol to act as a sort of foundation for future protocols to build on, with no ‘end state’ necessary to define in advance. Protocols such as HTTP, which web browsers use to connect to web sites, and SMTP, used to send email back and forth between users, rely on the underlying TCP/IP protocol to handle host addressing and connection routing, concerning themselves only with the higher-level semantics required by the applications that depend on them. This means that, for example, web browsers can be run on older operating systems and computers with outdated versions of TCP/IP, and that future upgrades to the HTTP protocol can be rolled out and supported by web browsers without needing to upgrade the underlying operating system.
In fact, by way of example, as of June 2021, HTTP, the Hypertext Transfer Protocol, had seen its fifth version (HTTP 0.9. 1.0, 1.1, HTTP/2 and now HTTP/3) since its inception in the early 1990s. The first live version, HTTP 0.9, only supported delivery of plain HTML documents — meaning no images, much less videos — and was perhaps as bare-bones as a higher-level protocol could be. It is often dubbed the ‘one-line protocol’, as that is all that’s required to construct an HTTP 0.9 request — a single line with only one possible method, GET, requesting an HTTP resource.
Nevertheless, a browser that supports the considerably more complex HTTP/3 can still be run (albeit suboptimally) on operating systems and through routers that make use of TCP/IP protocol implementation from the HTTP 0.9 era, with no complaints (or indeed, even any understanding required by the lower-level protocol about what version of HTTP is being used). The higher-layer HTTP protocol relies on the lower-layer TCP/IP protocols to route packets between different hosts on the Internet, and in turn, TCP/IP happily shovels packets of data back and forth without any inkling of what the higher-layer protocol is attempting to do inside of them. The lower-layer is left to do the relatively boring work of transmission control (the TC in TCP) and internetworking (the I in IP), while the higher layer is free to focus on whichever services it was designed to implement.
Takeaways:
- Network protocols are system level software artifacts that enable the operators of heterogeneous, geographically dispersed collections of hardware and software, representing a diverse cohort of users and organizations, to connect their systems and software applications to each other, by way of a shared common language.
- Requests For Comments (RFCs) are used to propose new Internet standards and communicate information between disparate groups of stakeholders. RFC compliance is opt-in by design; there are no ‘protocol police’ to enforce them. This means that users can voluntarily adopt modifications and changes made to protocols as long as care is taken to maintain backward compatibility for those who do not. RFCs specify how a protocol should operate — much like an architectural blueprint — and software implementations of those protocols are where the buildings of those blueprints come to life.
- Deprecating a protocol is no easy thing; the flag-day transition from the first Network Control Protocol to its successor, TCP/IP, was the first and last of its kind in the Internet’s history. A protocol can remain active as long as two parties agree to use it.
- Layering, the building of protocols on top of other protocols, with each layer serving as a sort of foundation for the one that comes after it, enabled the Internet to grow as quickly and widely as it did. By allowing a separation of concerns at the very start of the design of the ARPANET, core protocol designers were able to focus on the nuts and bolts work of connecting hosts to each other while leaving ancillary features and services to be specified and implemented in higher-level protocols at a later date and by different groups of engineers.
A point of clarification: In 1969, was BBN contracted to design and deploy four Interface Message Processors (IMPs) as a sort of embryonic test of packet-switching technology. This deployment is recognized today as being the first test of what came to be known as the ARPANET. The Host-to-IMP interface protocol, later formalized in BBN Report 1822 and now referred to by as some as ‘the 1822 protocol’, defined how host computers connected to these IMPs. While crucial for ARPANET’s operation, and a foundational layer, I find it difficult to describe this as the first internetworking protocol, given that it primarily governed connections within the BBN-provided infrastructure, only ran on a few hosts initially, and perhaps most importantly for the context of this book, was a) developed entirely in-house at BBN without the broad collaborative effort characteristic of later internetworking protocols, and b) wasn’t publicly documented until 1976, years after NCP had been deployed and when work on TCP was already well underway. That said, I also feel it would be shortsighted to dismiss the importance of the 1822 protocol out of hand. It is analogous perhaps to the woodblocks used by Chinese monks centuries before Gutenberg’s printing press — not an ancestor or a distant relative of later internetworking protocols, but a parallel species that evolved in the same environment to solve a related, underlying problem of connecting computers to the network fabric.
Chapter 2
Standard Wars - What Are They Good For?
With the successful deployment of the Network Control Protocol (NCP), used to connect a scattered handful of hosts to one another, the Network Working Group was able to bridge the gap between computer networking theory and the first functional network in practice. In doing so, they captured the attention of several networking incumbents, or rather, would-be incumbents: interested parties that could reasonably lay claim to having some sort of say in the design and implementation of networking protocols, if such things were ever to exist. If we broadly define network protocols as communications standards for connecting disparate computing hardware, then we can divide these interested parties into two camps: organizations that were charged with drafting communications standards, and vendors of disparate computing hardware.
The first camp, those charged with drafting communications standards, included the Consultative Committee on International Telegraphy and Telephony (CCITT), which had been granted partial responsibility by the United Nations for coordinating standards for European and transcontinental voice communications. It shared this responsibility with the ISO, the International Organization for Standardization (widely known as “ISO”), founded in 1946 to coordinate (no surprise here) international standards of all sorts. In the second camp, there was IBM, by far the largest provider of computer hardware at the time, and their arch-rival DEC (Digital Equipment Corporation), the company behind the popular PDP-11 minicomputer and its successor, the VAX.
By the time the NWG had cut their teeth on NCP and came to the realization that a new protocol would need to be drafted and designed to replace it, the ARPANET experiment had garnered sufficient interest from the customers and constituents of these incumbents to force them to a collective realization: get to work drafting your own networking protocols, or risk having someone else dictate the standards by which computers would connect to one another.
Ultimately, who should be responsible for designing and deploying a standard for enabling computers to communicate with each other? The vendors who already design and deploy the computers, the organization nominally responsible for the communications links that would connect them, or the organizations that saw themselves as being charged with managing international standards of all stripes? Or should it be none of these, and instead left to the original researchers of the NWG — now replaced by its successor, the International Network Working Group (INWG)? This implicit unanswered question led to a cold war of sorts, an unacknowledged but still very real standoff between vendors, standards bodies, and researchers, as each drafted their own competing standard with the presumption that other camps would surely be eager to adopt them.
The CCITT was first out of the gate, announcing their intention to begin drafting networking standards in 1968, when the NWG had just begun to assemble. Four years later, at their 1972 plenary session, they announced what came to be known as the X standards:
First, it published eleven recommendations with the new, soon-to-be-famous “X” prefix (including Recommendations X.1, X.20, and X.30 that specified basic interfaces between circuit-switched public and private data networks). Second, it formalized and extended the mission of the Joint Working Party into Study Group VII on New Networks for Data Transmission.
— Russell, Open Standards [1], pp.165
IBM’s System Network Architecture (hereafter SNA) came later, in September of 1974. Being a vendor in the business of actually manufacturing computer equipment, they had the advantage of being able to get a network protocol implementation up, running, and shipping with their own hardware. While the CCITT saw its role as defining standards for telecommunications, IBM’s objectives were more straightforward: to sell more IBM hardware to IBM customers. SNA’s design reflects those goals:
SNA’s objective was to reduce the costs of operating large numbers of terminals and thus induce customers to develop or expand interactive terminal-based systems as opposed to batch systems. An expansion of interactive terminal-based systems would increase sales of terminals and more importantly of mainframe computers and peripherals — partly because of the simple increase in the volume of work done by the systems and partly because interactive processing requires more computing power per transaction than batch processing.
– Wikipedia, https://en.wikipedia.org/wiki/Systems_Network_Architecture [2]
Lastly, DEC had its own protocol suite, DECNet, which launched its Phase 1 implementation in 1974 as well. DECNet’s explicitly stated design goal was to allow DEC computers running different operating systems to communicate with each other; otherwise, in practice, its objectives were no different than IBM’s, though it was significantly less influential.
The OSI Model
The joint CCITT/ISO effort, by nature, moved at a much slower pace than commercial enterprises like IBM and DECNet; they didn’t begin to work on specifying the Open Systems Interconnection (OSI) model of protocol development until 1977, with a first draft not being released until 1980. As the ISO shared responsibility for international telecommunications standards with the CCITT, and the CCITT had focused on nuts-and-bolts implementation details while the ISO initially focused on defining a layering model, it was natural enough for them to join efforts:
At its first meeting in February and March 1978, OSI leaders negotiated an agreement with members of CCITT Study Group 7 (the committee responsible for X.25) to share jurisdiction over proposed standards that would “have an impact on the basic design and features of Data Processing Systems.” The two groups also named official liaisons in 1978 and 1979 that would maintain a healthy flow of information and policy of consultation. Subsequent personal correspondence and official resolutions established additional points of agreement: both groups supported the urgent development of a layered architectural reference model, and both groups possessed the organizational capabilities to sustain a modular approach to the technical and social challenges of open system standardization…
— Russell, Open Standards And The Digital Age [1], pp. 205
The ISO’s seven-tiered layering model — more commonly, the ‘OSI model’— informed the implementation of every protocol suite that came after it. The OSI model was developed by Charles Bachman at Honeywell in consultation with the INWG, Cyclades, ARPA, and others. Honeywell, of course, had their own protocol suite and their own agenda; Bachman’s work on what would eventually become the OSI model was begun as an attempt to influence international standards in a way that would be complementary to Honeywell’s HDNA protocol suite or, failing that, to prevent IBM from becoming the sole authority dictating international protocol standards.
DECNet and IBM scrambled to retroactively define their protocol suites in terms of the OSI model, while researchers working on the TCP/IP suite (having no higher-level protocols yet and with the lower-level ARP protocol not being developed until the early 1980s) were able to use it as a guide for how layering might be implemented in practice.
It should be noted that, even from the start, these disconnected groups of researchers didn’t work in isolation from each other. The successor to the Network Working Group, the INWG (International Network Working Group), liaised with the CCITT, albeit cautiously. Realizing that the CCITT’s approach was focused on circuit-based switching (a design more readily adapted to existing telephone infrastructure), the INWG treaded carefully and focused on their preferred paradigm of packet-based switching. This approach allowed the two groups to co-exist while still working at cross purposes.
Beyond that, at the time, none of these parties — ISO/CCITT, IBM, DECNet or the ARPA team — was particularly concerned with forcing the use of their protocol onto other parties, so much as maintaining their own “protocol sovereignty.” We have the benefit of looking back at their efforts with eyes that have seen the adoption of TCP/IP by virtually every device on the planet that requires network connectivity; this would have been beyond the wildest dreams of even the most fervent believers in computer networking during the embryonic phase of network protocol development that was the 1970s.
We should instead try to view the effort to standardize network communications from the point of view of the incumbents at the time, where TCP/IP seemed to be nothing more than an ambitious experiment. Throughout the 1980s, these various efforts — DECNet and SNA, the OSI model and the X standards, and the TCP/IP suite — were all being developed and iterated on. They reflect three different approaches to standards adoption: vendor-driven, top-down imposition, and bottom-up adoption.
If viewed as attempts to force worldwide adoption of a proprietary standard, IBM’s SNA and DEC’s DECNet were practically doomed to fail almost from the start; their existence forced customers into an either/or decision, where neither vendor had reason to compromise. It is ironic that the very existence of two vendor-designed protocol suites demonstrated the need for a non-proprietary, open alternative better than any theoretical argument ever could.
It is a mistake, however, to presume that global adoption of their protocol was the ultimate goal of the vendors simply because it was the goal of their competitors in the OSI and TCP camps. Unfortunately, we don’t have any recorded notes of what DEC and IBM management were thinking when they began their protocol development efforts; unlike the OSI, CCITT, and the (I)NWG, there were no formal contemporaneous records kept of the design process. We can, however, attempt to infer their intent from their efforts; for the vendors, it appears this was a classic standards war.
Just what does it take to win a standards war? Your ability to successfully wage a standards war depends on your ownership of seven key assets: (1) control over an installed base of users, (2) intellectual property rights, (3) ability to innovate, (4) first-mover advantages, (5) manufacturing abilities, (6) strength in complements, and (7) brand name and reputation. What these assets have in common is that they place you in a potentially unique position to contribute to the adoption of a new technology. If you own these assets, your value-added to other players is high.
—Shapiro, Information Rules [3]
The standards negotiation problem is akin to the classic battle of the sexes game: each player prefers a standard to no standard, but each prefers its own standard to the other’s. As in any bargaining problem, the outcome depends critically on the threat point — the payoff the parties receive if the negotiations break down. The better off a bargainer is if the negotiations fail, the more concessions he will be able to extract from his counterpart. Thus it is common to see companies continuing to develop proprietary solutions, even while engaged in standards negotiation.
— Varian, Economics Of Information Technology [4]
Absent a monopoly, the vendor-driven approach was never going to lead to widespread adoption. Standards wars rarely lead to outright victory; they are defensive wars of attrition by nature. IBM, for example, didn’t need SNA to be adopted by the wider Internet community or by DEC (though either of those outcomes would have been welcome); their ultimate goal was to ensure that DEC wouldn’t steal their customers away with promises of new networking functionality by having their own networking protocols to offer instead. The development of SNA was as much about marketing computers as it was about connecting them.
The OSI model and the X standards that implemented them were considerably more successful in terms of both gaining mindshare amongst researchers and developers and in seeing usage in the real world. Since the ISO and CCITT (now renamed the ITU) were granted a mandate by the UN for coordinating communications standards internationally, any national telephone system operator that wished to experiment with networking used the X standards as their guide (and to the X.25 protocol in particular, as it specified the lower-level data layer). The telecom companies were accustomed to relying on the ITU’s standards for interconnecting with other national telephone systems; relying on them for internetworking seemed only logical. This top-down imposition of protocol standards set the stage for the conflict between the OSI and Internet camps.
OSI’s top-down hierarchy vs. TCP/IP’s bottom-up adoption
One notable decision of the X.25 design group was to favor virtual circuits over datagrams. The decision was a pragmatic one that followed logically from the power dynamics of CCITT. As Rybczynski later noted, telephone carriers unanimously supported virtual circuits because they “better fitted their service models, existing applications and user expectations.” X.25 also fit well with the networking approach of a dominant user, IBM. IBM’s System Network Architecture, according to IBM France engineer Marc Levilion, transmitted data between computers first by considering a “map of the routes available,” and then by sending “each new communication between applications” along “one particular physical route across a network.” This emphasis on fixed routes was, Levilion recalled, “very close to the philosophy of X.25.” Such a consonance of interest was important for Déspres, who knew that the French telecom monopoly’s users were “companies using IBM computers… IBM had at that time more than half the market.” Virtual circuits, from the vantage point of the PTTs and dominant computer firms, would provide an incremental and predictable transition into the world of digital data communications – a satisfactory solution for the leading providers and users of standards at the time. Datagrams, however, promised disruption. Datagram designs seemed likely to usher in a new era of unpredictable and radical technological change in networking.
— Russell, Open Standards [1], pp. 169
What was initially a stylistic disagreement between proponents of virtual circuits (like OSI and IBM) and researchers interested in datagrams and packet switching – one approach promoting incrementalism, the other promising disruption – quickly became an intractable conflict. This protocol cold war ebbed and flowed throughout the 1980s and was fought on multiple battlefronts; what started as perhaps a matter of taste became over the course of several years an intractable war of attrition cast in near-religious terms by those waging it, a battle between the ‘netheads’ supporting TCP/IP and the ‘bellheads’ that saw internetworking as an incremental improvement of the telephone system.
The TCP/IP camp advocacy for packet-switching allowed them to take a blue-sky approach to network protocol research and development; the OSI camp preferred circuit-switching, if only because their target audience of telephone carriers and computer vendors was more readily able to integrate circuit-switched protocols into their existing infrastructure. Ultimately, the implicit (and sometimes explicit) end goal of the OSI/CCITT camp was to have national networks operated by public telephone carriers, while the ARPA camp’s idea of internetworking necessarily implied a group of heterogeneous networks all connecting across national boundaries to each other using a single protocol (with each network presumably free to ‘speak’ whatever protocol they preferred to use internally). This vision of how disparate networks would be interconnected necessarily meant that the protocol suite would have to be adopted from the bottom up; the ARPA team had neither the resources nor the inclination to push for their protocol suite’s adoption on every network that could theoretically interconnect, and focused instead on making their software available to anyone who might wish to do so, leaving the details of implementation up to the host networks.
A curious dichotomy is revealed in the historic record of the ISO’s efforts at drafting standards; while they were receptive to outside input and in fact encouraged collaboration amongst researchers from competing efforts, there was an expectation that OSI standards, once finalized, were not to be questioned, much less debated or compared to alternatives. Looking back on the effort, one of the engineers working on the TCP/IP standards, David Mills, put it succinctly: “Internet [TCP/IP] standards tended to be those written for implementers. International [OSI] standards were written as documents to be obeyed.” (Russell, Open Standards [1], pp. 227).
The conflict between the two camps ebbed and flowed in intensity for decades; sometimes researchers would work together, sometimes at cross purposes, in fits and starts, as the idea of computing networking took hold, and with it, the need for some sort of standard for users to rely on for guidance. The OSI camp had an endless procession of meetings and committees, leading to more meetings and more committees — inevitably leading to more opportunities for different stakeholders to try to inject their own agenda into the standardization process. IBM, perhaps unsurprisingly, was the most active of these, in the hopes of delaying and stonewalling the OSI efforts in an apparent attempt to buy time for their SNA protocol suite to reach critical mass. Being a multinational corporation with tremendous financial resources and employee headcount, they were able to insert themselves into the OSI process at the level of local committees, obstructing the OSI’s standardization process with ‘constructive criticism’ at every turn. Although they weren’t the primary reason for the OSI effort’s loss of momentum and market share to the ARPA camp, they were a perfect example of the near-impossibility of reaching consensus between different stakeholders for agreements on a common standard, as their efforts led to repeated delays and stalled momentum.
While the “OSI way” was to design standards first and then implement protocols based on those standards, the ARPA team focused on getting software up and running and then writing specifications to match their implementations. This approach had two benefits. The first was that shipping software and having users vet it before iterating and refining it meant that real-world feedback was a prerequisite for initial success or failure of a given protocol. The second is that any attempts at obstruction or subversion of a given standard would have to be done by either “embracing and extending” a protocol or by proposing a competing protocol after the fact, instead of in a design committee before the protocol had a chance to be implemented. This didn’t immunize the TCP/IP protocol suite from attempts at interference by motivated stakeholders, but it did give the protocol designers a head start on their OSI counterparts. It’s easier to find reasons to object to proposed design specification than with actual running code. Software engineers refer to this phenomenon as ‘bike shedding’, a metaphor lifted from Northcote Parkinson’s Law of Triviality (1957), in reference to the tendency of committees to expend effort debating the most inconsequential aspects of a design — e.g., what color to paint a bike shed — as it provides an opportunity for even the most ill-informed participants to have their say in the design.
Years later, David Clark, one of the original ARPA team members, wrote about the nature of this sort of tug-of-war between different stakeholders — not just vendors and standardization bodies, but all of the different actors that played a role in the formation of the Internet:
Perhaps the most important consequence of the Internet’s success is that the common purpose that launched and nurtured it no longer prevails. There are, and have been for some time, important and powerful players that make up the Internet milieu with interests directly at odds with each other. The Internet is not a single happy family of people dedicated to universal packet carriage. There is contention among the players… The Internet, like society in the large, is shaped by controlled tussle, regulated not just by technical mechanisms but by mechanisms such as laws, judges, societal opinion, shared values, and the like. There is no “final outcome” of these interactions, no stable point, and no acquiescence to a static architectural model. Today, the Internet is more and more defined by these tussles.
— Clark, Tussle in Cyberspace: Defining Tomorrow’s Internet [5]
Once OSI’s design standards were deemed ready, local working groups across the world were formed in an attempt to implement the protocols. Instead of following a straightforward path of implementing the standards dedicated to them, each group found ambiguity and fault, room for interpretation, and pushback where they had been seeking clarity and guidance. At the same time, a ‘kitchen sink’ mentality took hold; the idea was that the OSI suite needed to be implemented in its entirety, all at once:
The dichotomy in OSI immensely complicated its output. Because vendors were unwilling to bet on the outcome, they felt they had to implement the whole thing. As with all organizations, it is the internal problems that are the most deadly. There were just too many disparate camps under one umbrella, too many wanting to “lead” OSI, and no common direction for OSI to have ever succeeded. And while there were elements buried in OSI that would have advanced the state of the Internet, it would have been impossible to extract them in the political climate, and there were still many things that were not “right.” The lesson to take away from the OSI experience is plain: No matter how reasonable it may appear, collaborating with the legacy will lead to failure. If the new model does not have enough advantage to stand on its own, it does not have enough to stand with the old guard either.
— Day, Patterns In Network Architecture [6]
In the meantime, DARPA’s focus on getting a working implementation up and running meant that the TCP/IP suite had been built into operating systems, for free, since its inception. Having a working protocol suite available and in the hands of end-users while OSI was still haggling out implementation details gave the Internet protocol a clear edge; implementors had to choose between a protocol suite that had spent years in committee, built to satisfy all stakeholders and yet delight none of them, or one that was already available to them, waiting to be turned on:
[The OSI] was cumbersome, expensive, and not very efficient,” Levilion recalled. “And at the same time came the Internet. Free! So on one side you have something that’s free, available, you just have to load it. And on the other side, you have something which is much more architectured, much more complete, much more elaborate, but it is expensive. If you are any director of computation in a company, what do you choose?”
— Russell, Open Standards And The Digital Age [1], pp. 208
By 1994, the rivalry between TCP/IP and OSI fizzled out in near silence, due in equal parts to TCP/IP’s market dominance – the presence of full working TCP/IP implementations in all major operating systems – and the decision of various governments to abandon their “OSI-only” policies forced the OSI camp to focus on inter-operating with TCP/IP instead of supplanting it.
Takeaways:
- Standards wars waged by vendors are a hindrance to widespread protocol adoption; they can be seen as a negotiating tactic or an opening gambit by vendors or stakeholders hoping to preserve their own sovereignty, or at the very least not to cede it to their competitors. A standards war at the protocol level is by definition a war of attrition, not annihilation.
- Top-down protocol imposition is vulnerable to subversion, as the tension between protocol designers and implementors tends to lead to unresolvable conflicts. The greater the number of stakeholders involved in the design and implementation of a protocol, the higher the likelihood of disagreements and conflict subverting the implementation.
- Getting a protocol implementation up, running, and into the hands of users as fast as possible is more important for reaching critical mass than hashing out implementation details. It’s better to ship a “good enough” implementation than to waste time drafting the perfect standard. Implementors and users alike will take any perfect standard that designers might come up with and find it lacking in some way, in need of debate and bike-shedding; better to prioritize getting user feedback on actual running code first (presuming the protocol is flexible enough to incorporate proposed changes later!).
Chapter 3
If You Can’t Beat Them, Co-Opt Them: CLNP vs IPv6 As The Future Of TCP/IP
The stagnation of the OSI protocol suite may have been perceived as a de facto victory for the proponents of open systems and bottom-up protocol adoption, but it certainly didn’t guarantee the supremacy of TCP/IP, nor did it lead directly to either OSI’s proponents or telecom operators conceding defeat. Instead, their working shifted from protocols that would compete with TCP/IP to work on protocols that hoped to replace TCP/IP as a successor. In this chapter, we’ll investigate two examples of these attempted replacements in the 1990s: the fight over whether CLNP (the last gasp of the OSI protocol suite) or IPng (what we now refer to as IPv6) would replace the existing version of the Internet Protocol, IPv4, in the TCP/IP suite.
(For historical reasons, the first working version of the Internet Protocol is referred to as IP version 4, or IPv4. For more information on the numbering scheme, from IPv0 to IPv10, refer to RFC 1700 or ARIN’s informative blog post at https://www.arin.net/blog/2022/05/12/other-ip-version-numbers/).
Once the TCP/IP suite outgrew the confines of the Network Working Group and International Network Working Group and saw widespread adoption in the academic and research communities, new institutions had to be set up to handle the task of coordinating between different Internet protocol stakeholders. Most important among these were the IAB (Internet Architecture Board), originally formed in 1979, a committee of the IETF (Internet Engineering Task Force). The IAB was a sort of council of thirteen Internet elders, twelve of whom were appointed by the IETF. The IETF itself was open to anyone who wished to contribute; in practice, this meant hundreds of engineers, ranging from employed software developers to hobbyists and enthusiasts. Among the primary duties specified in the IAB’s charter are providing architectural oversight, guidance over the standards process, and stewardship over RFC publication.
The initial TCP/IP protocols were specified and agreed upon with relatively little fanfare or involvement from outside stakeholders; the NWG and INWG were mostly left to their own devices as the IAB and IETF were still in their infancy and growing into their roles. As a result, the relationship between the IAB, the IETF, and the broader Internet community had not been tested in the ways that the OSI standards bodies had been in their attempts at hashing out protocol design with stakeholders. General satisfaction and enthusiasm with the pace of protocol development and Internet adoption meant that questions about what role the IAB might play in the event of disagreements between stakeholders over protocol development and design were matters of idle speculation and theoretical in nature.
That is, until the time came for the next round of protocol upgrades.
Recall from Chapter 1 that layering allows protocols to be built on top of one another and that adoption of higher layer protocols is opt-in by design; a common refrain at IETF meetings is that there is no ‘protocol police’ to enforce the rules. The exception that proves the rule here is the base layer of a protocol suite — in the case of the Internet, TCP/IP — which presents a special case where care needs to be taken to avoid making any breaking changes, as higher layers built atop the base layer are forced to make assumptions about the foundation that they are built on. Any breaking changes made to the base layer must be agreed upon by all network participants.
In the early 1990s the possibility of IP address exhaustion became a popular topic of conversation amongst Internet engineers. Every host on the Internet required an address – a unique identifier to distinguish one host from another — and so the TCP/IP protocol designers had to first come up with an addressing scheme. Unfortunately, their estimates for how many devices would require addresses didn’t account for the explosive rise in popularity of the Internet. Since addresses certainly didn’t appear to be a scarce resource when address allocations first began, they weren’t treated as such. No serious due diligence was performed to determine whether or not a network actually needed to use the amount of addresses it requested, nor was there any mechanism for re-allocating addresses. This meant that over time large, wasted blocks of addresses began to accumulate, laying dormant and unused.
Protocol designers in the early 1990s believed that in the not-too-distant future, every man, woman, and child — not to mention toaster, refrigerator and automobile — would each require their own separate public IPv4 addresses. IPv4’s relatively limited available address space — there can only be 2 to the power of 32, or 4,294,967,296, total IPv4 addresses — along with what were in retrospect “wasteful” allocation policies, meant that there would be orders of magnitude less public IP addresses than there would be human beings potentially interested in connecting to the Internet. This in turn meant that an entirely new protocol would need to be drafted to accommodate all of the new users and devices that would need addresses.
We have the benefit of hindsight that the engineers debating IPv6 adoption didn’t, and so this chapter will deviate a bit from the previous ones in that we’re able to not only review the history of IPv6, but also analyze its ongoing adoption. Over 25 years since it was first proposed, IPv6 adoption is still very much a work in progress, though it appears that the tipping point has been reached, and as of the publication of this text in early 2024, more of the Internet — just barely more, 53% to 46%— speaks IPv6 than not.
We should perhaps start by acknowledging what seems obvious in retrospect: the fears of address exhaustion were overstated at the time of the IPv6 debate. While it’s true that demands for address space have exploded, network designers have met those demands with technical fixes like network address translation (NAT) to allow multiple devices to share a single public-facing address and protocols like DHCP (Dynamic Host Configuration Protocol) that allow for devices to temporarily ‘lease’ addresses for a short period instead of hanging on to them permanently.
Nevertheless, at the time, the problems presented by address exhaustion and inefficient address allocation seemed dire, and the IAB began to take action to address the perceived looming threat.
Since the exhaustion of the address space and the explosion of the routing tables threatened to throttle the Internet’s growth, the council of elders assembled in the IAB began to consider their options with a certain sense of urgency. Lyman Chapin, who served as IAB chairman from July 1991 to July 1993, stated simply in an interview that the shortage of Internet addresses was “definitely the most significant engineering problem on the Internet now.” The IAB responded by convening three workshops in January and June 1991 and January 1992 where more than thirty leaders of the Internet community held extensive discussions that were “spirited, provocative, and at times controversial, with a lot of soul-searching.” The Internet elite eventually reached a consensus around four “basic assumptions regarding the networking world of the next 5–10 years”: the TCP/IP Internet and OSI would continue to coexist, the Internet needed to support multiple protocols and include diverse networking technologies, traffic would be carried both by public and private networks, and the Internet architecture needed to be able to scale far beyond its current capacity.
— Russell, Open Standards And The Digital Age [1], pp. 230
In July of 1991 a warning shot was fired in the form of RFC 1273; its abstract makes clear the desire to extend the Internet protocol suite to include support for OSI protocols:
The Internet is moving towards a multiprotocol environment that includes OSI. To support OSI in the Internet, an OSI lower layers infrastructure is required. This infrastructure comprises the connectionless network protocol (CLNP) and supporting routing protocols. Also required as part of this infrastructure are guidelines for network service access point (NSAP) address assignment. This paper provides guidelines for allocating NSAPs on the Internet.
— RFC 1273 [2]
If there was any ambiguity in the minds of the IETF hoi-polloi as to what this proposed ‘multi-protocol environment’ might look like, the IAB was happy to resolve it. In May of 1992, Lyman Chapin, the IAB chairman at the time, proclaimed in an interview that “The most comprehensive solution is to replace the Internet Protocol in the Internet with the Open Systems Interconnection Connectionless Network Protocol. That idea is already almost universally accepted.” [3]
Chapin’s words, with seemingly the full weight of the IAB behind them, set off a tempest in a teapot. It isn’t clear whether he was trying to rise to meet the challenge of the perceived address exhaustion crisis at the time or lay out a course for the IAB’s role in endorsing protocol updates — and thus implicitly to define the future standards of the Internet. What is clear is that it was met with vociferous resistance at the next IETF meeting, as the rank-and-file engineers that were responsible for operating and maintaining the Internet rebelled. The common perception was that OSI had simply failed in the ‘marketplace of ideas’, and the idea that an OSI protocol was to replace TCP/IP by diktat was a non-starter. The 1992 meeting was a raucous conflagration, as hundreds of IETF engineers voiced their disapproval of what they perceived as a top-down imposition of a protocol upgrade from the IAB. After much hand-wringing and debate, the meeting ended with CLNP — and more specifically, the very concept of a top-down imposition of a new protocol by the IAB — being firmly rejected out of hand. The next two years were a slow and deliberate process of building bottom-up consensus for a replacement for IPv4 (IPv6) and a new understanding of how changes would be adopted by Internet engineers, emblazoned on t-shirts to this day:
“We reject: kings, presidents, and voting. We believe in rough consensus and running code.” [4]
Switching Costs and Lock-In
While IPv6 was free of the ‘taint’ of OSI and the diktat of the IAB, it had a crucial flaw: it was “wire incompatible” with IPv4. That meant that by default there was no way for software or hardware running IPv4 to understand, relay, or interact with IPv6; any piece of hardware or software tasked with forwarding or accepting IPv6 packets would have to be upgraded to be able to do so. There was no way for an existing TCP/IP host or network — that is, anyone connected to the existing Internet — to fully support IPv6 without overhauling every last bit of their existing network to support both the IPv4 and IPv6 protocol stacks. Kuerbis and Mueller, in their paper on the subject, noted the difficulties with this approach to migration:
The proposed dual-stack migration model envisioned by IPv6 designers had two fatal problems:
1) It doesn’t reduce the demand for IPv4 addresses. 2) IPv6 deployers take on additional costs while those who remain on IPv4 do not.
– Kuerbis and Mueller, ‘The Hidden Standards War: Economic Factors Affecting IPv6 deployment’ [5]
In practice, IPv6 had several shortcomings, real and imagined, that needed to be ironed out before it could realistically be deployed alongside IPv4. Multihoming was one; how could IPv6 networks connect to more than an upstream network provider? Privacy was another issue; MAC addresses, unique identifiers that were hard-coded into network cards, were originally mapped into IPv6 addresses as a handy protocol feature. This effectively meant that anyone on the public IPv6 internet could be identified and tracked by the physical hardware that they had purchased. The list went on and on, but all of the snags boiled down to the same core issue: network implementors wanted “IPv4 without the address space limitations”, and IPv6 didn’t offer that; it was instead the closest protocol without address space limitations that rough consensus could be reached on.
The price(s) that needed to be paid to overhaul an IPv4 network to support IPv6 connectivity is an example of what economists refer to as ‘switching costs’, and they’re exactly what they sound like: the costs paid when changing between different technological standards. When switching costs become insurmountable, they lead to ‘lock-in’, or the inability to switch standards even though the desire to do so exists.
In the case of IPv6, IPv4 lock-in was almost guaranteed due to these compatibility issues. If the net benefit of IPv6 adoption — more available address space — accrues primarily to IPv4 networks that wish to grow beyond the limits of IPv4 address space, then for smaller networks that don’t require additional address space, the incentive to bear the costs of adopting IPv6 approaches zero. Paradoxically, since the size of the IPv6 Internet is obviously bounded by the number of networks that are running IPv6, the smaller networks’ unwillingness to bear these costs is the main hindrance to the rate of IPv6 adoption. New networks coming online, with no existing IPv4 installation to upgrade, would eschew IPv6 compatibility if only because so few existing networks were IPv6 compatible.
Two factors were important in overcoming the switching costs associated with the IPv6 transition: technology and time. Technological solutions like carrier-grade NAT (CGN) and IPv4/IPv6 transition mechanisms (of which there are too many to note: 6-in-4, Teredo, NAT64, IPv6 tunnel brokers, and more) were developed with the aim of connecting IPv4-only networks to the IPv6 network with as little cost to the IPv4 networks as possible. These technologies and their deployment were in effect subsidized by larger networks that, either by choice or preference, desired to transition to an IPv6-only network but still needed to maintain connectivity to the existing IPv4 internet.
The use of these ‘converters’ is a common strategy for enabling transition between older and newer technologies; if the converter is one-way (for example, a VHS-to-DVD converter device) it can increase the pace of transition by making the switching costs more bearable (in this example, removing the burden of replacing existing VHS titles and leaving only the costs of purchasing new DVD players). However, if the converter is two-way, as IPv4-to-IPv6 translation devices must necessarily be, they can also hinder the transition by reducing the risk of the older technology being orphaned.
The second factor, time, helped to overcome this hindrance. As more IPv6 networks came online, more and more software and hardware began supporting IPv6, while available IPv4 address space began to dwindle. The balance between switching costs and incentives to switch began to tip in favor of IPv6. New IPv6 networks had to reach a critical mass to be able to compete with the demands made on software and hardware vendors by operators of existing IPv4 networks, but they were unburdened by the golden handcuffs of their IPv4-native predecessors and free to focus on building IPv6-only infrastructure, safe in the knowledge that they at least wouldn’t have any worries about address space exhaustion for the foreseeable future.
I can’t claim to speak for other network operators, only myself, but I believe that if decades ago I had been given the opportunity to peer into the future and see how long IPv6 adoption would take, I would have advocated for exploring different approaches, or at least spending more time debating them instead of imagining a crisis — address-space exhaustion — that never materialized. IPv4’s inability to deal with the explosion in demand for address space before IPv6 was fully deployed proved that simply using IPv4 address space more effectively could allow networks to rely on IPv4 for decades longer than originally anticipated. Even now, in the post-IPv4 era, since new IPv4 addresses are not being allocated, and only reassigned from old ones, we are still seeing inefficient IPv4 networks being decommissioned and new networks purchasing their IPv4 address space at auction.
In hindsight, the imagined crisis was one of only two possible outcomes that could have led to an understanding amongst network operators of the necessity to overcome lock-in and move forward with the adoption of a new protocol, the other option being an actual crisis. I’m not sure of the provenance of the quote “Better to have, and not need, than to need, and not have,” though it’s most often attributed to Franz Kafka. I am quite sure that it is better to have an option for overcoming a hard protocol limit and not need it than to need it and not have it.
Takeaways:
- Since the 1992 IETF meeting rejecting the imposition of CLNP as a successor protocol to IPv4, ‘Rough consensus and running code’ has been the mantra of protocol designers, for better or worse. There are no protocol police to enforce the rules, and no kings to dictate them.
- The higher the switching costs associated with new technologies, the longer users will take to transition to them. Making IPv6 wire-incompatible with IPv4 all but guaranteed that the transition to IPv6 wouldn’t begin until IPv4 address exhaustion set in and IPv4 networks could no longer grow. Even then, converter technologies for IPv6 needed to be readily available to allow new IPv6 networks to connect to the existing IPv4 Internet.
- Base layer lock-in for networks is a strong inertial force; the longer that a functional network is providing utility to a majority of its users, the lower the incentive to make any changes that might benefit new users yet to be onboarded. In order to overcome this lock-in, either the provided utility must somehow begin to degrade for the existing set of users, or the perceived benefit to new users must be high enough to make the changes worth the effort to implement.
Chapter 4
DNS In Anger - From Centralized Directories to Distributed Names to Challenging Westphalian Sovereignty
In Chapter 3, IP (Internet Protocol) addresses were briefly discussed in the context of the debate over potential IPv4 address exhaustion, the concern being that there might not be enough addresses for every Internet-attached device. Most Internet users rarely actually have to deal with IP addresses, as they’re meant for software to understand, not humans. A typical IPv4 address is a 32-bit integer represented as four octets, each separated by a dot; for example, 192.168.1.1 would be the traditional representation for the 32-bit integer 3232235777.
Try keeping a list of those addresses straight in your head, and you’ll soon see the difficulty; 32-bit integers are not exactly as easy to remember as telephone numbers (and who even remembers telephone numbers these days?). But that’s just the tip of the iceberg; what to do when a computer needs to update its IP address or differentiate between messages meant for different recipients that happen to reside at the same address? How do you communicate that one address has “replaced” another without knowing every last person and piece of software that’s learned of the existence of the first address?
Addresses were never meant to solve these problems, and so these problems were never contemplated when the IPv4 addressing scheme was first drafted. An IP address is a location, similar to a physical location in many ways; it’s not meant to be a unique identifier any more than a mailing address is. For those, we need names.
What’s in a Name?
“A name indicates what we seek. An address indicates where it is. A route indicates how we get there.” This quote, often attributed to Jon Postel, originated with John Shoch’s 1978 paper “A Note on Inter-Network Naming, Addressing, and Routing” [1]. Network names had been discussed prior to this, but Shoch’s paper managed to capture in one succinct phrase exactly what problem naming was meant to solve — the identification of resources — and why it was important to separate that problem from the problem that addressing was meant to solve – identifying the location of those same resources.
In the early days of the Internet, the way to handle the problem of binding resources to their location — that is, the mapping of names to addresses — was for a single file, HOSTS.TXT, to be hosted in a central location on an academic server that other nodes could poll periodically for updates and rely on as a source of truth for name mappings. Recall from Chapter 1 that the switch from NCP to TCP on January 1, 1983, was coordinated between about 400 nodes; keeping a single file synchronized between them all with a single source of truth was a tractable problem [2]. HOSTS.TXT was managed and hosted by a trusted third party, which made sense right up until it didn’t.
HOSTS.TXT was maintained by SRI’s Network Information Center (dubbed “the NIC”) and distributed from a single host, SRI-NIC. [1] ARPAnet administrators typically emailed their changes to the NIC and periodically ftped to SRI-NIC and grabbed the current HOSTS.TXT file. Their changes were compiled into a new HOSTS.TXT file once or twice a week. As the ARPAnet grew, however, this scheme became unworkable. The size of HOSTS.TXT grew in proportion to the growth in the number of ARPAnet hosts. Moreover, the traffic generated by the update process increased even faster: every additional host meant not only another line in HOSTS.TXT, but potentially another host updating from SRI-NIC.
When the ARPAnet moved to TCP/IP, the population of the network exploded. Now there was a host of problems with HOSTS.TXT (no pun intended):
Traffic and load
The toll on SRI-NIC, in terms of the network traffic and processor load involved in distributing the file, was becoming unbearable.
Name collisions
No two hosts in HOSTS.TXT could have the same name. However, while the NIC could assign addresses in a way that guaranteed uniqueness, it had no authority over hostnames.
— Larson, DNS On Windows 2000 [3]
The problems were that the file, and hence the costs of its distribution, were becoming too large, and that the centralized control of updating did not fit the trend toward more distributed management of the Internet. Simple growth was one cause of these problems; another was the evolution of the community using HOSTS.TXT from the NCP-based original ARPANET to the IP/TCP-based Internet. The research ARPANET’s role had changed from being a single network connecting large timesharing systems to being one of the several long-haul backbone networks linking local networks which were in turn populated with workstations. The number of hosts changed from the number of timesharing systems (roughly organizations) to the number of workstations (roughly users). This increase was directly reflected in the size of HOSTS.TXT, the rate of change in HOSTS.TXT, and the number of transfers of the file, leading to a much larger than linear increase in total resource use for distributing the file. Since organizations were being forced into management of local network addresses, gateways, etc., by the technology anyway, it was quite logical to want to partition the database and allow local control of local name and address spaces. A distributed naming system seemed in order.
— Mockapetris, Development Of The Domain Name System [4]
This distributed naming system was designed by computer scientist Paul Mockapetris at the behest of Jon Postel, at the time a researcher at the SRI, and was specified initially in RFC 882 and 883 as the Domain Name System (DNS). It proposed an elegant solution to the problems with the existing HOSTS.TXT approach: a distributed, hierarchical database of names, with control of updates and changes delegated at every opportunity. Name servers would be granted the authority to respond to requests for information about different names from the top down; e.g., a given set of servers would be in charge of delegating requests for any domains ending in “.com”; those servers in turn would direct requests for specific names like “example.com” to a different set of servers that the administrator of the “example.com” domain had granted authority to broadcast any information related to that name; other servers called resolvers would wait for requests from users for names like “example.com”; upon receiving a request the resolvers would try to ‘resolve’ the request by querying the root servers to discover the names of the servers in charge of the “.com” label; after receiving a response, the resolvers would then query the servers they had discovered were in charge of the “.com” label for directions to the servers in charge of “example.com”, then ask the servers in charge of “example.com” for the addresses and related information associated with “example.com”, and finally these servers would cache the results for a period of time, so that any end users who wanted to know details about “example.com” would be able to receive a response directly, without having to repeat the whole question-and-answer resolution process for every single request made.
The design was impressively sturdy; at any tier of the hierarchy, more servers could be added to handle increased requests. As more names were added to the “.com” namespace, more servers could be given the task of directing users to the name servers in charge of individual domain names. As more traffic was directed to individual domain names, more servers could be given the task of answering requests for addresses bound to those names, and so on. Making extensive use of caching, where a stale copy of an answer would be stored locally for a set amount of time before a server would check to see if there had been any updates, in addition to having Internet service providers run their own servers for resolving name requests made by their users, contributed to the robustness of the system.
Before the DNS was implemented, the NIC administered all changes to HOSTS.TXT; all name additions, updates, and removals went through them. By removing the need for a trusted third party for most of these operations, and delegating the process of making updates to name mappings and transferring ownership of names to the original registrants of the names themselves, the DNS, almost accidentally, granted the domain name registrants a limited form of property rights. No longer were host identifiers merely lines to be added or removed to a centralized database by a single authority; once created, they were de facto assets that could be transferred between, and updated by, their ‘owners’. At least in the eyes of their holders, these names were now assets that they controlled, with some notional value associated with them.
In these early days of the DNS, before the explosion of popularity of the World Wide Web in the mid-1990s, only about 60,000 domain names were in existence, which meant that there wasn’t much trading of these assets, and the property rights implicitly granted to name holders — the right to transfer and reassign them — were rarely exercised. All that changed in 1995 when Network Solutions, the sole registry for the .com domain space, was granted the right to charge for domain registrations. This meant that the newly created identifier assets could now be assigned a price. If asset holders are able to express a disagreement on the value of an asset but agree on a price — by buying or selling the asset in question — then it can be said that a market exists for those assets. Unsurprisingly, in retrospect, with the (almost accidental) creation of this market, a land rush for names ensued. Fortunes were made and lost, and an entirely new breed of speculator —‘the domainer’, as people who bought and sold large numbers of domains in the hopes of reselling them at a greater value began to refer to themselves — was born.
The behavior of the speculators taking part in this land rush bring to mind the example of the speculators taking part in the California Gold Rush of 1847 — the mad scramble for unclaimed assets in a new frontier; the pareto-optimal allocation of assets (where the top echelon speculators became fabulously wealthy, the median made modest profits, and the long tail was materially worse off for having undertaken the endeavor); and, of course, the shared unspoken understanding in the value of the underlying asset being worth the effort spent to acquire it. Put plainly: everybody knew that domain names had value, because everybody knew that everybody knew that domain names have value. In 1958, Thomas Schelling referenced this kind of shared belief in his seminal paper “The Strategy of Conflict Prospectus for a Reorientation of Game Theory”, which first described what we now refer to as ‘Schelling Points’, focal points that members of an uncoordinated mob focus on:
…the greatest example of all may be the monetary role of gold, which can perhaps be explained only as the “solution” of a coordination game. (A common household version of the coordination game occurs when two people are cut off in a telephone conversation; if they both call back, they only get busy signals.)
— Schelling, The Strategy of Conflict Prospectus for a Reorientation of Game Theory [5]
This land rush dynamic repeated itself again to a much smaller degree with the rise of country domains as vanity identifiers (kicked off by bit.ly’s use of Libya’s country domain for their URL shortening service, and seen in country level domains like Anguilla Island’s “.ai” being used by artificial intelligence companies, or television companies using the Tuvalu “.tv” country domain); and then again with the opening of alternative global top level domains (gTLDs), like .xxx and .cash. Each successive land rush pulled in a smaller number of participants and a smaller aggregate reward, as market participants rushed in to buy and sell assets until the supply of greater fools to sell to was exhausted.
While the initial land rush for domain names may have served to cement the idea that the property rights embedded within them conferred some sort of value, the borders of those rights had yet to be explored. The first attempt at mapping these borders came in 1998, when it was discovered that Network Solutions had begun filtering the registration of certain objectionable words, after an application to register the name shitakemushrooms.com was denied. The revelation that, de facto, Network Solutions had exerted a measure of control over the registration process, where they had previously served as uninterested administrative functionaries, was in retrospect the proverbial canary in a coal mine.
Names serve as common identifiers that users can converge and agree on to identify and consume resources. We are given names at birth by our parents and have them validated by those who govern the jurisdictions we are born into; it is a relatively rare occurrence for a chosen name to be prohibited or rejected, and the “revocation” of a birth name is unheard of. The DNS was built to solve a technical problem, and the solution in turn forced DNS users to search for answers to questions that none of the designers of the DNS had thought to ask: who benefits from the registration of names? What authority is in charge of adjudicating disputes over names? What rules should they be held to?
These questions found their answers by piecemeal in repeated attempts over the years by various stakeholders, among them DNS server administrators, registry operators, and service providers, to assert control over the question-and-answer process by subverting assumptions of other participants in the DNS. Looking back on these attempts decades later, Paul Vixie, author of the popular Berkeley Internet Name Domain (BIND) DNS server software, described them as ‘the DNS wars’, and in recounting them, he came to the conclusion that the DNS itself had become a challenge to the notion of post-Westphalian sovereignty that underpins our modern understanding of the nation-state.
In a world where one corporate entity provides the operating system for some 90% of all handheld computers (Google with Android), where the same corporate entity is used as the browser by more than 70% of users (Google with Chrome) and where a single open resolver service is used as the preferred resolver by some 10% of users (Google with Quad 8), then if the Internet was regarded as a distinct realm of human activity on a peer level with realms defined by land and sea, then Google’s ability to assert sovereign rights over huge swathes of the information technology space, based on the ability to defend its assets, must be admitted. By that reasoning, the Westphalian model of nation-states applies here as well, and regulation is necessarily replaced with negotiation…
Part of the new world order is that the space defined by the actions of applications is well beyond the traditional domain of communications regulation and even beyond the domain of regulation trade and commerce. Applications use communications [protocols] as a service, but they do not define it. This is a new space, and the sovereign rights of nations are finding it extremely challenging to assert that they have primacy when they cannot defend their borders and cannot unilaterally enforce their will. Is the new definition of information technology nationhood equating to the ability to impose the national will on end-users irrespective of physical land and sea borders?
— Vixie, ‘The DNS Wars’, Presentation to NANOG 2019-10, https://pc.nanog.org/static/published/meetings/NANOG77/2033/20191028_Vixie_Keynote_Dns_Wars__v1.pdf [6]
As a concrete example of how the DNS can challenge “the sovereign rights of nations”, consider the following scenario: suppose that a DNS stakeholder — perhaps a registry operator like Network Solutions, a large DNS resolver operator such as Google, or a CDN operator such as Cloudflare — were to unilaterally decide to redirect certain requests for Ukrainian government domains (that is, anything in the top-level domain ending in .ua, such as www.gov.ua) to DNS servers under the control of the Russian Ministry of Information. What actions could the Ukrainian government take to defend and assert their sovereignty? The Peace of Westphalia established in 1648 assumed that sovereigns would assert control over land, sea, and air and defend them accordingly; the idea that the information space could be contested in such a way would have been simply incoherent at the time.
Vixie identified six different eras in the ongoing DNS wars; starting with the initial transition from a distributed database of names to a commercial service charging for domain registrations in the 1990s; moving on to the SiteFinder debacle, where all queries for unregistered domains – which would normally receive empty responses — were instead redirected to a landing page managed by a corporate for-profit entity, advertising their services; and on to the present day push by browser providers to implement DNS-over-HTTP (DoH) in an attempt to assert some modicum of control over the question-and-answer process of web browser users entering names of sites they wish to view.
In each of these eras, we can see echoes of “the tussle” first mentioned in Chapter 2, an ongoing contest between different stakeholders with interests in conflict with each other. What is particularly notable is that governments and states have no direct role to play in these tussles; they are instead left to negotiate with participants after the fact:
When the Mozilla Foundation announced its intention to ship the next version of its Firefox browser with a default setting that both enabled DoH and directed DoH to Cloudflare’s Open Recursive Resolver service, the United Kingdom called for a “summit meeting” with Mozilla. This was not the enacting of legislation, the adoption of a regulation or any other measure that is conventionally available to a nation-state, but a meeting of a different nature. Is this the resurgence of quasi nation-states such as the Honourable East India Company, a joint-stock company that ran its own army (twice the size of the British Army at its peak in 1803), fought its wars and established and defended its borders in a manner that was fully consistent with the actions of any other nation-state?
— Vixie, The DNS Wars [6]
Reading this paragraph, it is difficult to imagine the rise of a new East India Company as being either the intended or unexpected outcome of Mockapetris’s project to build a distributed naming system back in 1983. The original assignment was to remove the central point of failure that the HOSTS.TXT file represented. Mockapetris succeeded in doing exactly that. With the removal of a central authority that could be coerced or targeted, a power vacuum emerged, one that stakeholders have tussled over ever since.
Names and Control
The assignment and revocation of names for resources has become a cornerstone of Internet authority. From information services like AOL and Prodigy using keywords to access content, to IRC networks designating channel names, to social media platforms assigning unique usernames, the ability to bind a name to a resource, or to restrict or revoke it, is a fundamental locus of control. It is a right that users rely on as well as a privilege that central authorities require to maintain legitimacy.
For a real-world example, I might tomorrow decide to change the address associated with my physical location to 1600 Pennsylvania Avenue and my given name to The Coca-Cola Company; perhaps a few friends would be willing to go along with the joke. But I cannot expect mail carriers to begin delivering packages destined for me to my new address or for businesses or governments to enter into contracts with me using my new identifiers. If I were able to somehow unilaterally seize these identifiers from their rightful owners without any interference, then the naming system would be unusable. In practice, I would have had to register these new identifiers with a central authority — my local, state, or federal government — that would ultimately be charged with assigning these identifiers and adjudicating disputes over identifier assignment according to existing treaties and agreements. Nick Szabo’s 1998 paper ‘Secure Property Titles with Owner Authority’ describes agreements over these identifiers in terms of property rights:
In all cases of property rights there is a defined space, whether a namespace or physical space, and the task is to agree on simple attributes of or rights to control subdivisions of that space. In some cases a name or other symbol corresponds to a person or object owned or controlled by that person. For example, Internet users must agree on which domain name corresponds to which web site operator. In other cases we are simply concerned with control over a subdivision of the space. With real estate we must agree on who owns various rights (to occupy the surface, to mine the minerals under, etc.) to a piece of land. With radio spectrum we must agree on who owns what range of frequencies and in what physical space (or transmitting power as an easily observed approximation of physical space used).
It is the author’s hypothesis that all such agreements of control, including control over the semantics of symbols, to be made and respected across trust boundaries are problems of agreeing on and maintaining property rights… Highlighting the property rights nature of public directories also highlights the limitations of these mappings — for example that names, addresses, and other symbols whose semantics are controlled by a person can often be delegated, just as property can be given or rented.
— Szabo, Secure Property Titles With Owner Authority, https://nakamotoinstitute.org/secure-property-titles/ [7]
Although the Domain Name System is a self-contained namespace that exists outside of any single state’s control, domain holders themselves — that is, domain registrants that are neither in the business of operating registrars that assign names nor running resolvers that answer questions about the bindings of names — have few tools to exercise their rights over their property. Domains can be seized and redirected by government decree, if the underlying technical infrastructure — that is, the nameservers and their registries — are located in physical territory that the governments can assert control over — but as names in a global namespace are available in multiple overlapping jurisdictions, the ability of states to exert control over them can be subverted. Consider the example of The Pirate Bay, a website sharing BitTorrent magnet files that allowed users to download copyright-protected mass media:
The neverending cat and mouse game between the world’s biggest pirate site and government authorities continues: thepiratebay.se now belongs to Sweden.
It’s the first default domain seizure since the legally plagued website went offline for three months — a shockingly long time for the Pirate Bay — in late 2014 and early 2015.
That’s not happened again this time. Trying to access thepiratebay.se now simply redirects visitors to thepiratebay.la, a Los Angeles suffix.
According to the Swedish newspaper Dagens Nyheter, the nature of the seizure means that domain name thepiratebay.se, suspected of enabling illegal copyright infringement, now belongs to the Swedish government, and can’t be sold off. The paper also reports it’s the first time that a Swedish prosecutor has successfully demanded an Internet address “permanently disappear.”
Not fans of subtlety, the Pirate Bay currently sports the logo of a hydra — you know, the whole “cut off one head and several more spring up” thing — surrounded by the suffixes .mn, .gd, .la, .am, .gs, a reference to five other domain names the site has at the ready.
— Collier, The Pirate Bay lives on despite Swedish Domain Seizure, [8] The Daily Dot https://www.dailydot.com/debug/the-pirate-bay-domain-sweden-seized/
The case of The Pirate Bay shows how it is possible to circumvent state control of the DNS by using registrars, resolvers, and businesses in different jurisdictions; an edict by one state to restrict a name can only be enforced upon the rights holders occupying physical space under their control. In practice, several of the top-level domains (TLDs), including the most popular TLD, .com, fall almost exclusively under the physical jurisdiction of a single state, and evading its control requires significant effort.
In cases where ownership of a domain is in dispute not because of government edict but instead because of competing civil claims of ownership, domain holders are compelled to submit to arbitration through a non-governmental organization. Instead of a state, it is the World Intellectual Property Organization (WIPO), a United Nations body that has been tasked by ICANN with adjudicating disputes between stakeholders in different jurisdictions through its administration of the Uniform Domain Name Dispute Resolution Policy (UDRP). The UDRP is a process designed to resolve disputes related to domain names, particularly cases of domain name cybersquatting and trademark infringement. It is a de facto supranational treaty laying out the civil rights and responsibilities of the holders of property rights to domains.
Several alternative sets of root servers exist, managed either by states that wish to exert control over their own separate namespaces or ideologically motivated server operators rejecting these sorts of restrictions and offering their own set of rules governing names. Accessing these alternative namespaces for average Internet users would require all of five minutes of effort to reconfigure their operating systems or routers to access them; since the default set of root servers requires no effort at all to use, the combination of relatively high switching costs and lack of user awareness of the existence and importance of alternative namespaces has limited their popularity. While any user is able to in theory choose their own set of name servers with their own sets of rules over name ownership, in practice, making names available to other users of the Internet means playing by the same sets of rules that other users have unknowingly opted in to. While there is no central DNS authority for states to coerce or subvert, each tier of the hierarchy can serve as a de facto chokepoint for states to target or for bodies like WIPO to assert control over.
Takeaways:
- The Domain Name System was built to replace a centralized name mapping system and primarily focused on making the name mapping system more distributed and maintainable, removing a single point of control over names.
- The ‘dotcom’ gold rush of the late 1990s cemented the shared understanding of the value of domain names amongst Internet users. This value was derived from the de facto property rights granted to domain holders, who are able to buy, sell, and modify the contents of domain names. There were future gold rushes built around similar domain name dynamics, with each wave of speculation leading to diminishing returns.
- The DNS succeeded beyond its creator’s wildest dreams and, in doing so, created an accidental chink in the armor of Westphalian sovereignty. While nation-states can assert control over physical territory, their inability to exert complete control over all aspects of the domain name system compels them to negotiate with participants in the DNS, specifically the registrars that allow titles for names to be issued and the resolvers that answer questions for names. Conversely, domain name holders that do not participate in the operation of the underlying DNS infrastructure are compelled to obey government edicts in criminal cases or submit to WIPO arbitration for civil claims, with their only apparent resources being jurisdictional arbitrage.
Chapter 5
Internet Routing: You Can’t Get There From Here
Recall from Chapter 4: “A name indicates what we seek; an address indicates where it is. A route indicates how we get there.” Routing protocols advertise and distribute routes between network gateways so that hosts behind one gateway know the path between systems on the Internet that must be taken to establish connections to hosts behind another gateway. Accordingly, the history of routing protocols is inextricably linked to the underlying physical structure of the Internet, as the connections between gateways are physical telecommunications links in the real world. It follows that before we can begin to examine the history and evolution of routing protocols, we’ll need to take a bit of a detour and discuss the evolution of the Internet’s physical topology. For the purposes of this chapter, we’ll describe this evolution in terms of six distinct epochs.
The first epoch, when the ARPANET was the de facto backbone of the network we now call the Internet, lasted from the ARPANET’s inception in 1973 through the TCP flag day in January of 1983. The second epoch began in October of 1983 with MILNET, the collection of military networks that actually relied on ARPANET for production service, splitting away from the rest of the ARPANET. The third epoch began in 1986, when phase one of the NSFNet, the successor to the ARPANET, became operational. This involved six so-called “Fuzzball” gateways across the country being connected. The fourth epoch began in June 1988, when the non-profit Merit network assumed operation of the NSFNet backbone. In the fifth epoch, the LSI-11 mailbridges separating the ARPANET and MILNET were replaced with Butterfly gateways in December of 1988. And finally, in the sixth epoch, the NSFNet backbone was replaced by a collection of NAPs (Network Access Points), the forerunners of today’s IXPs (Internet Exchange Points).
These epochs in turn map to the three phases of the Internet’s structure: the ARPANET proper, the three-tiered NSFNet, and today’s current NAP/IXP paradigm.
During the first epoch, after the flag day cutover from NCP to TCP/IP on January 1, 1983, any host that was connected to the Internet by definition had to have an IP address. Since the DNS had yet to come into wide usage, the mappings between common names and addresses were all handled locally and manually; this meant that only a handful of administrators needed to even be aware of what addresses were, as users would continue to connect to hosts by name, regardless of the address associated to the name.
Connections to hosts in remote networks were facilitated by the DARPA Internet Gateway, described in RFC 823, running the Gateway-to-Gateway protocol (GGP). GGP was not the first attempt at a proper protocol that would enable gateways — what we now call routers — to send packets to each other, but it was the first that was formally specified in an RFC:
Early versions of gateway software were implemented using the BCPL language and the ELF operating system. This implementation evolved into one which used the MOS operating system for increased performance. In late 1981, we began an effort to produce a totally new gateway implementation. The primary motivation for this was the need for a system oriented towards the requirements of an operational communications facility, rather than the research testbed environment which was associated with the BCPL implementation….
The gateway must make a routing decision for all datagrams that are to be to forwarded. The routing algorithm provides two pieces of information for the gateway: 1) the network interface that should be used to send this datagram and 2) the destination address that should be put in the local network header of the datagram.
The gateway maintains a dynamic Routing Table which contains an entry for each reachable network.
— RFC 823, https://www.rfc-editor.org/rfc/rfc823#section-4.4.6 [1]
GGP, described in RFC 823, was a strange choice for an internetwork routing protocol, as it was designed for routing internally between gateways instead of between remote systems. However, the ARPANET in practice was de facto a single “autonomous system”, the name given to remote networks with their own unique topologies. In the first epoch, the ARPANET backbone was administered by Bolt, Beranek, and Newman (BBN), with the GGP protocol being used to exchange routes between gateways internal to ARPANET, and the Exterior Gateway Protocol (EGP) being used to exchange routes with other external networks connecting to the ARPANET. EGP was described first in RFC 827 before being formally specified in RFC 904.
This combination of GGP and EGP worked well enough during the first epoch; the second epoch, when MILNET split away from ARPANET in October of 1983, demonstrated the weaknesses of these protocols in an internetworking environment. The MILNET split was meant to move sensitive traffic to a private network while still allowing military labs and defense and intelligence agencies to access the public ARPANET. Hosts that wanted to send traffic to both networks without direct connections would take inefficient, circuitous paths to get there; balancing traffic flows between networks became an exercise in futility, leading to congested network links and a poor experience for users. The routing protocols that had been battle-tested for the original ARPANET design were ill-suited for this new split network structure. As luck would have it, very shortly after the MILNET/ARPANET split, the National Science Foundation began to build its own network with a backbone: NSFNet. Crucially, the NSFNet made use of ARPANET’s TCP/IP protocol suite and connected directly to ARPANET via several gateways. The beginning of NSFNet’s operations in 1986 serves as a clear marker for the start of the third epoch:
A limitation of EGP is that it only allows for a tree-like network topology. That means that there can only a single path between any two parts of the network. But that’s not a big deal back in the 1980s, as in 1986, the National Science Foundation created the (56 kbps!) NSFNet which is the “backbone” of the internet. As such, all long distance traffic passed through the NSFNet, which was upgraded to 1.5 Mbps in 1988 and 45 Mbps in 1991. EGP introduced the notion of “autonomous systems”, with each separate network engaging in EGP routing having its own autonomous system (AS) number.
— Van Bejnum, “GGP, EGP and 25 years of BGP: a brief history of internet routing”, retrieved from https://www.routerfreak.com/ggp-egp-and-25-years-of-bgp-a-brief-history-of-internet-routing/
If the MILNET/ARPANET split revealed the flaws in the GGP and EGP routing protocols, the NSFNet backbone gave those protocols a new lease on life with a bit of a clever hack: the NSFNet’s gateways, all six of them, ran a third routing protocol, HELLO, specified in RFC 891, that connected the old ARPANET and the new NSFNet backbone. HELLO managed to bridge the gap between GGP and EGP, at least for the six gateways that it ran on. (I mention HELLO here only for the sake of completeness. A protocol that only runs on six hosts, all managed by the same administrators, doesn’t seem likely to offer us enough insights to be worth analyzing…)
NSFNet became wildly popular — perhaps too popular for its administrators, who found themselves in a near-constant battle almost from the start to outrace the pent-up demand of ARPANET users for congestion-free connectivity. Connectivity to the new backbone allowed users to escape the constraints of the ARPANET; NSF was able to continuously upgrade the speed of the backbone, and in doing so, became a more attractive option for providing connectivity to hosts than the heavily congested ARPANET. The timing couldn’t have been better; the managers of the ARPANET, mulling over different options for fixing the problems of network congestion, decided that the quickest solution would be to simply transition hosts over to the new NSFNet instead of attempting to upgrade their way out of the congestion issues. NSFNet’s choice to use TCP as its underlying transport protocol played no small part in this decision. The ‘switching costs’ of migrating to a new protocol weren’t a concern, so there was only potential upside — speed, speed and more speed — for hosts willing to make the switch. The transition from the third, fourth, and fifth epochs all happened during this period in the NSFNet’s history, with no real change in the underlying routing protocols used.
In June of 1988, the non-profit corporation Merit bid on a contract to upgrade the NSFNet backbone from 56 Kbps links to 1.5Mbps T1 lines, and upon winning the contract became the de facto administrator of the Internet’s physical superstructure. In late February of 1990 the old ARPANET was decommissioned, as the new NSFNet backbone had effectively taken its place. On September 17, 1990, Merit, IBM, and MCI publicly announced the formation of ANS, a new corporation that was allowed to solicit commercial users. ANS then divided the network it managed into two pieces, NSFNet and ANSnet, which used the same physical facilities but had different acceptable use policies. This allowed ANS to provide services that would have otherwise violated the NSF’s acceptable use policy prohibiting commercial activity. A year later, ANS set up a for-profit subsidiary to sell commercial access to the NSFNet — a controversial decision, as it effectively allowed a private entity to profit by offering access to a government-subsidized network. In late 1991, acknowledging the tension between the NSF’s desire to provide stable backbone connectivity and the desire for commercial access to the Internet, the NSF announced a new project development plan that proposed the creation of multiple competing backbones instead of just the NSFNet. This addressed concerns from network providers brought on by the formation of the ANS and extended the cooperative agreement with MERIT for eighteen months. The plan proposed the use of Network Access Points (NAPs) as interconnection points for federal, regional, research, and commercial networks, allowing them to share information and users while still allowing users to choose their network provider:
During the privatization, Kahin commented that the NSF’s redesign fundamentally reshaped the infrastructure of the Internet. Instead of one major backbone — the NSFNet — the new network depended on multiple backbone providers. The NSF created the NAPs to interconnect these networks and to prevent a balkanized Internet. While this design was inspired by the CIX and the FIX interconnection points, it went substantially beyond them. This new network was designed to create competition for backbone services.
— Kesan and Shah, “Fool Us Once Shame On You, Fool Us Twice Shame On Us” [4]. Retrieved from https://openscholarship.wustl.edu/cgi/viewcontent.cgi?article=1399&context=law_lawreview
The transition from a monolithic backbone to a series of network access points (now known as Internet Exchange Points, or IXPs) redefined what it meant for a network to be ‘connected to the Internet’. In the ARPANET and NSFNet epochs to be connected to the Internet meant sharing access to the same backbone provider as the other organizations – primarily universities and research labs – that had signed up for access. In the IXP era, it meant have the ability to (at least) receive information on how to connect to other networks, and ideally to advertise information on how other networks could reach yours. This transition away from a monolithic backbone structure opened the door to the final epoch of Internet routing protocols and the successor to EGP, BGP.
BGP is perhaps the first example in the Internet (or at least the first substantially worked-out example) of a protocol designed to shape industry structure. The predecessor of BGP was designed in the era when the ARPANET was the backbone of the Internet. Exterior gateway protocol, or EGP, assumed a hierarchical pattern of interconnection among the regions (the ASs), with ARPANET and then NSFNet at the top of the hierarchy. If EGP had become the routing protocol for the commercial Internet, a single commercial provider would have taken the place of NSFNet, in a position that seems close to an architecturally created monopoly.
— Clark, Designing an Internet [5]
The first version of BGP, BGP version 1, was defined in RFC 1105. It’s been known ever since as the ‘three napkin protocol’, because that’s what the initial specification was written on, by a team of Cisco and IBM engineers in January 1989 at the 12th IETF meeting in Austin, Texas. This version of BGP, like EGP before it, only supported hierarchical network topologies (e.g., up, down, and horizontal connections). It was BGP version 2, defined in RFC 1163 in 1990, that added support for carrying multiple routes to the same destination, as well as a new UPDATE message type. BGP version 3, defined in RFC 1267 in 1991, added support for carrying IP multicast routing information and introduced a new message type called the “NOTIFICATION” message, which is used to report errors and other important information between BGP speakers. BGP version 4, defined in RFC 1771 in 1995, is the current version of the protocol and the standard for routing on the Internet. This version added support for carrying Classless Inter-Domain Routing (CIDR) routes, which is useful because it meant that more of the IPv4 address space could be chopped up than the original “Classful” network allocations. A discussion of classful vs. classless addressing is out of scope here, but it’s important to note that in supporting CIDR, BGP-4’s design helped to conserve the limited number of IPv4 addresses available and improved routing efficiency on the Internet.
To recap, GGP was specified in 1982 and EGP in 1984; both managed to last more than a decade before being replaced by BGP, which went through four iterations from 1989 to 1995, with no successor protocol taking its place in the decades that have elapsed since. BGP-4 was apparently ‘just good enough’ for the needs of today’s Internet.
Despite BGP-4’s longevity, it is not without its flaws. In particular, the protocol has no way of guaranteeing the authenticity of routing updates. Instead, each BGP speaker implicitly trusts its peers to announce routes to addresses they own; predictably, every few years, this assumption breaks as a misconfigured (or even malicious) BGP speaker announces routes for addresses that are assigned to another organization. For example, MAI Network Services accidentally leaked most of the routes they knew about to their peers back in 1997 and blackholed a significant portion of the Internet; Pakistan Telecom intentionally tried to blackhole access to YouTube inside of Pakistan in 2008 and ended up blackholeing it for half of the outside world as well. Most accidental BGP route hijackings aren’t quite as dramatic as these, but they do happen on a consistent basis. What’s worse: not all of them are accidental. BGP route hijacks are a common tool used by advanced threat actors.
There are at least two proposals to fix this core problem with BGP-4: Secure BGP (sBGP) and Secure Origin BGP (soBGP). Both of them take different approaches to solving the same problem; neither was able to gain enough momentum to overtake BGP-4. Why?
We again return to the switching costs we first learned about in Chapter 3 — the costs paid by implementors when changing between different technological standards. In the case of BGP, these costs are steep: new firmware would have to be rolled out to routers supporting a new protocol, requiring maintenance and downtime. Operators would have to be trained in best practices for implementing that protocol, and finally connections would have to be updated one by one, peer by peer, organization by organization. This is almost infeasible in practice. The transition to the Network Access Point/Internet Exchange Point (NAP/IXP) paradigm kicked off a rush of peers eager to establish connections with each other (the theory being that better connected peers were more valuable). Due to the NAP model of interconnection, the number of connections between BGP speakers is exponentially greater than the number of connections between EGP speakers in the old NSFNet backbone – meaning that the number of organizations that would have to coordinate to upgrade to an entirely new protocol is also exponentially greater. An added wrinkle: the two competing proposals, soBGP and sBGP, came with two distinctly different sets of tradeoffs, which naturally meant two different camps of researchers and vendors were advocating past each other for their preferred protocol, instead of collaborating on a way to move forward to fix BGP-4’s problems. This led to an inevitable stalemate, as network operators waited for a clear victor to emerge between the two camps, preferring to incur the costs associated with the occasional routing blip than to go through the hassles of picking the potentially losing side of a standards war. Knieps Gunter, in his textbook Network Economics refers to this behavior as ‘the penguin effect’:
…penguins hovering on the edge of the ice attempt to let the others jump first into the water; although they are all hungry, each fears that there might be a carnivorous fish in the water (cf. Farrell & Saloner, 1987, pp. 13f.). The consequence may be a persistence of the status quo, even though all would prefer a switch to the new technology.
— Gunter, Network Economics [6]
Taking a step back from this particular issue and examining the history of Internet routing, we can see the shadows of what Internet engineers refer to as ossification — the tendency for protocols to become ‘set in stone’ as they become more widely adopted. The term “ossification” in reference to network protocols was first introduced in academic literature as early as 2005; researchers were grappling with the apparent inability to upgrade the TCP protocol, not because of the difficulties inherent in coordinating an upgrade between TCP speakers but because of the problems posed by “middleboxes”— in-band network devices, such as load balancers and firewalls, capable of intercepting, filtering, and modifying traffic in between TCP speakers. These middleboxes made assumptions about the behavior of the TCP protocol and mediated connections between TCP speakers, contributing to the process of ossification:
The Internet architecture assumes a common internetwork layer is sufficient substrate for building a global network, considering communication on a host-to-host basis [1]. Time has shown this to be only part of the story, however. Not only does the network need to know the hosts that are party to a communication, it has become clear that knowledge of higher layers are also required. This is visible in the plethora of middleboxes that have become part of the network, inspecting both transport layer headers and higher layer protocol data to ‘help’ endpoints and perform various policy enforcement functions. The tussle space [2] between application and service provider needs is not effectively realised by the IP layer of the Internet. Deep packet inspection and other tools are used to extract the information needed to enforce provider goals. This has resulted in widespread ossification of the network. Knowledge of the transport layer, and higher layer protocols, is encoded in middleboxes throughout the Internet. It is steadily more difficult to change the core network protocols, because doing so requires changes to increasingly many middleboxes and policies, in addition to the endpoints of the communication.
— McQuistin And Perkins, Reinterpreting The Transport Protocol Stack To Embrace Ossification [7]
In the earliest stages of the protocol adoption lifecycle, new features and functionality draw in new users, which in turn have new demands for new features and new functionality, leading to a virtuous cycle of constant experimentation until a tipping point is reached where stability and compatibility are preferable — where the demands of new users hoping for new features lose out to the demand of the existing user base for the continuation of the status quo, and where the existing user base is either unwilling or unable to coordinate the deployment of updates to the protocol. In the case of Internet routing, there was near universal agreement by users of the existing routing protocols on the need for a new topology and less reliance on backbone providers that served to drive the migration from EGP to BGP; similarly, there was a shared need for more efficient use of address space that lead the charge from BGP-1 up to BGP-4; and finally, the abrupt cessation in routing protocol changes was due to the explosive growth in BGP-4 speakers — an exponential increase in the existing user base for Internet scale routing — combined with a relatively weak demand to solve the problems of BGP-4, as most network operators were either ignorant of the consequences potential security issues inherent in BGP-4, unsure of which alternative protocol to choose if they wished to so much as contemplate upgrading, or simply content to live with the occasional disruption caused by inadvertent BGP route hijack snafus.
In his 2014 paper ‘Why The Internet Only Just Works’, Mark Handley describes the problem succinctly:
BGP is probably the most critical piece of Internet infrastructure — it holds the entire network together. The problem with BGP is that it mostly works. If it actually failed, then it would need to be replaced. Network operators actually think in terms of BGP — it is very hard for them to consider alternative ways of doing routing. If BGP is ever replaced, it seems at this point like it will not happen until BGP has been shown to fail. Such a moment would not be the ideal time to try and design alternatives.
– Handley, ‘Why The Internet Only Just Works’, [8]
Takeaways:
- Since its inception, the Internet has had three major transitions between exterior routing protocols, from GGP to EGP to BGP. Each protocol transition was correlated – though not necessarily caused by — changes to the underlying physical network topology. Routing protocols are inextricably linked to the underlying physical structure of the Internet, as underlying the virtual connections between routing endpoint gateways are physical communications links in the real world.
- The transition from a private research network to one where for-profit entities were free to interconnect was a massive undertaking. Most popular Internet histories trace the origin of the Internet with the ARPANET and then skip ahead to the days of the World Wide Web, skipping over the significant role that the NSFNet epoch and a handful of contractors at Merit played in shaping our present-day Internet. These changes were enabled by a cycle of iterating and updating the protocols responsible for routing packets; new commercial relationships and business models were made possible by the conscious design and implementation of new protocols.
- Over time protocols can become more difficult to update and upgrade due to ‘ossification’, becoming rigid and less flexible over time, when there is more aggregate benefit to network operators in maintaining the status quo than there is potential benefit to be earned by drawing in new network operators with new features and functionality. More users make coordinating upgrades more difficult, and with differing incentive structures, the path to coordinating a mass upgrade to a protocol becomes less straightforward.
Chapter 6
Asleep, Tired, Or It Malingers: How Protocols Wither and Die
So far, the protocols we’ve examined can be sorted into two buckets: those that were or are part of the core protocols that make up the Internet (NCP, TCP, DNS, BGP, IPv4 and the like), and those that weren’t able to reach escape velocity beyond the community of researchers or vendors advocating for them (e.g., CLNP, SNA, DECNet, and most of the OSI suite of protocols). As we progress up from the foundational protocols to higher layers, we’ll examine a new kind of failure mode: protocols whose adoption has plateaued or that have “failed to thrive”. Each of the protocols we will examine in this chapter — NNTP, the protocol that powered Usenet; Gopher, an early competitor to HTTP; and SMTP, the protocol that to this day is used for e-mail delivery — solved real-world problems when first conceived, saw significant adoption in the early Internet, and each failed in their own unique way, whether by fading into obscurity or by living on as unloved zombies.
NNTP and the Strange Death of Usenet
While the protocol we will examine first, NNTP, was used to power Usenet, Usenet itself predated not only the protocol but even TCP. First conceived in 1979 and launched in 1980, Usenet was a sort of distributed message board, roughly analogous to the web forums of the early 2000s.
Usenet is a set of protocols for generating, storing and retrieving news “articles” (which resemble Internet mail messages) and for exchanging them among a readership which is potentially widely distributed. These protocols most commonly use a flooding algorithm which propagates copies throughout a network of participating servers. Whenever a message reaches a server, that server forwards the message to all its network neighbors that haven’t yet seen the article. Only one copy of a message is stored per server, and each server makes it available on demand to the (typically local) readers able to access that server. The collection of Usenet servers has thus a certain peer-to-peer character in that they share resources by exchanging them, the granularity of exchange however is on a different scale than a modern peer-to-peer system and this characteristic excludes the actual users of the system who connect to the news servers with a typical client-server application, much like an email reader.
— Wikipedia, https://en.wikipedia.org/wiki/Usenet#Binary_content [1]
If today’s modern social media platforms were organized by users subscribing to topics instead of connecting to other users on a social graph, they might look a bit like Usenet did. As Usenet’s design predated the DNS, it had its own peculiar naming convention: a top-level hierarchy of groups (for example, alt, rec, sci and comp) followed by one or more subjects. For example, the rec.sports.football.eagles Usenet group would be for discussions about the Eagles football team, while rec.sports.basketball would cover all topics related to the sport of basketball. Each leaf of the hierarchy after the top level would provide more granularity on a given topic. Users can subscribe to a newsgroup, and rely on their client software to keep track of the articles they have read. Most articles are responses to other articles, and the set of reply articles tracing back to one non-reply parent article is called a thread. Usenet articles are transmitted in a similar manner to Internet e-mail messages, in a format first specified in RFC 850, but unlike email they can be read by any user whose news server carries the group where the message was posted. Usenet does not have a central governing body. Each news administrator is independent, has complete control over their server, and can choose which newsgroups to offer access to. Nobody can dictate how a news administrator should configure their server. There is a de facto list — the “Big-8”— of newsgroups that serves as a jumping-off point for news administrators who wish to offer a default list of newsgroups to users. It’s still true that any news administrator can choose to carry only the groups they want; the management board of the Big-8 simply creates a list of recommended groups and hopes that most news administrators will agree with their suggestions.
It’s hard to describe how central Usenet was to the early experience of “being online” to present-day Internet users accustomed to only interacting with the World Wide Web or “apps” on their mobile phone; with the Web and Gopher not even in their infancy, it was the primary venue for discussion, conversations and interactions among groups of Internet users. There were careers built upon and (literal) religions formed on top of it, and yet, despite a first-mover advantage of more than a decade over HTTP, NNTP usage dramatically declined and fell out of favor over the years. In 2008, when PCMag published an article titled ‘R.I.P Usenet: 1980-2008’ [2], the response from network operators was a collective shrug of the shoulders. How did a protocol with so many users and so much demand, that was part of the core daily experience of almost every early Internet user, wither away and almost completely die off? There is no one definitive answer to this question; complex systems have complex ends. That said, there are two commonly cited contributing factors that we will take the time to examine: one social and one economical.
First, the economical: the free rider problem in economics describes a situation in which an individual or group enjoys the benefits of a public good but does not contribute to its production or provision. Public goods are goods or services that are non-excludable, meaning that once they are provided, it is difficult or impossible to prevent others from benefiting from them, and non-rival, meaning that one person’s consumption of the good does not diminish its availability for others, such as, for example, Usenet content.
The free rider problem arises when individuals or groups have an incentive to “free ride” on the contributions of others, rather than contributing themselves. This is because they can enjoy the benefits of the public good without paying the cost of producing it, which in turn leads to underproduction. For example, if a group of individuals decides to clean up a public space, there is a risk that some people will choose not to contribute to the effort, knowing that they will still be able to enjoy the clean space. If too many people make this same decision, not enough of the space will actually be cleaned up for anyone to enjoy it, which is harmful to everyone, regardless of whether they contributed to the clean-up effort or not.
The Usenet user experience was initially built around threads of conversations, but it didn’t take long for users to realize that a medium used to share and broadcast messages could be used to share and broadcast any sort of binary data, simply by encoding that data as if it were text. It wasn’t long after Usenet was first made available to commercial ISP users that new newsgroups under the alt.binaries. umbrella began to proliferate, with threads of dozens of messages representing pirated software and pornographic movies, encoded as text and split into multiple posts.
It was tremendously inefficient to share binary data in this way, but those inefficiencies didn’t impact the users; since the costs weren’t born by the uploaders or downloaders, but instead the newsgroup server operators — typically Internet Service Providers offering Usenet access to their users —alt.binaries traffic proliferated until the free riders had exhausted the resources made available to them. As the amount of storage required to maintain access to the alt.binaries newsgroups increased over the years, many larger ISPs decided that the Usenet juice wasn’t worth the storage squeeze, and stopped offering access to Usenet entirely. Smaller ISPs, desperate not to lose customers, found themselves racing to purchase high-end storage appliances to store all the alt.binaries data in a desperate attempt to keep the free rides going.
This economic problem was felt primarily by service providers offering access to Usenet newsgroups; the social problem we’ll examine was a seismic shift felt by anyone who contributed to or consumed content from Usenet newsgroups. Decades later, this shift is referred to by those who experienced it at the time with two simple words: “Eternal September.”
In 1993, dial-up information services like Delphi and AOL (America Online) began providing their users with access to Usenet newsgroups. Prior to this, access to Usenet was typically limited to people affiliated with universities and other academic institutions, and the community was relatively small and tightly-knit.
With the sudden flood of new users from these services, most of whom were not familiar with the netiquette and customs of Usenet, there was a massive increase in spam, off-topic posts, and other forms of disruptive behavior. The existing community, which had previously been able to self-regulate and maintain a certain level of civility and coherence, found itself overwhelmed and unable to cope with the changes ushered in by the stampede.
The term “Eternal September” was coined by a Usenet user who compared the flood of new users to the annual September influx of new university students discovering and connecting to Usenet. However, in this case, the influx of new users was “eternal” because the new users never stopped coming.
The negative impact of Eternal September on Usenet is difficult to overstate. Many longtime users of the network became frustrated and disillusioned by the influx of new AOL and Delphi users and their disruptive behavior, and some chose to abandon Usenet entirely. The pre-existing culture of Usenet — at times respectful and productive, at times irreverent and bizarre — was irrevocably changed, and the newsgroup conversations never fully recovered their original character. What was once a core Internet protocol eventually became, effectively, just another website. First, users’ demand for access to Usenet began to decline after Eternal September, and as less users demanded access to Usenet, ISPs took the opportunity to relieve themselves of the burden of hosting and storing all of that alt.binaries data. As less conversations took place on Usenet, centralized websites — first Deja.News, then Google Groups — offered access to Usenet archives and the ability to post new messages to Usenet through an HTTP gateway without making use of the NNTP protocol, while “web forums”— software that replicated the experience of posting to a newsgroup, except served via HTTP, and accessible via a web browser — obviated the need for a separate NNTP protocol to offer largely the same experience.
Usenet still exists today; there are many who still consider it useful; discussions happen there that simply don’t occur anywhere else. But the one-two punch of alt.binaries congestion and Eternal September marked the beginning of Usenet’s slide into irrelevance for the average Internet user, even with die-hard users still maintaining and contributing to newsgroups to this day, 16 years after PCMag had written Usenet’s obituary.
Gopher and The Myth of The First Mover Advantage
In the early 1990s, before the World Wide Web came online, when dial-up information services like Prodigy and AOL were ascendant, a team of programmers at the University of Minnesota started working on a project to launch a Campus-Wide Information System (CWIS) at the behest of university administrators. The hope was that the CWIS would provide access to university information and email for faculty, and students; the unfortunate reality was that almost immediately after the project was announced, it became mired in a morass of academic red tape and bitter factionalism that one of the programmers would later describe as “a classic design-by-committee monstrosity.”
In the wake of the CWIS project’s failure to launch, the same team of programmers spun up a skunkworks project outside of administrative control they jokingly called ‘Gopher’. The name was both a play on ‘Go-fer’, as in an assistant to fetch things (in this case, information), as well as a reference to the University of Minnesota’s mascot. The programmers developed the code for Gopher without official approval (and under the radar of the different academic factions that had hamstrung the CWIS project). Using readily available personal computers instead of the university mainframe (which they would have been required to request access to), they had a working prototype done in a matter of months.
Gopher was a hypertext-oriented client-server protocol similar to HTTP, albeit with an entirely different design sensibility. An emphasis was placed on presenting hierarchical menus to users for accessing information, perhaps as an unconscious (or deliberate) attempt at mimicking the user interface of the existing information services like AOL and Prodigy that users were accustomed to.
The initial proof of concept of the Gopher software worked beautifully, but was initially rejected by the University of Minnesota because it didn’t suit their vision for the initial CWIS project. In response, the Gopher team released the Gopher server and client code for free on the university’s FTP site as well as a demo available over a Telnet session, the two most popular ways to share software with students and faculty at the time. The hope was that making it available for as widely as possible would help Gopher avoid the fate of the scuttled CWIS.
Gopher programmer Robert Alberti later recalled:
By providing a public login over Telnet individuals who were curious about Gopher could immediately try out the UNIX client, and this demonstration was often sufficient to persuade them to download and install the Gopher software via Xmodem or FTP transfer. Distribution was quite rapid once people were aware Gopher existed. By facilitating its own distribution, Gopher may have been the Internet’s first “viral application.”
The Gopher team members were not shy about explaining to those who wrote with suggestions for improvements that we were prohibited from working on Gopher by the campus CWIS group, and we happily provided the names of University administrators to whom comments might be addressed. Senior University personnel began receiving calls and e-mails from other institutions complimenting them on Gopher and asking about the Gopher development prohibition. After a couple of months, Gopher’s popularity and undeniable utility became an embarrassment to the CWIS committee, and we were able to formally resume the work we had continued doing on our own time.
— Alberti, B. Internet Gopher: A Bridge To The Web [3] Retrieved from https://archive.org/details/internet_gopher_bridge_to_the_web
With the users raring to go and the wind at their backs, the Gopher team was off to the races… for approximately eighteen months, from the initial release in 1992 to the summer of 1993. It’s clear now, thirty years later, that Gopher never stood a chance as a competitor to HTTP, but why? How did a protocol with such a clear first-mover advantage and an enthusiastic base of early adopters collapse so completely?
There are two commonly cited reasons for Gopher’s failure. The first is that the administration of University of Minnesota, feeling that they had been forced to support Gopher’s development despite never having approved of it in the first place, made the decision that Gopher needed to “pay for itself” and the project needed to earn enough revenue to cover the salaries of the software developers working on it. As a result, in February of 1993 at the GopherCon convention, the Gopher team announced that for-profit users would be charged a licensing fee.
The GopherCon announcement was not received with enthusiasm:
Many users felt betrayed. In the open-source computing spirit of the day, they had contributed code to Gopher, helping the team keep up with the times. Now they were being asked to pony up.
The reaction deflated the team. As hard as they were expected to work on Gopher, they were never relieved of other duties, including answering the University’s help line. “We never got additional funding, we were going broke,” Alberti says. “We had the whole internet yelling at us, when are you going to update your software, when are you going to put images on Gopher pages, and make this smoother and better? And we’re like: ‘We’re six guys!’”
Asking for a contribution seemed reasonable. When it backfired, the team posted a defensive letter to Gopher users in March 1993 [stating in part]: “… There has been a lot of hysteria, misinformation, and rumor floating around. … In a time where we are having budgets slashed, it is impossible to justify continued (increasing) resources being allocated to Gopher development unless some good things result for the University of Minnesota. This is a fact of life. … Before you go off and flame once more, ask yourself if you want to get YOUR particular server going with as little fuss and expense as possible … or if you just want to stir up the soup.”
“That socially killed Gopher,” Alberti says of the licensing fiasco.
– Gihring, T. The Rise And Fall Of The Gopher Protocol [4] https://www.minnpost.com/business/2016/08/rise-and-fall-gopher-protocol/
The second reason, though perhaps less commonly cited, was cited by the programming lead of the Gopher project, Mark McCahill, in an interview years later: that the new Mosaic HTTP web browser, by providing a substandard Gopher client, curtailed user enthusiasm for the Gopher protocol:
What killed the protocol off was when people stopped using Gopher clients specifically and started using Mosaic. Mosaic had a shitty implementation of Gopher, and they had zero interest in improving their implementation of Gopher, or supporting any of the new stuff. But once the bulk of the people were running Mosaic 9 we could think of new things to do with the protocol and improve the client all we wanted. It didn’t matter. I learned a good lesson. Control the client.
– Frana, P. Interview with Mark McCahill [5]
As a project lead, McCahill would be the best placed to make such a judgment; I personally find it a more convincing argument than the idea that the licensing fee, if only because it was the announcement of a licensing fee and not the imposition of an actual licensing fee itself, was the cause of the rapid decline in Gopher’s popularity.
As I read through the oral histories and interviews with the Gopher team, I found myself considering a third explanation for HTTP’s “victory” over Gopher. Namely — was there really even a competition between Gopher and HTTP in the first place? The Gopher team may have seen the World Wide Web as its competition, but that doesn’t necessarily mean it makes sense to model them as ‘competitors’.
What would have happened if all of the human resources (in development, content creation and user traffic) that were thrown at the Web in the early 1990s had instead been thrown at Gopher? Or stated in the terms of this paper: What would Gopher be like today if it had captured the mind share that was instead captured by the Web?
A meaningful answer to this question cannot simply be derived from looking at the current state of technology and then drawing a rigid line of causation into the past. Using the current state of Gopherspace to argue for its technologically inevitable demise is tantamount to arguing for the obvious superiority of VHS over Beta by pointing out how much more selection of cassettes and players are currently available for the former format than the latter. There is clearly much more going on in that case than picture quality and storage capacity. Broad social trends, roughly what an economist would call “network externalities,” play a significant part in the adoption of one technology over another. I contend that, much as one brand of blue jeans does not win over more consumers than another brand simply based on the comfort and durability of its material, the Web did not overshadow Gopher simply based on the technical qualities of HTTP as a protocol.
— Lee, C. Where Have All The Gophers Gone? [6] https://ils.unc.edu/callee/gopherpaper.htm
Christopher Lee’s paper, ‘Where Have All The Gophers Gone’, is a deep and nuanced investigation into the root causes of Gopher’s failure. His conclusion was that, ultimately, Gopher’s downfall was due to the inability to capture mindshare beyond its initial enthusiastic user base — with bad publicity from the licensing fiasco, institutional influences, failure to follow the open systems model of development, and social inertia all serving as contributing factors to this inability to gain traction. Put more plainly: it is inaccurate to say that Gopher somehow lost the battle for hearts and minds against HTTP. Instead, the very nature of what Gopher was made it impossible to enter that battle in the first place. As an academic skunkworks project crafted by a tight-knit group of developers at a single university, it was doomed to lose against HTTP, whose arrival began with an open call for collaboration with developers across the world.
The major issue was not the original configuration of the protocol but instead the fact that it did not become a test bed. Mind share moved from Gopher to the Web, leaving only a handful of developers at the University of Minnesota and a few other places to tweak Gopher to meet a rapidly increasing demand and level of expectations.
— Lee, C. Where Have All The Gophers Gone? [6] https://ils.unc.edu/callee/gopherpaper.htm
SMTP (Email): Too Old to Die Young
It feels strange to put SMTP on a list of failed protocols. It’s widely used and adopted, with no apparent competitors and no signs of slowing down any time soon. For decades now, every new university student and almost every new employee of a business of any reasonable size is issued a new email address, and a familiar part of the process of signing up for or logging in to almost any software application is being prompted for your email address.
The concept of ‘electronic mail’ existed well before SMTP was ever specified; as soon as multi-user computing systems were invented, organic methods for sending messages between users began popping up. The first formal(ish) specifications for email came as extensions to the File Transfer Protocol (FTP) in RFC 385 in August of 1972; it took another decade for the Simple Mail Transfer Protocol (SMTP) to be specified in RFC 788 and RFC 821.
SMTP’s initial specification was drafted in a different Internet era, one where academic researchers and enthusiasts communicated with each other without any concerns about surreptitious eavesdropping, and the idea of widespread commercial access to the Internet was patently absurd. No precautions were taken to guard against bad actors attempting to abuse the SMTP protocol for their own purposes. Enter the spoofers and the spammers.
Spoofing refers to the practice of forging the email sender’s address to make it appear as if the email originated from a different source than the actual sender. Spamming refers to the practice of sending unsolicited emails in bulk to a large number of recipients, often for advertising or other illicit purposes. Spoofing and spamming tend to go hand-in-hand. Bulk emailing services rarely want to be associated with a long-lasting identity for deliverability reasons (as sending even a single spam message from an email address will most likely quickly lead to that address being blocked by recipients).
With no means of guarding against spoofers or spammers specified in the initial protocol, SMTP mail server operators were forced to adopt means outside of the protocol. What was initially a minor nuisance became a major threat as the number of email users increased linearly over time, and the potential rewards for spamming and spoofing increased almost exponentially. Preventing email spoofing has been a decades long battle; while present-day email spoofing can be prevented using advanced measures like SPF, DKIM, and DMARC records (all DNS-based methods of “proving” that a domain is not associated with spoofing or spamming), it falls on each individual SMTP mail server operator as well as individual SMTP client operators – that is, anyone with an email client — to configure their software to check these records. Failing to do so will not prevent email servers from sending or receiving email, but it may lead to received emails being labeled as spam by higher level software. No changes have been made to the SMTP protocol to address these problems; instead the focus has been to build tools for senders to prove their legitimacy – by inserting data into the DNS — instead of for receivers to protect themselves.
Spam prevention was dealt with more bluntly in the early days of SMTP. When confronted with the first wave of commercial spammers in the 1990s, Internet Service Providers almost immediately began blocking access to SMTP servers run by residential users by filtering port 25, the TCP port associated with sending mail via the SMTP protocol. Mail servers that allowed open access to port 25 would find themselves on blacklists managed by private companies. Ostensibly, these blacklists were a public service meant to help identify spammers and protect users, but in practice, the implementation and adoption of these private blacklists as an anti-spam measure had unintended consequences for the average user experience:
One of the earliest blacklists was the Mail Abuse Prevention System (MAPS.)…MAPS’ purpose was to educate users to relay mail through an acknowledged ISP instead of running their own mail servers. Doing this would bring various advantages and disadvantages. ISPs can generally afford to monitor their systems more thoroughly in order to avoid viruses, hijackers, and other threats. Furthermore, it paved the way for effectively exploiting policies like SPF, which relies upon SMTP authentication in order to block email address abuse. In addition, ISP email relays are incompatible with fine-grained IP address blocking: if they relay spam and get blocked, it impacts all of the ISP’s customers.
The first victims of collateral damage from blacklisting efforts were those whose mail servers were blocked through no fault of their own. This happened when overzealous or lazy blacklist operators added entire blocks of IP addresses to their lists, rather than only targeting the specific offending addresses. Some blacklist admins argued that the lists should err on the side of caution to prevent these problems, while others believed that this would put extra pressure on the open relay administrators to cease operations….
Today’s email servers will claim to accept incoming mail but will delete it as soon as it is received, a practice referred to as blackholing. The email servers operated by tech giants permanently blacklist wide swaths of IP addresses and delete their emails without notice. Their back end infrastructure runs algorithms across huge datasets so that most spam doesn’t appear on your inbox.
– Lopp, J. The Death Of Decentralized Email [7] https://blog.lopp.net/death-of-decentralized-email/
While in theory, SMTP is an open protocol and email deliverability is guaranteed end-to-end to anyone capable of running a mail server that speaks SMTP; in practice, a mail server that finds itself on a blacklist that is used by Google’s Gmail or Microsoft’s Outlook mail servers is effectively useless, even if it’s able to deliver email unimpeded to the rest of the Internet. The large user base of these centralized SMTP service providers allows them to serve as de facto arbiters of email deliverability. A choice between reaching 80% of users 90% of the time or 100% of users 100% of the time for the same price is no choice at all; users will naturally flock to the larger email providers to ensure they are able to deliver email without worrying about their messages silently being dropped. As more users transition to larger email providers, smaller providers become less relevant, and a de facto oligopoly emerges.
SMTP’s failure state is perhaps the most personal for me; I’ve run my own mail server for nearly three decades, starting with begging my ISP in the 1990s to allow access from the outside world to port 25 for my statically assigned IP address, working my way up to setting up a mail server at my own ISP, setting up DKIM and SPF records, only to finally find myself silently blacklisted by Google’s Gmail servers, sporadically, with no error messages returned and no support desk to complain to. As of the publishing date of this text, I’ve migrated all of my domains to a centralized provider.
Takeaways:
- NNTP (Usenet): NNTP was omnipresent in the days before the World Wide Web became popular but was abandoned by many ISPs as well as early users. There were two reasons for this, the first being the economic problem of free riders, where users could enjoy the benefits of storing data on Usenet without paying any costs to contribute to its operation. The second was the social problem caused by “Eternal September,” the influx of users of online information services such as AOL and Delphi in the early 1990s, whose lack of understanding of Usenet’s social norms disrupted the existing community and its culture, leading to frustration and disillusionment among users who had played a part in building Usenet culture before their arrival.
- Gopher: Gopher, a hypertext-oriented client-server protocol, “lost” mindshare and users to HTTP despite having a first-mover advantage and early adopters. One commonly cited reason for Gopher’s failure was the decision by the University of Minnesota, which supported Gopher’s development, to charge licensing fees for for-profit users. This move led to a backlash from the user community, who felt betrayed and were not willing to pay for a previously free and open-source protocol. The lack of additional funding and resources also hindered Gopher’s development, as the small team behind it struggled to meet user demands and expectations. Finally, the stark cultural disparities in the software development lifecycle as managed by a group of engineers used to writing software for university administrators, hindered by budget constraints and bureaucratic red tape, versus the HTTP project’s focus on dialogue and collaboration with other developers around the world, made contributing to the Gopher project less appealing in comparison.
- SMTP: SMTP, the protocol used for email delivery, has not declined in usage in the same way as NNTP and Gopher. Indeed, most Internet users interact with the SMTP protocol on a regular basis. However, since the problems of authenticating identity and preventing spam have not been addressed within the protocol, third-party vendors have filled the gap by providing solutions outside of the protocol. Users have migrated their email inboxes en masse to the vendors offering the best solutions to their problems (e.g. Gmail), and this massive crowd of users has in turn given the vendors leverage to filter and exclude connections from other smaller SMTP servers and corral their users into a de facto “walled garden”. In theory, anyone is free today to spin up an SMTP server using open source software and their choice of Internet Service Provider; in practice, the ability to guarantee the deliverability of email for end users and to avoid the endless headaches of sent emails being silently dropped or filtered are limited to a handful of large vendors — a de facto email oligopoly.
Chapter 7
The Street Finds Its Own Use For Things: The World Wide Web From CERN to Snowden
Writing a single chapter on the Hypertext Transfer Protocol (hereafter, HTTP) is a daunting proposition. An exploration of the protocol that changed the world and our understanding of how we relate to each other as a species could easily fill its own separate text. As the focus of this book is on the history of Internet protocols, we’ll narrow our scope to two points in particular: the first being the ‘founding myth’ of the free and open Web as contrasted with the reality of the HTTP’s client-server design, creating a new paradigm of restrictive control; the second being the transition from HTTP to HTTPS in the wake of the Snowden revelations in 2013.
A New Hope
HTTP was first proposed as a literal science experiment ‘concerning the management of general information about accelerators and experiments at CERN’ by physicist Tim Berners-Lee. Berners-Lee wanted to leverage the concept of hypertext to deal with the problem of keeping track of information related to large projects in a distributed organization where employees were constantly leaving:
A problem, however, is the high turnover of people. When two years is a typical length of stay, information is constantly being lost. The introduction of the new people demands a fair amount of their time and that of others before they have any idea of what goes on. The technical details of past projects are sometimes lost forever, or only recovered after a detective investigation in an emergency. Often, the information has been recorded, it just cannot be found. If a CERN experiment were a static once-only development, all the information could be written in a big book. As it is, CERN is constantly changing as new ideas are produced, as new technology becomes available, and in order to get around unforeseen technical problems. When a change is necessary, it normally affects only a small part of the organisation. A local reason arises for changing a part of the experiment or detector. At this point, one has to dig around to find out what other parts and people will be affected. Keeping a book up to date becomes impractical, and the structure of the book needs to be constantly revised.
— Berners-Lee, T. Information Management: A Proposal CERN, March 1989 http://cds.cern.ch/record/369245/files/dd-89-001.pdf [1]
The information management system that Berners-Lee envisioned was to be powered by hypertext — articles linked to other articles — a metaphorical ‘web’ of articles all connected to each other instead of a hierarchically indexed system.
The first version of HTTP, 0.9, was released in 1991 as a software artifact first — a web browser and a server — and the specifications were written later describing the behavior of the software. The code implemented in this early version of the software is sometimes referred to as ‘the one-line protocol’ due to its simplicity, and was focused on implementing just a subset of the features that had been proposed, with an emphasis on getting a proof-of-concept working. Berners-Lee released the software by posting an announcement on the Usenet newsgroup alt.hypertext, asking anyone who might be interested to offer their feedback and inviting them to collaborate.
This call to action laid the groundwork for the ‘founding myth’ of what came to be known as the World Wide Web. According to this myth, the Web was open, free for all, dynamic; in comparison, other means of distributing information online — whether protocols like Gopher or online information services like AOL and Compuserve — were closed, limited, and stagnant. As the years passed and HTTP and the World Wide Web exploded in popularity, Tim Berners-Lee was lionized as a visionary, a magnanimous creator gifting his invention to the world, eschewing personal benefit for the greater good, even as other entrepreneurs in the early days of the Web saw great commercial success.
Myths are the stories that human beings tell themselves to help them explain their perception of the world around them. While it’s true that the underlying protocol that powered the Web was open and free when compared to its competitors, and the initial call to action was an open invitation to anyone interested in collaborating, in practice the most popular early Web browser was closed source, the most popular sites were all commercial enterprises, and Berners-Lee himself explored launching his own startup to capitalize on the rise of the Web. These fundamental truths are of little import; the myth was easier to digest than reality, and so it was able to displace reality in the popular imagination.
Notably, this myth of the Web’s origins contrasts sharply with the military narrative of the Internet’s origins, thus reinforcing the idea of a re-appropriation of the technology by research centres and academics. Hence, the Web’s origins at CERN tally with its supposed decentralized and egalitarian character; it was born in an open environment thanks to the effort of a collaborative and horizontal organization, and a mind of genius free to experiment within it. This social sphere represents the principles of sharing and unifying knowledge, expertise and skills for the progress of science and human wealth…
As Marcel Mauss has shown in his classic essay (Mauss 1990), every gift bears its donor’s identity. In this regard, the sacrifice of Berners-Lee and the gift of the Web to society is a ‘personal renunciation that nourishes social forces’ (Hubert and Mauss 1964: 102). It is not just a technological transfer; it is also a transfer of meanings and values. The sacrifice is thus an act that reinforces the characterization and the identity of the hero as much as the intrinsic value of his invention. More broadly, the refusal to receive money or any other advantage from his invention, which coincides with the sacrifice of the hero, does not only contribute to the hagiography of Berners-Lee, but also strengthens the analogy between the Web’s inventor and the Web itself, which is also portrayed as a neutral space in terms of economic interests and power. In this way, the sacrifice of the hero makes the Web a milestone, a final step in the dominant narrative of Internet history which depicts the final evolution of the ‘network of networks’ as a horizontal space for information exchange and peer-to-peer production — a sacred gift to society.
— Bory, The Internet Myth [2]
In contrast to the founding myth, the early days of the 21st century saw the rise of new corporate players — Amazon, Google, Paypal, and eBay being the most recognizable to us today — that were able to amass significant power due to first-mover advantages. There were relatively few websites competing for user mindshare compared to the present day Internet, and those that were able to attract and monetize users reinvested their profits in improving their user experience, attracting more users, bringing them more revenue, and repeating the cycle. Network scientists like Albert-László Barabási have described in detail the concept of ‘preferential attachment’— the tendency for new users joining a network to choose to connect to popular nodes that already have large numbers of existing users they can interact with. This tendency can lead to what network economists such as Varian and Shapiro describe as positive feedback loops — more vendors at your local farmers market will lead to more shoppers visiting the market, which in turn will attract more vendors — and their obvious corollary, negative feedback loops, as for example when too many shoppers leads to a crowded marketplace where buyers are stuck waiting in queues and vendors are unable to complete sales faster than new buyers arrive, eventually driving more and more shoppers away from the market.
The client-server model of HTTP communication meant that users of Web browsers — i.e. the clients in the client-server model – would have to connect to computers – servers — that were managed directly by the commercial enterprises they wanted to interact with. As succinctly as possible users run web browsers, which are used to connect to websites, and those websites, in turn, are hosted on web servers. Compared to other protocols such as NNTP (Usenet newsgroups), chat rooms (IRC), or e-mail (SMTP), the Web offered a different model of user interaction without any intermediaries sitting between the parties communicating. The experience of ‘surfing’ the Web and hopping from site to site was qualitatively different. The ‘switching costs’ incurred by users of surfing from site to site weren’t just low, they were negligible by design. The Web was meant to be browsed, with sites linking to other sites linking to other sites in a literal web. The potential upside for demand-side-driven positive feedback loops was far greater than the potential downside of negative feedback loops. Client-server protocols remove the restrictions of geography, space, and time from “brick-and-mortar” buyer-seller interactions in the physical world; large pools of clients in different physical locations are all able to queue and submit requests simultaneously to servers in ways that would be infeasible in the crowded physical marketplace described earlier.
The feedback-driven explosive growth of the early entrants to the Web has turned them, in the eyes of sovereign States, to become what Nicola Tusikov refers to as ‘macro-intermediaries’. Websites with extremely large pools of users are in a unique position to exert control over policies that are traditionally the domain of state actors, from regulating commerce to curtailing speech. While it may be true in theory that anyone on the Web is free to communicate and transact with anyone else, in practice, the logic of preferential attachment means that a few small players are able to regulate and intermediate large swaths of user traffic. If a vendor is allowed to transact with anyone they choose on the Web and is allowed to list their goods and services on any search engine they please, but is barred explicitly from listing them on Amazon, eBay or Google, they are effectively invisible to a large portion of the Internet, banished to a sort of ‘digital ghetto’ and only able to interact with a digital underclass. HTTP clients are geographically distributed, meaning that users around the world browsing sites operated by companies located in the United States (e.g., Google or Paypal) or China (Alibaba and Tencent) are compelled to comply with laws enforced in those jurisdictions as interpreted by these companies, with the companies themselves serving as intermediaries. A U.S. citizen viewing a Chinese website for information on the Tiananmen Square protests of 1989, for example, will have the same experience of state censorship as a Chinese citizen residing in mainland China, all from the comfort of their own home in the United States.
Large intermediaries like Google, eBay, and PayPal can be thought of as “macrointermediaries” owing to their global platforms, significant market share, and sophisticated enforcement capacities that protect their systems and users from wrongdoing like fraud or spam. Macro intermediaries can set rules that govern hundreds of millions of people who use their services. They are in a powerful position to shape the provision of essential Internet services, such as search and payment processing, by virtue of their ability to monitor their platforms, remove unwanted content, and block suspicious transactions and behavior. Given their regulatory capacity, cooperative macrointermediaries can allow rights holders to police mass populations globally in ways that were previously unattainable, technologically unfeasible, or prohibitively expensive…
More importantly, government officials in the United States and Europe forged the enforcement partnership between rights holders and macrointermediaries outside of traditional democratic, legislative processes. The result: prominent, mostly U.S. and European rights holders have the capacity and authority to crack down on the online trade in counterfeit goods and copyright-infringing content. Simply put, companies like Nike, Disney, and Apple have the explicit support of the U.S. government (and official support from the U.K. government and European Commission) to set and enforce rules globally on the Internet. Their rules, crafted in the United States and Europe and exported globally, can potentially affect hundreds of millions of people who, every day, use the services of the U.S. Internet companies and payment providers.
— Tusikov, Chokepoints [3]
The rise of these “macrointermediaries” has turned the initial narrative of the Web on its head, inverting it; the myth of a free and open digital frontier giving way to a new myth of a panopticon-like “control society” [4], to borrow Deleuze’s phrasing, one where users are obliged to submit to the rules agreed upon in secret between the macrointermediaries and the State, with few avenues for appeal or recourse if they find their transactions blocked by PayPal, their content demonetized by YouTube, their websites delisted by Google, or their accounts deleted by Twitter.
Control societies are characterized by a distributed system that consists of continuous visibility across time and space and the short-term regulating of movement and acts via modulation. As Deleuze states, in control societies, “The numerical language of control is made of codes that mark access to information, or reject it.” Borrowing from William Burroughs, Deleuze’s conception of control is not about influence or restraint, but rather a logic, or…a protocol, that gives the impression of freedom while one’s choices and behaviors are modulated and generatively shaped for them.
— Dixon-Román, E. Control Societies @ 30: Technopolitical Forces and Ontologies of Difference [5]
Just as the founding myth of the HTTP protocol is not quite an accurate portrayal of reality, so is this modern inversion. The Web is in fact just as open as it was in its inception; any Web user is free to operate their own website, and no one is compelled by force to make use of Facebook or Amazon. But, again, as the myth is easier to digest and comprehend than reality, it is able to displace reality in the collective unconscious. Reality is messy and resists narration. The lack of mythical heroes means there are no hero’s journeys and no narrative arc to imbue life with meaning; no mythical villains means there are no challenges to overcome, no battles to win, and thus no victories to celebrate. We may live in reality, but we need myths to feel alive. The myth of the web as techno-feudal dystopia is deeply satisfying and easily digestible; the reality that anyone can spin up their own web server and take a stab at being the next Instagram, Facebook, or TikTok if they’re willing to put in the work is almost infuriating in comparison, as it reminds us that we are unwilling to do that work. The Web we have today is quite literally the Web we deserve.
Panic at the Data Center
In 2013, the Internet engineering community was hit by a seismic shock in the form of what has come to be known as the Snowden revelations. A series of documents leaked by former NSA contractor Edward Snowden unveiled a vast surveillance infrastructure that had been operating in the shadows, undermining the sense of privacy and security on the World Wide Web. In particular, the National Security Agency, or N.S.A., the intelligence arm of the United States Department of Defense, was revealed through its PRISM and MUSCULAR programs to be intercepting and analyzing vast amounts of online communications, including emails, chat messages, and browsing histories, of both U.S. citizens and individuals worldwide. The impact of the Snowden leaks sent shockwaves that reverberated not only through the Internet engineering community but also across governments, civil liberties organizations, and the global public at large.
The Snowden disclosures exposed the extent to which government agencies had penetrated the very fabric of the Internet, challenging previously held assumptions users had about their privacy on the Web. Internet engineers and technologists found themselves grappling with the threat of surreptitious surveillance posed by a global adversary with nation-state-level resources.
The HTTP protocol, as initially specified, transmitted data between clients and servers in plain text without any encryption, making it vulnerable to passive surveillance by malicious actors in theory. In practice, the assumption was that traffic was safe; the level of sophistication and the amount of resources required to covertly execute a so-called “Man-in-the-Middle” attack across network boundaries were outside of the reach of common criminals, and who else would have the incentive to monitor and intercept HTTP traffic? HTTPS, the ‘secure’ version of HTTP that makes use of Transport Layer Security (TLS) certificates to authenticate and encrypt traffic between clients and servers, was readily available but not widely deployed, despite the origins of TLS stretching as far back as 1986 (ironically, or perhaps presciently, with the NSA being involved in the design and specification process) and a first working version of HTTP over SSL being shipped by Netscape in early 1995.
The ‘best current practice’ that most engineers were familiar with at the time of the Snowden revelations was to implement HTTPS when sensitive information such as user credentials and passwords were being exchanged, and only at the edges of the network. This was partly due to concerns about cost: certificates to secure names are sold by certificate authorities that browsers trust implicitly, and these trusted certificate authorities serve as a de facto cartel of sorts. In 2013, it was not uncommon to pay a few hundred dollars for a wildcard SSL certificate that could be used to secure all the records tied to a single domain name, and this was a relative bargain compared to the early days of the Web, where the difference between an HTTP and HTTPS server license from Netscape was several thousand dollars. There were also concerns about operational overhead: every SSL certificate is a new resource that needs to be managed, protected, and renewed in what was at the time a complicated and error-prone manual process. Finally, there were concerns about resource usage: HTTPS connections require more resources in terms of processing time and bandwidth for servers when compared to HTTP, and while that resource usage may be relatively small for each connection, over hundreds of thousands of connections, the theoretical numbers can add up.
All of these — in retrospect, minor — points of friction contributed to the collective common wisdom amongst engineers that using SSL to secure HTTP was only necessary when secure credentials were being transmitted between network boundaries and a hassle that could be avoided otherwise. The assumption was that plain text HTTP was acceptable for communications within networks (e.g., from Google’s web servers to their database servers in the same data center) and non-sensitive traffic from the outside world (e.g., users browsing the Google Search Page).
The Snowden leaks revealed that the NSA and Britain’s GCHQ had taken advantage of this assumption by secretly infiltrating the networks of large communications service providers like Google and Yahoo, passively intercepting the plain-text HTTP traffic flows therein, monitoring and recording the data in bulk, and copying it to their own data centers for further analysis.
An analysis of the legality of these operations is outside of the scope of this text; what is of interest to us is that the NSA operations were perfectly acceptable from the point of view of the HTTP protocol. It was no secret amongst engineers that HTTP traffic was unencrypted; the prevailing belief was that the bulk of it was useless to attackers who were primarily motivated by profit and operating with limited resources. Google and Yahoo themselves don’t log and store all of their HTTP traffic flows but only sample a small portion of them for troubleshooting purposes. Passively capturing enough HTTP data to be useful for any sort of large-scale analysis would require enormous resources, with entire data centers needed. The probability of an attacker possessing such resources being interested in deploying them seemed remote at best until it was proved that it wasn’t.
The response from the engineers that managed the networks in question was immediate, furious, and largely unprintable. (‘F— these guys’ is a direct quote from a Google engineer.) The Snowden revelations kicked off initiatives internally at both Google and Yahoo to migrate all HTTP traffic to HTTPS, as well as a broader movement to get TLS certificates deployed across the Web. Google led the charge by incorporating HTTPS as a crucial ranking signal in its search algorithm, promoting HTTPS-secured websites in search results ahead of HTTP-only sites. A non-profit certificate authority, Let’s Encrypt, came online in 2014, offered HTTPS certificates for free, removing any frictions related to certificate costs, and provided tools to streamline and automate the process of provisioning and renewing certificates. This substantially lowered the entry barriers for website owners eager to adopt HTTPS. Finally, in response to user concerns about privacy, major web browsers like Chrome and Firefox began explicitly flagging HTTP websites as “Not Secure”. This noticeable warning to users, combined with the threat of a Google search ranking downgrade, helped to encourage HTTP server operators to transition to HTTPS.
The effect of these combined efforts took the Web from an HTTPS adoption rate of less than 50% (and by some estimates as low as 30%) in 2013 to well over 95% as of 2023, according to an internal review by Google [6].
The switching costs of this endeavor seem almost insurmountable in retrospect — a slow, painstaking campaign requiring deliberate action by every web server operator and every browser vendor, one that was destined to take years, with a significant amount of work required of all interested parties — but the threat posed by a global adversary with unlimited resources at its disposal was simply too great to ignore. The only thing more astonishing than the collective willpower to resist this threat amongst users of the Web was their resounding success in doing so.
The migration from HTTP to HTTPS was the largest uncoordinated protocol transition in the history of the Internet. In contrast to the transition from NCP to TCP, the switch to HTTPS wasn’t managed by a single party setting a flag day for cutover; mercifully, in contrast with the switch to IPv6, HTTP traffic could be ‘upgraded’ to HTTPS without having to modify anything beyond the boundaries of the web server (as, crucially, web browsers were already able to view HTTPS content). Even with these tailwinds, it still took roughly a decade from the time of the Snowden revelations to achieve a 95% adoption rate for HTTPS. The success of the transition and the collective effort by web server operators, web browser developers, and certificate authorities to confront the threat posed by the NSA and GCHQ attacks were a turning point for the Web and an important milestone in the history of network protocols. ‘Who is allowed access to exactly what information’ is a foundational question asked of any political body; the transition from HTTP to HTTPS was the Web’s resounding, thunderous answer: ‘not you’.
Takeaways:
- Founding myths, being easier to digest and comprehend than messy realities, cement the underlying conceits of protocols in the minds of their users. Like all narratives, they are ripe for inversion with the passage of time. The founding myth of the web was that it was free, open, and available to all; the current myth is that it is a corporate oligarchy dominated by predatory companies like Facebook and Google, peddling disinformation and ads. These myths may not bear much resemblance to reality, but that doesn’t make them or the impact they have on the subjective experience of users of the Web any less ‘real’.
- The client-server paradigm of the HTTP protocol, combined with positive feedback loops, has led to the transformation of large Web server operators into ‘macro-intermediaries’, capable of enforcing and regulating speech and commerce for large crowds of users.
- The Snowden revelations of 2013 and the realization that intelligence agencies had infiltrated and exploited flaws in the implementation of the HTTP protocol led to a mass migration away from HTTP to the more secure HTTPS variant. This “upgrade” was a massive, uncoordinated effort spurred on by the largest threat to online communications contemplated in the Internet’s history — a competent, persistent threat actor with nation-state level resources at its disposal. It still took a decade. Notably, the intelligence agencies didn’t subvert the HTTP protocol itself; they merely took advantage of erroneous assumptions made by web server operators. The legality of their actions aside, from the point of view of the protocol, surreptitious surveillance was perfectly acceptable insomuch as it was possible to achieve without disrupting user communications.
Chapter 8
Here Comes Everybody: Overlay Networks
The great thing about the Internet is that it’s, well, the Internet. One giant, teeming interconnected network of networks, each network individually free to implement whatever weird policies it pleases as long as the weirdness stays within its boundaries. Internetworking as a concept provided the best of both worlds for network administrators and their users: global connectivity and reachability to billions of users spread across thousands of other networks while maintaining local control.
This lousy thing about the Internet is that it’s, well, the Internet. There’s just the one network with a protocol suite that, by definition, has to be adopted by all Internet users. If your use of the network is in some way at odds with that protocol suite, your options are to either a) stay within your local network boundaries and only use the TCP/IP suite when you want to connect to other networks, or b) find some other way to internetwork and lose the benefits of being able to attach to the same global network that everyone else jacks into by default.
Overlay networks are an attempt to untangle this Gordian knot — to have applications leverage the Internet as a bootstrapping mechanism for finding peers to connect to and then joining those peers in creating a new network topology with its own internal logic specific to that application, while other applications on the same machine connect to the Internet as usual.
An overlay network is a network that is built on top of an existing network. The overlay therefore relies on the so-called underlay network for basic networking functions, namely routing and forwarding. Today, most overlay networks are built in the application layer on top of the TCP/IP networking suite. Overlay technologies can be used to overcome some of the limitations of the underlay, at the same time offering new routing and forwarding features without changing the routers. The nodes in an overlay network are connected via logical links that can span many physical links. A link between two overlay nodes may take several hops in the underlying network.
— Tarkoma, Overlay Networks [1]
The widespread use of VPN services means that most Internet users are familiar with overlay networks in practice, if not in theory. What is a Virtual Private Network, after all, if not the simplest possible routing overlay on top of the existing Internet? In this chapter we will narrow our focus to the two arguably most successful — certainly most widely adopted — overlay networks: the anonymity network Tor and the file-sharing network BitTorrent.
Tor
Tor, short for The Onion Router, was developed in the 1990s by David Goldschlag, Mike Reed, and Paul Syverson at the United States Naval Research Lab. Realizing that the design of the routing and addressing layer of the Internet was fundamentally incompatible with privacy and anonymity for Internet-connected hosts, they began to work on a new approach to routing packets, with the goal of keeping user information private by encrypting routing information at the source and adding layers of encryption for each stop between the source and destination. The name ‘Onion Routing’ was chosen to describe the peeling off of these layers of encryption.
Onion Routing is an approach to achieving the separation of the identity of the user from the routing information used to guide signals around the Internet. This effectively nullifies one of the major “control points” built into the Internet, making the origin and destination of traffic no longer trivially identifiable by the Internet Service Provider (and hence the state). In the Onion Routing design, users’ Internet traffic is bounced around a network of volunteer-operated servers (known as “relays”) located around the world in order to disguise its origin and destination. First, the administrative information that routes users’ traffic around the Internet is wrapped in three layers of encryption. This traffic is then sent as a series of packets to the relay network. The traffic is first sent to an entry relay — a server that decrypts the first layer of encryption and reveals the address of the next relay in the chain. This next, “middle” relay decrypts the next layer of encryption, revealing the “exit” relay’s address. The exit relay then decrypts the last layer of encryption, finds the final destination of the traffic, and sends it on. Thus, no part of the network knows both the origin and the destination of the traffic, and anyone observing a particular user only sees them connecting to the network, not which websites they are accessing. This allows users’ identities to be concealed, becoming indistinguishable in a crowd of millions of other users.
– Syverson, Dingledine, and Mathewson (2004) Retrieved from https://svn-archive.torproject.org/svn/projects/design-paper/tor-design.html [2]
A core requirement for a functional onion routing overlay network is a large crowd of users. While onion routing disassociates user identity from routing information, in a network where only a handful of users are sending traffic, it’s trivial to deanonymize user data by running exit relays that inspect outbound user traffic and try to correlate it to known users. The larger and more diverse the set of users, the easier it becomes for any single user’s traffic to be ‘lost in the crowd’. Accordingly, an onion routing system requires a large set of uncorrelated users with different usage patterns, as well as different reasons for desiring privacy and anonymity in the first place. Consider an anonymous overlay network that was used exclusively by members of a criminal gang of malicious hackers; any inbound connections from known exit relays of the network would simply be dropped by hosts, as there would be no benefit in allowing them.
Realizing the power of literal safety in numbers, the U.S. Naval Research Lab released their code with an open-source license and encouraged its adoption amongst privacy-conscious technology enthusiasts and hobbyists in the hopes that a large enough base of users would allow their preferred user base — United States military personnel and intelligence agents looking to evade control and surveillance in remote jurisdictions — to blend in amongst them. This proof-of-concept code was examined and re-examined, designed and redesigned, with several test onion routing overlay networks being brought online and subsequently abandoned by different teams of developers, until finally a working, public implementation of an onion routing overlay network, Tor, was released in October 2002. By the end of 2003, a dozen volunteer nodes were running, and a few short months later, when the Tor design paper was released, that number had more than doubled:
As of mid-May 2004, the Tor network consists of 32 nodes (24 in the US, 8 in Europe), and more are joining each week as the code matures. (For comparison, the current remailer network has about 40 nodes.) Each node has at least a 768Kb/768Kb connection, and many have 10Mb. The number of users varies (and of course, it’s hard to tell for sure), but we sometimes have several hundred users — administrators at several companies have begun sending their entire departments’ web traffic through Tor, to block other divisions of their company from reading their traffic. Tor users have reported using the network for web browsing, FTP, IRC, AIM, Kazaa, SSH, and recipient-anonymous email via rendezvous points. One user has anonymously set up a Wiki as a hidden service, where other users anonymously publish the addresses of their hidden services.
– Syverson, Dingledine, and Mathewson (2004) Retrieved from https://svn-archive.torproject.org/svn/projects/design-paper/tor-design.html [2]
Tor’s popularity amongst Internet users increased in 2008 with the release of the Tor browser, a Web browser with support for Tor integrated directly, so that users can access both Tor.onion URLs as well as plain HTTP URLs alongside each other without having to worry about setting up relays or putting in any more effort than they would have to do when switching between other browsers such as Chrome or Firefox. The Arab Spring protests of 2010 and the Snowden revelations of 2013 led to widespread awareness among Internet users of the lack of privacy on the Web, the risks of surveillance and censorship by nation-state actors, and the utility of Tor for protecting themselves against these risks. This has led to a veritable flood of users adopting Tor and volunteering to run relays, increasing the size of the crowd and thus the anonymity of users of the Tor network, leading to increased protection and more users adopting Tor, leading to the current network size of thousands of nodes and millions of active users.
In times of protest and political turmoil, authoritarian governments routinely attempt to clamp down on access to the Tor network, either through active attacks like filtering access to bridges or passive attacks like deanonymization. The size of the Tor crowd and all the downstream benefits, such as performance and anonymity, that come from that size, have been instrumental in blunting the impact of these attacks. Every new attempt at filtering or blocking access has been met with a new wave of bridges, trackers and proxies being brought online, and new methods of obfuscating Tor usage from prying eyes. The ongoing efforts and support from the global community of volunteers who run the Tor network to resist these attacks has played a decisive role in ensuring the network’s resilience and utility, enabling it to survive and thrive.
BitTorrent
BitTorrent rose from the ashes of Mojonation, a for-profit company that employed BitTorrent architect Bram Cohen. Mojonation’s goal was to bootstrap a peer-to-peer network where users could buy and sell computational resources such as disk space, bandwidth, and processing time. Frustrated with Mojonation’s inability to attract funding and launch an operational network, in April of 2001, Bram began to focus on his own personal project, inspired by what he saw as a fundamental flaw in the design of the Internet: transferring data in spite of asymmetric bandwidth limits.
The asymmetric bandwidth problem, in a nutshell, was caused by Internet service providers optimizing their networks for their typical users’ satisfaction. As users download significantly higher amounts of data than they upload, it made sense to optimize user connectivity for high download capacity and limited upload capacity. This was ideal for users browsing the World Wide Web, where users connected to servers within data centers with high bandwidth connections, but for transferring files between users, upload capacity became a natural chokepoint. This meant that:
…in environments where information transferred between home users, speeds would suffer as the transfer was limited by the upload rate of the individual with the file. The downloader may have been able to download at 2 megabytes per second, but if the uploader could only send at a tenth of that speed, that was what the downloader received. For relatively small files such as text, images or music which maxed out at an average of 4 megabytes per file, this wasn’t much of an issue, but if you wanted to distribute larger files without paying out for server rental and high-capacity bandwidth there was little you could do. […]
Cohen’s idea to overcome the problem of asymmetric bandwidth was to have a large quantity of users upload a file at once. Although individually each one of them would be uploading at a fraction of the receiver’s download rate, communally they would max it out, providing a much faster transfer than a simple one-to-one transfer. If the file was broken down into pieces, the downloader could receive different pieces of the file from different sources at any one time. Furthermore, if the software could ensure that the pieces were ordered properly after they arrived, it wouldn’t even be necessary to download them in order. This would make things even quicker as pieces would be downloaded if they were available, regardless of whether they were the ‘next’ piece or not. As download order didn’t matter, downloads could be stopped and resumed later, allowing them to be spread out across multiple sessions. Having files in pieces even meant that it was not necessary to have the entire file before you were able to supply other users with copies of the pieces you already had. As more users joined in the ‘swarm’ there would be more copies of individual pieces available and more sources to download from, making the network faster as the load increased, a complete turnaround in data distribution principles.
— Robertson, Digital Culture Industry [3]
Solving the asymmetric bandwidth problem enabled users to distribute data exponentially faster than they had before. Cohen first demonstrated BitTorrent’s speed and utility by distributing a free pornographic video at blazing fast speeds. From that initial publicity stunt he moved on to more ‘acceptable’ use cases, like helping users distribute large collections of live music recordings and Linux software distribution images. These demonstrations showcased BitTorrent’s core value proposition: download speeds for large files actually became faster as they became more popular. Perhaps unsurprisingly, in retrospect, this led to widespread BitTorrent adoption amongst users trading copyrighted videos such as popular feature films and prime-time television shows. Before BitTorrent, a centralized server offering such files for download would be quickly saturated and unusable for most clients; on the off chance a server could survive and stay online, the server operators could expect to receive a legal cease-and-desist notice for offering copyrighted media for download. The swarm of BitTorrent users turned that problem on its head. The approach of distributing data across a swarm of users, perhaps not coincidentally similar to Tor’s approach of anonymizing traffic by routing it through a large crowd of users, inadvertently solved a problem that had led to the demise of the peer-to-peer file sharing application Napster: the removal of a centralized chokepoint that could be served with a copyright infringement lawsuit.
Launched in 1999, Napster was the first peer-to-peer file sharing application of the dot-com era. Its primary use was for sharing digitally encoded music files — MP3s — whose instant popularity with college students that had high-bandwidth connections available to them popularized the idea of file sharing for an entire generation of Internet users; its downfall in 2001 coincided with BitTorrent’s release. While Napster’s peer-to-peer network allowed users to share files stored on their own computers, it was backed by a centralized indexing server that kept track of the locations of all the files, managed and run by a for-profit company. Naturally, this led to Napster being targeted by lawsuits for sharing copyrighted content and ultimately being forced to shut down their indexing service in July of 2001, crippling the network.
BitTorrent’s equivalent to Napster’s index servers, torrent trackers, were run by volunteers, and switching between trackers was relatively straightforward. While delegating the task of providing a searchable index of content provided an arms-length distance between the BitTorrent company and the copyrighted media files their users were sharing, public trackers soon became targets for take-down by law enforcement, leading in turn to private invitation-only trackers, and finally the invention of “trackerless” torrents powered by a distributed hash table.
A distributed hash table, or DHT, is a replicated database that every BitTorrent user can connect, read and write to. Clients downloading a torrent ‘magnet’ file automatically decode its contents, submit a request for the specified file to the other users connected to the DHT, and are passed along from peer to peer until finding a peer that has contact information for other peers storing the requested data. That list of peers is returned to the client, who then requests pieces of the file from them. By storing contact information for every peer in the DHT, every user effectively became part of a tracker, removing the centralized choke point and making it impossible to take down a torrent without targeting every node participating in the swarm.
With the invention of trackerless torrents, lawyers tasked with taking down copyrighted content were forced to switch tactics. Targeting a few dozen web server operators indexing data was a far more tractable problem than serving notice to tens of thousands of users in a swarm. A war of attrition ensued between copyright holders and BitTorrent users. Internet Service Providers began receiving legal threats with lists of IP addresses of users connected to the DHT sharing copyrighted media. Upon receiving warnings from their ISPs, users would switch to private trackers, or use VPN clients to hide originating IP addresses, or simply ignore the warnings and dare their ISPs to respond. Copyright holders have tried seeding torrents with corrupted data to frustrate them and sending cease-and-desist letters, and offering settlements to individual BitTorrent users to intimidate and make public examples of them. This war of attrition is still raging to this day, with no clear victors and no signs of the core BitTorrent swarm being taken down.
Designed to solve the technical problem of slow file transfers, BitTorrent ended up, almost accidentally, solving the social problem of legal attacks targeting centralized choke points serving copyrighted content. Scholars from Canetti to Thoreau have discussed the power of crowds and mass movements to effect change by engaging in political action ranging from civil disobedience to outright rebellion. By leveraging the ability of network protocols to connect users across disparate legal jurisdictions, BitTorrent enabled a new sort of mass civil disobedience — a crowd too large in size to feasibly be dispersed, with a pack of zealots at its core that simply refused to yield.
Takeaways:
- Overlay networks are networks built on top of the global Internet, using it as a bootstrapping mechanism for finding peers to connect to and build new virtual networks with. These new virtual networks are able to incorporate features and functionality that the existing Internet cannot.
- The most successful overlay networks rely on the size of the crowd of users connected to them to provide functionality that users cannot individually leverage by themselves — the anonymity of being lost in the crowd in the case of Tor and bandwidth that is more than the sum of its parts in the case of BitTorrent.
- By removing central points of control and distributing responsibilities among their participants, overlay networks like Tor and BitTorrent have inadvertently opened a new realm where users are able to contest and resist legal coercion and government control. Members of the crowd collectively empower each other to assert their rights in ways that users relying on client-server protocols, such as HTTP with clients running web browsers and servers operating websites, cannot.
Chapter 9
Bitcoin: A Fork In The Road
For an overlay network that has made such an outsized impact on recent Internet history, Bitcoin came from relatively humble beginnings. In 2008, an anonymous programmer (or group of programmers, perhaps) answering to the name ‘Satoshi Nakamoto’ shared a nine-page white paper on a mailing list for cryptography enthusiasts describing his design for a peer-to-peer electronic cash system:
A purely peer-to-peer version of electronic cash would allow online payments to be sent directly from one party to another without going through a financial institution. Digital signatures provide part of the solution, but the main benefits are lost if a trusted third party is still required to prevent double-spending. We propose a solution to the double-spending problem using a peer-to-peer network.
— Nakamoto, S. Bitcoin White Paper [1]
Addressing the problem of double-spending, where the value of a banknote or coin is redeemed more than once, is a core requirement for any viable monetary system. Governments expend a tremendous amount of resources and effort to mitigate one facet of the double-spend problem: counterfeiting, where new tender is issued without the backing or endorsement of the State. In pre-modern times, coins were stamped with pictures of monarchs or appeals to Almighty God. This served the dual purpose of promoting the authority of the monarch over the realm and assuring citizens that the integrity and authenticity of the coins were vouched for by a central authority. The concern that coins could be issued and stamped without approval was constant, and counterfeiting was considered tantamount to treason and, in many countries, punishable by death.
A second kind of double-spending is debasement, which can appear in several different guises. Coins could be clipped or shaved or undetectable amounts of metal, and the shavings collected to melt into new coins. In this way, one hundred coins could be turned into a hundred and one (for example), each with the same nominal face value but with one percent less monetary metal in each coin. After the invention of paper currency, debasement by the central authority vouching for the currency became standard practice; notes would be issued and declared redeemable for a given amount of precious metals, only for the given amount to be decreased at the discretion of the state. (While it’s true that debasement was common practice before the invention of paper currency, it was impossible to restamp or revalue metal coinage silently – coins had to either be restruck at the mint or have their value redeclared in a public decree.) The necessity of a central clearing authority in preventing the first kind of double-spending, counterfeiting, rendered the second kind of double-spending, debasement, inescapable; if the authority was charged with determining when double spending had occurred, it was free to turn a blind eye to its own double-spending when it was convenient to do so.
Nakamoto proposed an elegant hack to work around the need for a central authority in the first place — what he called a ‘timestamp network’, where peers race each other to prove that they’ve performed a unit of work, and attach that proof to a list of transactions called a “block”. Once a peer creates a valid block with a provable unit of work, they timestamp it and broadcast it to other peers, who race to build another provable unit of work that references the previous block, creating a chronologically ordered chain of blocks:
The steps to run the network are as follows:
1) New transactions are broadcast to all nodes. 2) Each node collects new transactions into a block. 3) Each node works on finding a difficult proof-of-work for its block. 4) When a node finds a proof-of-work, it broadcasts the block to all nodes. 5) Nodes accept the block only if all transactions in it are valid and not already spent. 6) Nodes express their acceptance of the block by working on creating the next block in the chain, using the hash of the accepted block as the previous hash.
— Nakamoto, S. Bitcoin White Paper [1]
Nodes that participate in the Bitcoin proof-of-work contest are issued virtual representations of coins for each block found as a reward for winning the contest, as well as all of the fees paid by users to have their transactions included in that block; they are then free to distribute these coins as they see fit. While it is relatively difficult to win the proof-of-work contest — participants have roughly ten minutes to solve the puzzle before anyone else, broadcast a block, and expend a tremendous amount of computational resources to do so — it is trivial to verify the results of the proof-of-work contest and thus confirm the entire history of coin issuance and transactions of coins. Removing the ability of central authority to debase the currency was one of Nakamoto’s primary goals in designing Bitcoin [1]; making coins difficult for any member of the network to create but trivial for any member to audit was how he accomplished it.
In rendering the need for a central authority to verify transaction history obsolete, Nakamoto created a new sort of public directory of data — one unlike anything that had come before it in the Internet’s history. One way to model the information available on the Internet is as a collection of loosely connected data silos. The Internet superstructure takes on the responsibility of routing users from their source to their destination; from there, the destination — websites, chat rooms, newsgroups, FTP servers, etc.— serves the user with the data they request, with the destination implicitly taking on the responsibility of assuring that data is available and “correct”, with “correctness” being a property mutually (if only implicitly) agreed upon by the source and the destination.
An online retailer like Amazon, for example, has one idea of what it means to serve “correct data”; a stock exchange like Nasdaq has another; a State licensing agent like the DMV or a federal agency like the IRS has still another. Crucially, the Internet superstructure itself has no ‘neutral’ data store. It is primarily concerned with moving packets from end to end, and the relatively minimal amount of data that clients need to be aware of to accomplish the quotidian task of shoveling packets from A to B — that is, IP addresses and DNS names — is not ‘neutral’ but in fact administered and maintained by third parties such as ICANN, the Regional Internet Registries, and the root DNS servers.
Recall Nick Szabo’s words quoted in Chapter 4 about the “property rights nature of public directories”. In each of our previous examples, property rights — literally, the rights to create, read, update, or delete data — over the directory are held by a central stakeholder and delegated (or revoked) only with their permission. Customers have the ability to view Amazon store listings, and vendors have the ability to update the inventory they have listed on Amazon because Amazon grants them the right to do so. Citizens have the right to view or update their personal information on the DMV website only because the state DMV licensing authority, as the owner of the directory, allows them to do so. If you’ve ever found yourself in the situation of having paid a debt with the IRS and waiting ages for them to update their database to reflect the payment, or had an item delisted from eBay for violating their terms of service, or having a social media account suspended for posting something a moderator didn’t appreciate, you’ve experienced the downsides of these rights-by-permission-only. The unique value proposition of Bitcoin compared to centralized databases is that there is no way within the Bitcoin protocol to modify the rights that a Bitcoin keyholder has over the coins their keys protect. Thieves, hackers, and law enforcement personnel, must compel keyholders to forfeit access to their keys if they wish to part with their coins.
In the first few years after the Bitcoin network came online, coins were given away for free or traded as novelties, with little if any real economic value assigned to them by users of the network. With few participants and little processing power attached to the network, proof-of-work contests – colloquially, ‘mining blocks’— was relatively easy for any user willing to run the software. As user adoption grew, so did enthusiasm and speculation (along with processing power dedicated to mining Bitcoin blocks and collecting the associated rewards). A land rush dynamic, similar to the one that took place when domain names first became available for purchase, soon took hold, but not before the majority of early adopters had either given their coins away, lost access to the keys that secured them, or squandered them on ill-fated speculative ventures. This initial ‘golden age’ is one of the reasons that many users of the Bitcoin overlay network consider it impossible to replicate; alternative attempts at bootstrapping distributed timestamp networks would need to both a) either solve a more compelling problem than Bitcoin has (that is, obviating the need for a central clearing authority) or solve it in a more compelling fashion and b) solve a problem that was resolved by an accident of fate, the “fair launch” problem of founders and early adopters hoarding coins.
The Blocksize War
In the early years of Bitcoin’s operational history, sending coins from one address to another was fast and inexpensive…sort of. It was ‘fast’, in the sense that early users didn’t fully appreciate the need to wait for nodes to chain more blocks on top of the blocks that included their transactions, to be sure the block that includes their transaction isn’t replaced in a reorganization of the chain.
…the blockchain is a series of n blocks, and at any given time the most recent several blocks are not guaranteed to be permanently included. It is possible for the blockchain to fork by having multiple potential (often inconsistent) blocks which claim to be the last block in the chain. Eventually one of these blocks will win and be permanently included, but it won’t always be immediately clear which block this is. When an apparently valid block is replaced by a competing block, this is called a blockchain reorganization and the replaced block is called an orphan block.
Given this we might be tempted to say a transaction is “confirmed” once it has been included in a block which is not the very last block in the blockchain. However, it is possible (although rare) for the last n blocks to be orphaned in a reorganization. This is exponentially less likely to occur the larger n gets. It typically happens multiple times a day, for example, that a single block is orphaned, but has happened only a few dozen times in history for n between 2 and 4, and exactly once for greater than 4 (a 24-block reorganization in March 2013 due to a technical glitch).
Acceptable Confirmation
Barring technical glitches, formal modeling of Bitcoin suggests that large reorganizations are exponentially unlikely, but possible. Therefore we can never say with certainty that a transaction is “confirmed” because it is always possible that a transaction will apparently be included in the blockchain but be replaced by a large reorganization.
In practice, the community has adopted 6 blocks as a standard confirmation period. That is, once a transaction is included in a block in the blockchain which is followed up by at least 6 additional blocks, the transaction is called “confirmed.”
– https://www.coincenter.org/education/crypto-regulation-faq/how-long-does-it-take-for-a-bitcoin-transaction-to-be-confirmed/ [2]
Despite the present-day understanding that at least six blocks need to be built on top of the block including a transaction before that transaction can be considered “confirmed”, many early adopters treated transactions with no blocks built on top of them – “0-conf” transactions – as if they were effectively irreversible. This misunderstanding was compounded by how inexpensive transactions were in those early years. All Bitcoin transactions have fees associated with them; the fees are paid to the miners as an incentive to include the transaction in a block.
Every Bitcoin transaction spends zero or more bitcoins to zero or more recipients. The difference between the amount being spent and the amount being received is the transaction fee (which must be zero or more).
Bitcoin’s design makes it easy and efficient for the spender to specify how much fee to pay, whereas it would be harder and less efficient for the recipient to specify the fee, so by custom the spender is almost always solely responsible for paying all necessary Bitcoin transaction fees.
When a miner creates a block proposal, the miner is entitled to specify where all the fees paid by the transactions in that block proposal should be sent. If the proposal results in a valid block that becomes a part of the best block chain, the fee income will be sent to the specified recipient…
Historically it was not required to include a fee for every transaction. A large portion of miners would mine transactions with no fee given that they had enough “priority”. Today, low priority is mostly used as an indicator for spam transactions and almost all miners expect every transaction to include a fee. Today miners choose which transactions to mine only based on fee-rate.
– Bitcoin Wiki, https://en.bitcoin.it/wiki/Miner_fees [3]
The fee associated with a transaction can be modeled as a sort of congestion pricing; when many transactions are outstanding, miners choose which transactions to include in the next block based on how profitable the fees are. Each unconfirmed transaction is essentially bidding for space in the next block against every other unconfirmed transaction. If there are not enough pending transactions to fill a block, then transactions with higher fees attached should not, in theory, crowd out transactions with lower fees; only with full blocks will transactions become ‘expensive’.
Misery, it is said, is a function of misplaced expectations. As the Bitcoin network became more popular and more users began sending more transactions, there was a radical change in shared expectations for how long a transaction would take to confirm and how much a user would pay for a transaction to be included in a block. This mismatch in expectations laid the groundwork for what has since come to be known as ‘The Blocksize War’ [4], a conflict between users, developers, miners, and businesses that relied on Bitcoin for their daily operations. What initially started as a disagreement about the optimal size of a Bitcoin block (which determines the number of transactions in each block, and, in turn, how high transaction fees would be and how long a transaction would take to be confirmed) quickly became about much more than that:
As the war progressed, it emerged that the struggle was perhaps more complex than just the maximum size of blocks; the battle went right to the core of Bitcoin’s DNA. The contention was essentially about four somewhat interrelated issues:
- The level of blockspace available in each Bitcoin block: Essentially, whether the eventual state should consist of surplus capacity available in the blocks, or consistently full blocks.
- How to modify the rules of the Bitcoin protocol: Whether the rules on the validity of Bitcoin blocks should change relatively easily, or whether they should be more robust and only change in exceptional circumstances, with broad support from all interested parties.
- The significance of the nodes of ordinary users: The extent to which, if any, validating nodes of the ordinary end users had a say in enforcing Bitcoin’s protocol rules.
- Time preferences: Whether Bitcoin was like a tech startup which should prioritise gaining market share in the short term; or if it was a long-term project, a new global money.
— The Blocksize War [4]
Putting aside technical and philosophical motivations, the question the participants in the conflict found themselves trying to answer was as straightforward as it was perplexing: “Who gets to make changes to the rules of Bitcoin?”
Initially, Bitcoin’s leadership model was that of a de facto “benevolent dictator for life” (or BDFL, for short), a term first popularized by the Python programming community; project management was led by Nakamoto, who would opine on proposals and weigh in on disagreements between developers in public forums and mailing lists. When Nakamoto stepped away from the project a scant few years after the launch of his timestamp network, other developers took over his administrative duties; uncontroversial changes and bug fixes were made in his absence, and development continued apace until the question of increasing the block size came about. The problem of shipping a controversial change to the network consensus layer was that it requires near-unanimous adoption; a partially-adopted consensus change could lead to the network being partitioned, with pre-change nodes having one view of what the Bitcoin transaction history should look like and post-change nodes having another. This would effectively ‘unsolve’ the double-spend problem, which in turn could call into question the utility of Bitcoin in the first place. When Nakamoto was serving as de facto BDFL, this was never an issue; users blindly adopted any changes he approved or endorsed. In his absence, there was no single champion to dictate which changes were necessary to implement and which would be best avoided.
A few of the more vocal contributors to the Bitcoin project took it upon themselves to become vocal champions for the block size increase, advocating for the imposition of a breaking change in the protocol and cementing the de facto BDFL model of development. One developer with a long history of Bitcoin development was quoted as saying:
“A lot of people are pushing me to be more of a dictator… that may be what has to happen with the block size. I may just have to throw my weight around and say this is what it’s going to be. If you don’t like it, find another project.”
This statement was followed by attempts to coordinate a block size increase with several Bitcoin exchanges, in an apparent attempt to circumvent the process of software peer review. These attempts in turn led to reactions (consternation and outrage from other developers), counter-actions (alternative Bitcoin clients being released with changes implementing the block size increase), and counter-counter-actions, in a years-long internecine conflict that saw developers, miners, and users all at odds with each other.
Recounting the full history of the conflict would require its own book; Jonathan Bier’s ‘The Blocksize War’ is an excellent rundown of the players involved and the blow-by-blow history, and is recommended for further reading. What matters to us in this context is the outcome of the conflict: the attempt to make breaking changes to the protocol without the consent of the users was ultimately thwarted. At the 11th hour, miners, exchanges, and developers bent on supporting the block size increase were forced to announce publicly that they had abandoned their efforts, as there was no confidence that users would update their nodes to run any new Bitcoin client software implementing the block size increase.
It was now, finally, widely accepted that arranging meetings with large corporations in the space and trying to decide on changes to the protocol rules would not work. The majority of miners also did not have the ability to relax protocol rules. If one wants to change the protocol rules, one has to persuade and campaign for the support of end users and investors, who need to opt-in to the new rules. It was ordinary users who had the final decision-making power, and this was the financial sovereignty that made Bitcoin unique and compelling. After an astounding victory, the small block narrative, that end users had to agree to protocol rule changes, was finally seen as compelling.
— The Blocksize War [4]
… And Everything After
Nakamoto’s invention, and particularly the land rush dynamic that emerged during the early golden age, spawned literally thousands of alternate cryptocurrencies, each scrambling to implement new features and functionality to differentiate themselves, with promises of future riches woven into narratives about being The Next Bitcoin. An analysis of other cryptocurrencies besides Bitcoin might be theoretically interesting at this point, but practically speaking, there are no new protocol dynamics at play here; the implementation of a distributed timestamping network was Nakamoto’s invention, and the actual protocol that nearly every attempted Bitcoin successor relies on is HTTP, inheriting the limitations of HTTP’s client-server model. It’s difficult to convince users to download, install, and run peer-to-peer software when clients can readily connect to default HTTP-accessible nodes with little effort. Almost every alternative to Nakamoto’s software relies on this dynamic of clients (at least, the bulk of them) connecting to HTTP gateways instead of running their own nodes in a peer-to-peer network, and accordingly, the analysis of the HTTP protocol in Chapter 7 applies to them as well. It’s not possible for users of a client-server protocol like HTTP to assert their rights as Bitcoin users did during the Blocksize War; clients are at the mercy of the server. As of the writing of this text, it doesn’t appear likely that Bitcoin will find any competition for the services it provides as a timestamping network; the switching costs for Bitcoin users to move to a new network grow higher with every block.
Takeaways:
- Satoshi Nakamoto’s invention — a distributed timestamp network to remove the need for a central clearing authority to issue and verify the integrity of a currency — created a new type of public directory of data, one with a fundamentally different set of property rights than any preceding Internet-connected data store.
- The Blocksize War was a watershed moment in the history of the Internet and peer-to-peer software. Crowds of users were able to subvert the will and intentions of developers trying to assume the mantle of benevolent dictators for life, cryptocurrency exchanges, well-monied investors, and CEOs of large corporations simply by choosing which software they were willing to run.
- Because no other network has passed through the same trial-by-fire that Bitcoin was forced through by the Blocksize War, and no other network appears to have had a large population of users willing to run their own software and assert their rights over what rules the network should adhere to, there doesn’t appear to be any alternative to Bitcoin for the functionality it provides. We’re stuck with this version of the network, come what may.
Chapter 10
The End Of The Beginning (Of The Internet)
The state is not something which can be destroyed by a revolution, but is a condition, a certain relationship between human beings, a mode of human behaviour; we destroy it by contracting other relationships, by behaving differently.
– Ward, Colin. Anarchism: A Very Short Introduction [1]
This text began with a history of the lowest layer of the TCP/IP protocol suite from the very first protocol, NCP, on to TCP/IP, and up the entire stack of Internet protocols, visiting DNS, BGP, and HTTP/S — the protocols at the core of daily Internet experience for billions of users — and finally arriving at the overlay networks. There are no higher levels to ascend to.
When I first began the research for this book, I had in mind Mark Twain’s words about history never repeating but often rhyming. By investigating and cataloging the history of network protocols, how they are developed, and how they came to life, I had hoped it would be possible to anticipate future problems that engineers working with protocols might face in the future, only (hopefully!) armed with knowledge from past efforts. I drew a simple diagram of the OSI model that I had been taught as a young network engineer, mapped real-world protocols to each layer of the model, found as many relevant texts, magazines and interviews as I could for each protocol that I had mapped, and immersed myself in them, chapter by chapter.
As the text began to take shape, I found that old assumptions I had made about network protocols I worked with on a daily basis were incorrect or even sometimes incoherent. Stories I had told myself or had absorbed over the years as part of my experience working as a network engineer were just that — stories — with bits and pieces of folk wisdom spread over a smattering of opinions that tied loosely, at best, to actual historical fact. I had thought that it wasn’t possible to successfully transition from a centralized protocol to a distributed one, but then what to make of HOSTS.TXT and the move to DNS? I had thought that a mass protocol upgrade after the NCP flag day and the ossification of TCP was not possible, but then what to make of the migration from HTTP to HTTPS in the wake of the Snowden revelations? If asked whether I would consider SMTP to be a successful protocol, I would have said of course, there are more users of email than the World Wide Web, after all, but I myself found my own private mail server suddenly unable to send and receive messages for large swaths of Gmail users, banished to some unknown blacklist for the unpardonable sin of being hosted on a server in an IP address range adjacent to a suspected spammer.
Theoretically, our Internet is a globally connected end-to-end network of networks; in practice, it is riddled with chokepoints and balkanized into de facto fiefdoms, and has been from the start. We, as users of the Internet, have the ability to evade and sidestep these obstacles, to re-assert and regain power in these inherently asymmetric relationships, but most of us lack either the knowledge or the will to do so. Put simply, Internet protocols work well enough, most of the time, for most users, and so we are willing to forgive them for their flaws. If tens of thousands of users are inconvenienced by a BGP routing failure here or an ISP filtering websites there, that inconvenience simply does not inflict a high enough cost on the other five billion or so users to walk away from the benefits the Internet provides — truly global connectivity — much less any incentive to adopt new rules for a new global communications network from scratch. Even the most high-profile attack against network protocols to date, the surreptitious eavesdropping and surveillance of the bulk of HTTP traffic by the NSA and GCHQ revealed by Edward Snowden, resulted in a mass migration to an in-place upgrade of the HTTP protocol to HTTPS. The idea of abandoning the Web as an alternative was simply unthinkable.
There’s an argument to be made that these protocols have a sort of baked-in hierarchy to them — that is, that users of the Internet with the most control over their presence on the Internet have reaped the most reward, building towering business empires, for example, or elevating themselves to the status of global celebrities, while those with the least find themselves trapped inside an endless feed of mind-numbing content interspersed with privacy-violating advertisements. My only comment on that argument is that the Internet is a voluntary system; you can pick and choose who you interact with and what means you use to do so. No one is forcing you to log in to Facebook to snoop on your old high school crush; it is not compulsory for you to buy and sell goods through Amazon. Being connected to the Internet and connecting to others can mean anything from bootstrapping your own Internet Service Provider, to signing up for an email address from a hosted email service provider, to — as most users understand the experience of ‘being on the Internet’— merely passively consuming content from websites. Wherever you land on this spectrum of ‘being online’, it is completely within your power to migrate above or below your station.
Mass participation in this voluntary system, and the ability that users have to increase or decrease their level of participation, has profound implications for its participants, which at the time that I write this covers fully half of humanity. It is my belief after having undertaken the research for this text that the protocols that power the Internet and our collective decision as a species to rely on them have fundamentally altered the distribution and nature of political power — literally, how the polity experiences and projects power — and in turn, irrevocably altered relations between individuals and the sovereigns that govern them.
If you have been reading all this and are surprised at this last paragraph: believe me, I know the feeling. I wasn’t expecting to come to this conclusion either. What was meant to be a dry history of internetworking software is now instead a roadmap for what comes after the nation-state.
We will first examine how network protocols are governed — that is, how the policies, actions, and affairs of internetworking protocol development are managed — so that we can begin to try to understand how the Internet poses an existential threat to sovereign nation-states and how it weakens and degrades the “system of social control under which the right to make laws and the right to enforce them, is vested in a particular group in society”. We need to try to understand who is supposedly ‘in charge’ of the Internet and what it is they are ‘in charge’ of, before we can actually understand who is actually in control.
Protocol Governance: From Within
The Internet protocol suite — the layered nest of protocols that we have examined over the last 9 chapters, from TCP/IP to DNS to BGP to HTTP and HTTPS — is watched over by several different groups in what is commonly referred to in academic literature as a multi-stakeholder governance model. These groups are the bodies ‘that govern from within’ — the actors that are part of the internal process of protocol management. They are, in no particular order:
ICANN and IANA - The Internet at its core is made up of names and numbers. Recall again from Chapter 4: “A name indicates what we seek; an address indicates where it is. A route indicates how we get there.” The Internet Corporation for Assigned Names and Numbers, or ICANN, was set up in 1998 to administer and coordinate these names and numbers:
To reach another person on the Internet you have to type an address into your computer — a name or a number. That address must be unique so computers know where to find each other. ICANN coordinates these unique identifiers across the world. Without that coordination, we wouldn’t have one global Internet.
In more technical terms, the Internet Corporation for Assigned Names and Numbers (ICANN) helps coordinate the Internet Assigned Numbers Authority (IANA) functions, which are key technical services critical to the continued operations of the Internet’s underlying address book, the Domain Name System (DNS)). The IANA functions include: (1) the coordination of the assignment of technical protocol parameters including the management of the address and routing parameter area (ARPA) top-level domain; (2) the administration of certain responsibilities associated with Internet DNS root zone management such as generic (gTLD) and country code (ccTLD) Top-Level Domains; (3) the allocation of Internet numbering resources; and (4) other services.
— ‘What Does ICANN Do?’, Retrieved from https://www.icann.org/resources/pages/welcome-2012-02-25-en [2]
The ISOC, IAB and IETF - The alphabet soup that oversees Internet resources and steers protocol development. The Internet Society (ISOC) oversees the Internet Architecture Board (IAB), which in turn oversees the Internet Engineering Task Force (IETF).
The Internet Society (ISOC) is responsible for promoting open development, evolution, and Internet use throughout the world. ISOC facilitates the open development of standards and protocols for the technical infrastructure of the Internet, including the oversight of the Internet Architecture Board (IAB).
The Internet Architecture Board (IAB) is responsible for the overall management and development of Internet standards. The IAB provides oversight of the architecture for protocols and procedures used by the Internet. The IAB consists of 13 members, including the chair of the Internet Engineering Task Force (IETF). IAB members serve as individuals and not representatives of any company, agency, or other organization.
The IETF’s mission is to develop, update, and maintain Internet and TCP/IP technologies. One of the key responsibilities of the IETF is to produce Request for Comments (RFC) documents, which are memoranda describing protocols, processes, and technologies for the Internet. The IETF consists of working groups (WGs), the primary mechanism for developing IETF specifications and guidelines. WGs are short term, and after the objectives of the group are met, the WG is terminated. The Internet Engineering Steering Group (IESG) is responsible for the technical management of the IETF and the Internet standards process.
– ‘Internet Standard organizations’, retrieved from http://wiki.ciscolinux.co.uk/index.php/Internet_Standard_organizations [3]
The W3C: The prominent role of the World Wide Web in the average Internet user’s experience has meant that the World Wide Web Consortium, or W3C for short, plays an outsize role in setting Web standards [4]. While the IETF is charged with coordinating working groups and publishing RFCs for the HTTP protocol, the W3C liaises with them as a sister organization.
ICANN, ISOC, and the W3C are collectively responsible for the administration and governance of the protocols that make up the TCP/IP suite.
The overlay networks we have explored — Tor, BitTorrent and Bitcoin — each have their own unique internal process for proposing modifications and changes. As each overlay network started with an initial software artifact with specifications codified after release, their governance processes — to the extent that there is one — is an ad hoc, ex post facto jumble of ‘the best laid plans of mice and men.’
Tor:
…to make a change to the Tor protocol, you write up a little design document, and send it to the tor-dev mailing list. Once it meets editorial quality, it can go into the proposals repository. Once it’s implemented, it can be merged into the spec.
– ‘Tor design proposals’, Retrieved from https://blog.torproject.org/tor-design-proposals-how-we-make-changes-our-protocol/ [5]
BitTorrent:
The BitTorrent Community Forum coordinates the development of the BitTorrent protocol suite and its reference implementation. It is the wish of Bram Cohen that the BitTorrent mainline python implementation remain open source and that the protocol development process be modelled after the Python Enhancement Proposal (PEP) process.
When a new proposal is submitted, one of the BitTorrent.org editors assigns a BEP number and updates the BitTorrent Enhancement Proposal index appropriately. Each document has a number that never changes and the history of the document is maintained in git.
— ‘Index of BitTorrent Enhancement Proposals’, Retrieved from https://www.bittorrent.org/beps/bep_0000.html [6]
Bitcoin:
Bitcoin governance is the process by which a set of transaction and block verification rules are decided upon, implemented, and enforced, such that individuals adopt these rules for verifying that payments they received in transactions and blocks fit their subjective definition of “Bitcoin”. If two or more individuals adopt the same set validation of rules, they form an inter-subjective social consensus of what “Bitcoin” is.
The process of updating or changing these rules to address bugs or implement new features begins with research leading to implementation proposals by developers. Community consensus is crucial; contentious proposals may not be integrated into the main codebase. That said, developers can fork the Bitcoin codebase to bypass consensus, as seen in BIP-148 during the Blocksize War.
Rule changes are implemented as softforks or hardforks. Softforks, compatible with older nodes, are safer but potentially coercive; hardforks create separate networks. Persuading users — node operators — to adopt new software versions is key, with varying influence among node operators like blockchain explorers and exchanges.
Rules are enforced by individual nodes verifying transactions and blocks, rejecting and banning peers that submit invalid transactions.
— Rochard, P. ‘Bitcoin Governance’ Retrieved from https://pierre-rochard.medium.com/bitcoin-governance-37e86299470f [7]
Protocol Governance: From Without
So much for how protocols are governed from within. To understand how protocols are governed from without — that is, how actors outside the formal governance process attempt to influence and control the development and operation of network protocols — we will revisit David Clark’s words quoted in Chapter 2, describing the tussle [8]:
Perhaps the most important consequence of the Internet’s success is that the common purpose that launched and nurtured it no longer prevails. There are, and have been for some time, important and powerful players that make up the Internet milieu with interests directly at odds with each other. The Internet is not a single happy family of people dedicated to universal packet carriage. There is contention among the players… The Internet, like society in the large, is shaped by controlled tussle, regulated not just by technical mechanism but by mechanisms such as laws, judges, societal opinion, shared values, and the like. There is no “final outcome” of these interactions, no stable point, and no acquiescence to a static architectural model. Today, the Internet is more and more defined by these tussles.
The tussle to influence protocol development and operation finds the actors identified in the previous section — ICANN, the ISOC, etc.— on one side of the field of play, who we can loosely describe as ‘those charged with setting the rules of the Internet’. On the other side of the field is every actor who either wishes they were in the position of setting the rules governing how Internet users connect and communicate, or worse, those who assumed they were already in that position. This includes every large Internet macro-intermediary from Chapter 7 — with Google, Apple, and Cloudflare at the top of the list — as well as every nation-state (and, by extension, every intelligence agency and military apparatus) in the world. The billions of users of the Internet are stuck in the middle.
The options these actors have for asserting control over the users in the middle are surprisingly limited. These users apparently cannot be cajoled by vendors into switching to a new suite of protocols, as we learned in Chapter 2; building something bigger, better, or faster in the hopes of convincing users to bear the switching costs and uncertainty of moving to a new protocol en masse is a non-starter. Similarly, they cannot be co-opted by technocrats into switching to a new set of protocols, as we learned in Chapter 3; the most ardent amongst them will fight tooth and nail to maintain the status quo. As more users adopt a given protocol, a land rush dynamic is bound to take hold, whether it be speculators hoarding domain names as we saw in Chapter 4 or new routing peers racing to establish commercial relationships with each other as we saw in Chapter 5 with BGP. The winners of this land rush “anchor” the protocol, so to speak, in the sense that once they have acquired resources worth protecting, they have little incentive to adopt any new changes to it that might negatively impact their piece of the pie. This anchoring can be seen as the start of the process of protocol ossification, where coordinating changes to a protocol becomes more and more difficult and the marginal benefits to the existing user base become negligible in comparison to the potential risks, to the detriment of potential future protocol users. A failure to protect these benefits may lead to a sort of ‘soft failure’ of the protocol, as we saw in Chapter 6 with SMTP; it can continue to function in theory but in practice be only available to large corporate players able to dedicate resources to mitigate the various forms of soft failure. It is possible for a protocol to become so widely adopted, to be of benefit to so many disparate players, and thus so far along in the process of ossification, that changes can only be made in the face of an existential threat — as we saw in Chapter 7, with the mass migration to HTTPS from HTTP — and even then, these changes would have to be rolled out incrementally in a voluntary opt-in process that could take years if not decades. This is perhaps the best opportunity that state actors have for asserting control over protocol users — subverting the operation of an existing protocol and taking advantage of the inability of the user base to adapt to the threat due to the inherent delays in coordinating upgrades amongst the crowd. While these sorts of delays may not be ideal, they are a symptom of success, not failure; the higher the number of stakeholders, the longer it will take to coordinate the deployment of changes amongst them from the bottom up. Finally, just as protocols can be layered on top of each other, networks can be overlaid on top of each other, such as Tor, BitTorrent, and Bitcoin, if the underlying Internet is unable to meet the needs of their users, allowing for not just new means and methods of communication but in fact entirely new structures that allow their users to rise above the protocol tussle completely, extricating themselves from the middle ground.
Lacking either enough carrots to lure the crowd of users to a new protocol or enough sticks to force them off of an existing one, the state finds itself in an existential bind. In order to exert control and maintain sovereignty over their territory and citizens, states have several different means at their disposal. Helmut Willke [9] identified four of these means that, alone or in concert with each other, can be used to exercise power: force, law, money, and knowledge. Accordingly, nation-states can — and have — used physical force against Internet users engaging in behavior they find threatening (for example, arresting users that have disseminated information damaging to the state); created new laws and reinterpreted existing laws to regulate and intermediate online interactions (e.g., the Digital Millennium Copyright Act, or DMCA), and sent representatives to subvert existing Internet governance processes as a form of de facto ‘lawfare’; provided money to fund projects that challenge other rival states and rival firms in a de facto contest to re-assert their sovereignty and resist hegemony; and finally (and perhaps most perniciously) made use of techniques such as filtering, blocking, intercepting and monitoring connections between protocol users, and leveraging protocols to blast propaganda and misinformation to their own citizens and citizens of other States, all in an attempt to control the flow of information and define exactly what constitutes ‘knowledge’ (as opposed to e.g., propaganda, disinformation, or ‘fake news’).
If we observe that the state has used and is using all of its capabilities to exercise its power over the Internet, we are forced to ask ourselves why — to what ends does the state aspire? The answer is, simply, survival. The Internet and the protocols that underpin it, represent:
…an international network of computers and computer networks connected to each other, sharing a common name and address space —[that] differs from earlier advances in information technologies because it combines global reach with extremely low barriers to entry. Governments have far more difficulty imposing border controls on the Internet because it relies on packet switching rather than circuit switching. The difficulty in imposing border controls on Internet communications is compounded by the low barriers to entry-anyone with a laptop computer, access to an Internet service provider, and appropriate software can publish and read in cyberspace. The Internet relies on already existing physical communication infrastructures, making it unnecessary to expend huge amounts of money to communicate globally. The ease with which people can participate in cyberspace activities enabled the Internet to grow exponentially with virtually no governmental oversight. This growth has created a cyber-culture that celebrates freedom and distrusts traditional political institutions trying to come to grips with the implications of this profound electronic revolution in information technology. No such transnational culture developed in the early days of the telegraph, radio, or television. Cybernauts most closely resemble medieval merchants who developed substantive rules and practices to regulate transnational trade — the lex mercatoria — outside traditional political institutions. Commentators have seen in the Internet a threat to sovereignty because the Internet challenges the three historic functions of the state: providing national security, regulating economic activities, and protecting and promoting civic and moral values.” In short, the Internet threatens the government’s ability to control power, wealth, and morals within its territory.
— Perrit, Henry H. The Internet as a Threat to Sovereignty? Thoughts on the Internet’s Role in Strengthening National and Global Governance [10]
By serving as a shared lingua franca for users across the Internet to communicate with, network protocols can connect the entire globe into one shared, sprawling ‘contact zone’— an arena where radically different cultures and ideas come into contact with each other. Mary Louise Pratt’s 1991 lecture, ‘Arts of the Contact Zone’ [11], describes how shared languages shape and codify communities:
Many commentators have pointed out how modern views of language as code and competence assume a unified and homogeneous social world in which language exists as a shared patrimony—as a device, precisely, for imagining community. An image of a universally shared literacy is also part of the picture. The prototypical manifestation of language is generally taken to be the speech of individual adult native speakers face-to-face (as in Saussure’s famous diagram) in monolingual, even mono dialectal situations — in short, the most homogeneous case linguistically and socially. The same goes for written communication. Now, one could certainly imagine a theory that assumed different things — that argued, for instance, that the most revealing speech situation for understanding language was one involving a gathering of people each of whom spoke two languages and understood a third and held only one language in common with any of the others. It depends on what workings of language you want to see or want to see first, on what you choose to define as normative…
In keeping with autonomous, fraternal models of community, analyses of language use commonly assume that principles of cooperation and shared understanding are normally in effect. Descriptions of interactions between people in conversation, classrooms, medical and bureaucratic settings, readily take it for granted that the situation is governed by a single set of rules or norms shared by all participants. The analysis focuses then on how those rules produce or fail to produce an orderly, coherent exchange. Models involving games and moves are often used to describe interactions. Despite whatever conflicts or systematic social differences might be in play, it is assumed that all participants are engaged in the same game and that the game is the same for all players. Often it is. But of course it often is not, as, for example, when speakers are from different classes or cultures, or one party is exercising authority and another is submitting to it or questioning it.
– Pratt, Mary Louise. Arts Of The Contact Zone [11]
This is the crux of how the Internet supercedes the role of the nation-state. The implicit agreement of its users on a set of shared rules implementing the means and methods of global communication, allows them to subvert intermediaries or become intermediaries themselves without the intercession of any state or government. By entering this massive, shared contact zone and leaving ‘meatspace’ behind, users bridge the distance between disparate jurisdictions, casting off the cultural mores and legal bindings of the physical world to join together in a single virtual ‘crowd.’ Canetti, in Crowds and Power, wrote that:
Only together can men free themselves from their burdens of distance; and this, precisely, is what happens in a crowd. During the discharge distinctions are thrown off and all feel equal. In that density, where there is scarcely any space between, and body presses against body, each man is as near the other as he is to himself; and an immense feeling of relief ensues. It is for the sake of this blessed moment, when no-one IS greater or better than another, that people become a crowd…
The open crowd is the true crowd, the crowd abandoning itself freely to its natural urge for growth. An open crowd has no clear feeling or idea of the size it may attain; it does not depend on a known building which it has to fill; its size is not determined; it wants to grow indefinitely and what it needs for this is more and more people. In this naked state, the crowd is at its most conspicuous, but, because it always disintegrates, it seems something outside the ordinary course of life and so is never taken quite seriously. Men might have gone on disregarding it if the enormous increase of population in modern times, and the rapid growth of cities, had not more and more often given rise to its formation.
The closed crowds of the past, of which more will be heard later, had turned into familiar institutions. The peculiar state of mind characteristic of their members seemed somewhat natural. They always met for a special purpose of a religious, festal or martial kind ; and this purpose seemed to sanctify their state. A man attending a sermon honestly believed that it was the sermon which mattered to him, and he would have felt astonished or even indignant had it been explained to him that the large number of listeners present gave him more satisfaction than the sermon itself All ceremonies and rules pertaining to such institutions are basically intent on capturing the crowd; they prefer a church-full secure to the whole world insecure. The regularity of church-going and the precise and familiar repetition of certain rites safeguard for the crowd something like a domesticated experience of itself These performances and their recurrence at fixed times supplant needs for something harsher and more violent. Such institutions might have proved adequate if the number of human beings had remained the same, but more and more people filled the towns and the accelerating increase in the growth of populations during the last few centuries continually provided fresh incitements to the formation of new and larger crowds. And nothing, not even the most experienced and subtle leadership, could have prevented them forming in such conditions.
– Canetti, Crowds and Power [12]
A single teeming crowd of billions of users is effectively a supranational entity. Any individual may be subject to the rules and regulations of a state, or bound to follow and obey the social and cultural mores they were born into. Ten of them may come together, located in different physical locations and legal jurisdictions, to agree on a new set of shared rules for commerce and communication; in doing so, they become something akin to a social club. Ten thousand become something closer to a tribe; ten million, a nation. With 5 billion active Internet users as of January 2024 and an average time spent online approaching 7 hours a day, an entirely new entity has come into being: an always-connected, teeming mass of living information, whose only shared bonds are their agreement to run the TCP/IP suite of software and the applications built on top of it. A single institution or set of institutions — state or otherwise — has no hope of imposing its will on this mass; it must expend all of its energy to work through the crowd, targeting chokepoints and intermediaries, filtering and blocking information and commerce, erecting its own chokepoints, and injecting its own information and knowledge in order to simply survive.
In an effort to better understand how conflicts between the diverse set of actors that makes up this teeming crowd – states, users, and Internet governance bodies — may play out in practice, we will now examine two episodes in the history of the Internet where users, Internet governing bodies and nation states have found themselves at odds. There is no final end state for conflicts such as these; each one leads to a new opening position for the next conflict to build upon. We will turn our attention first to the redirection of the Internet root name servers by Jon Postel in 1998, before moving on to the reaction of Internet companies and their customers to the attempted passage of the Stop Online Piracy Act (SOPA) and the PROTECT IP Act (PIPA) in 2012.
The Root Name Server Redirection Test
Recall from Chapter 4 that one of the original goals of the domain name system (DNS) was to remove the central point of control of management of names. From the inception of the DNS in 1983 until one day in late January of 1998, this goal appeared to have been theoretically achieved. It took fifteen years to discover the difference between theory and practice.
On that day — January 28, to be precise — Jon Postel, the Director of the Internet Assigned Numbers Authority, the researcher who first charged Paul Mockapetris with designing the DNS in the first place, sent an email out to eight of the twelve operators of the Internet root servers, instructing them to stop looking for updates to the root zone from the Network Solutions name server A.ROOT-SERVERS.NET (198.41.0.4) located in Herndon, Virginia, and to instead to look for updates from a server managed by the IANA at DNSROOT.IANA.ORG (198.32.1.98), located at the University of Southern California.
Postel’s role as the director (and sole employee) of the Internet Assigned Numbers Authority, and his longstanding role in the community of Internet engineers as the de facto ‘czar’ of Internet addresses and protocols — meant that he was perhaps the only person who could make this sort of request and expect to have it blindly followed. The faith that the other root name server operators had in him was total; he had commissioned the design of the DNS, after all, and he had offered direct assurances that this change was no more than a ‘test’:
“As a verification that such a transfer can be accomplished smoothly and without interruption to the operational service, a test is being performed to rearrange the flow of root zone information,” Postel wrote in his request to the operators.
– Gittlen, Sandra, ‘Taking The Wrong Root?’ [13]
In the aftermath of the event, Postel was accused of ‘hijacking the Internet’; critics called him a ‘fraud’ for explaining away his actions as a mere test, with one even going as far as calling for his arrest and conviction.
To understand the reaction to the redirection of the root nameservers, a bit of background is necessary. With the explosive increase in Internet usage in the mid 1990s came increased demand for domain names and specifically new top-level domains, as well as increased scrutiny of the existing domain name management process. In 1996, Postel proposed that the (IANA) authorize the creation of up to 150 new generic top-level domains, a massive expansion from the status quo. As the proposal was drafted and reworked, Postel in his role as IANA, in co-operation with the Internet Society, formed what they referred to as the “Internet Ad Hoc Committee” (IAHC), an umbrella entity where different aspiring stakeholders in the new TLD space could coordinate on their vision for the new TLD space. The committee drew from an eclectic pool of representatives from organizations such as the International Telecommunications Union (ITU), the International Trademark Association (INTA), and the World Intellectual Property Organization (WIPO).
The deliberations of the IAHC were met with significant resistance. The INTA representative, advocating for prudence (apparently), urged a significant reduction in the number of proposed top-level domains, and so the IAHC’s ambitious plan was reduced to a more modest proposal to add seven new top-level domains as a first step. This proposal also outlined a novel framework wherein businesses would transition into what we now know as “registrars,” responsible for interfacing with domain name registrants. Collectively, these registrars were to be organized under a nonprofit consortium known as the Council of Registrars, or CORE, which would manage a unified “registry” — the authoritative zone files for all new top-level domains. A newly conceived Policy Oversight Committee was slated to oversee pivotal policy decisions, including the timeline and criteria for introducing additional domains.
The hope was that, upon the expiration of existing agreement with Network Solutions for management of the DNS, CORE might also assume stewardship over existing top-level domains, further centralizing its influence. The proposal encountered vehement opposition. Critics decried it as an overreach by a select coterie of institutional powers, perceived as an insidious takeover of the domain name system. Detractors highlighted the composition of the IAHC, predominantly drawn from the Internet’s engineering echelons rather than its burgeoning commercial stakeholders, critiquing it as exclusive and opaque.
Amidst these debates, the IAHC’s approach was criticized for potentially instituting a centralized control antithetical to the foundational ethos of the Internet. Moreover, the European base of the proposed new organizations, coupled with a perceived lack of diplomatic finesse from its founders, further alienated support within the United States. At the same time, the undeniable reality that the extant governance model could not continue indefinitely was becoming apparent. The de facto role that Postel had played for decades as IANA czar was seen as insufficient for the evolving complexities of Internet governance.
In 1997, under the auspices of the Department of Commerce and the President’s senior Internet adviser Ira Magaziner, the U.S. government convened a comprehensive working group comprising myriad federal agencies to deliberate future governance structures. This group, amidst ongoing negotiations with Postel, grappled with the legitimacy of the U.S. government’s authority over Internet governance, given the existing contractual relationships with IANA and the National Science Foundation (NSF).
The result of their deliberations was the release of the “Green Paper” in January 1998 by the U.S. government, which conspicuously omitted any reference to the IAHC proposals, signaling their effective demise. The Green Paper proposed a new framework for governance of the DNS that did not incorporate the IAHC’s plans, one that made bold assertions about the justification for the U.S. government’s oversight of the root zone and the Domain Name System:
The Green Paper remains noteworthy, however, for its account of why DoC felt it had the power to make rules about the DNS at all, given the absence of direct congressional authorization. The Green Paper listed five sources of DoC’s statutory authority for this proposed rulemaking: (1) the general mission statement of DoC for the promotion of U.S. commerce; (2) DoC’s National Telecommunications and Information Administration’s (NTIA) power to coordinate telecommunications activities and assist in the formation of policies; (3) NTIA’s authority to “develop and set forth telecommunications policies”; 180 (4) NTIA’s power to conduct studies and make recommendations; and (5) NTIA’s authority to “issue such regulations as may be necessary to carry out” its functions.
— Froomkin, Wrong Turn In CyberSpace [14]
Two days before the Green Paper was officially released, Dr. Postel — who almost assuredly would have had some inkling about the contents of the document — sent the following email to root name server operators:
Hello. As the Internet develops there are transitions in the management arrangements.The time has come to take a small step in one of those transitions. At some point on down the road it will be appropriate for the root domain to be edited and published directly by the IANA. As a small step in this direction we would like to have the secondaries for the root domain pull the root zone (by zone transfer) directly from IANA’s own name server. This is “DNSROOT.IANA.ORG” with address 198.32.1.98. The data in this root zone will be an exact copy of the root zone currently available on the A.ROOT-SERVERS.NET machine. There is no change being made at this time in the policies or procedures for making changes to the root zone. This applies to the root zone only. If you provide secondary [sic] service for any other zones, including TLD zones, you should continue to obtain those zones in the way and from the sources you have been.
—jon
– Postel, Jonathan. Personal e-mail, sent 1/27/1998 [15]
Was the test a preliminary shot across the bow to warn the U.S. government and Ira Magaziner’s group to change their tack? We’ll never know; Dr. Postel didn’t keep extemporaneous notes about his motivations. It was certainly seen that way by Magaziner, who went so far as to threaten Postel with criminal charges if he didn’t immediately revert the test. (It is unclear under which statute Postel could have been charged.) In the wake of the test, the Green Paper was abandoned; intentionally or not, Postel had demonstrated that the gap between theories about government oversight and control of the root zone and the reality — that root name server operators were free to fetch and serve data from whoever and wherever they pleased — was simply too wide to bridge. Five months after the test, Magaziner’s group issued a non-binding ‘White Paper’, containing proposals and advice for how the functions of the IANA should be managed, largely inherited from the Green Paper that came before it. Instead of dismissing the earlier IAHC proposal out of hand, the White Paper acknowledged the obvious — that the IAHC plan was “not able to overcome” criticism of both its substance and the process by which it had been developed”. Instead of laying down concrete requirements, the White Paper merely offered recommendations. These recommendations included the formation of a “new, not-for-profit corporation formed by private sector Internet stakeholders to administer policy for the Internet name and address system.” This corporation would absorb any responsibilities, real or imagined, that the U.S. government had over the administration of names and numbers on the Internet. It came to be known as the Internet Corporation for Assigned Names and Numbers, or ICANN.
The White Paper did not speak precisely to how the new corporation it described would be formed. It suggested that if the new entity were formed by “private sector Internet stakeholders,” the U.S. government was prepared to recognize it by entering into agreements with it, seeking international support for it, and ensuring that it had appropriate access to databases and software controlled by NSI. In October 1998, after a series of negotiations between IANA and NSI — and more wide-ranging consultations on the interim board’s composition with the U.S. government, a variety of foreign governments, Jon Postel’s lawyer (a Jones, Day partner named Joe Sims), IBM, and others — Dr. Postel transmitted to the Department of Commerce documents reflecting what he described as “the consensus judgment of the global Internet community as to how to form a corporation that will include the IANA function.” Those documents included the articles of incorporation of the new entity, which had already been incorporated in California as the Internet Corporation for Assigned Names and Numbers, or ICANN, biographies of its initial board of directors, and a set of proposed bylaws.
– Weinberg, Jonathan. ICANN And The Problem Of Legitimacy [16]
A mere two weeks after these documents were transmitted, Dr. Jon Postel passed away from complications of open heart surgery at the age of 55. The Internet had lost its assigned numbers authority.
A long time ago, in a network, far far away, a great adventure took place!
Out of the chaos of new ideas for communication, the experiments, the tentative designs, and crucible of testing, there emerged a cornucopia of networks. Beginning with the ARPANET, an endless stream of networks evolved, and ultimately were interlinked to become the Internet. Someone had to keep track of all the protocols, the identifiers, networks and addresses and ultimately the names of all the things in the networked universe. And someone had to keep track of all the information that erupted with volcanic force from the intensity of the debates and discussions and endless invention that has continued unabated for 30 years. That someone was Jonathan B. Postel, our Internet Assigned Numbers Authority, friend, engineer, confidant, leader, icon, and now, first of the giants to depart from our midst.
Jon, our beloved IANA, is gone… When we needed to keep track of all the hosts and protocol identifiers, Jon volunteered to be the Numbers Czar and later the IANA once the Internet was in place.
— RFC 2468, “I REMEMBER IANA” [17]
In his absence, Jon’s lawyer worked to ensure that ICANN was brought online exactly as Jon had envisioned it and in line with the recommendations made in the whitepaper. What could perhaps have been a tense and delicate transition of power was instead a somber and muted affair; ICANN seamlessly took over the role that IANA had played, and instead of a single “protocol czar”, a multi-stakeholder nonprofit corporation was charged with managing the names and numbers that served as the foundation of the Internet.
The root server redirection test is the perfect example of the sort of brinksmanship that can occur between government officials that presume themselves to be somehow ‘in control’ of the core functionality of the Internet and the operators that are actually charged with managing the underlying infrastructure. The common vulgar idiom “Money talks, bullshit walks” is meant to convey the idea that “attempting to accomplish a goal by demonstrating possession of material resources will succeed, while attempting to accomplish the same goal through mere rhetoric will fail.” In this case, when push came to shove, Dr. Postel’s actions made it clear that when protocol participants talk, government bullshit has no choice but to walk.
The SOPA and PIPA Blackout
Imagine that you are a global hegemonic power, intent on coercing sovereign nations into obeying your every wish and command. You have carrots in the form of money and resources, and sticks in the form of troops and armaments. With these tools you should be able to impose your will upon any actor you wish, as long as you have the most carrots and the most sticks — that is, unless you somehow find yourself on some new terra incognita, one where your carrots aren’t the recognized coin of the land and your sticks can’t quite reach as far as they do back home. From the point of view of the State, the Internet was (and in many ways still is) exactly such a place. Compelling online entities in foreign jurisdictions to obey the will of the hegemon is an exercise in futility, and often an unnecessary one; better to go after physical, tangible chokepoints in local or friendly jurisdictions and avoid the unexplored intangible terrain altogether.
The Stop Online Piracy Act and the PROTECT IP Act were two bills introduced in the United States House of Representatives and the Senate in late 2011. Intended to serve as mechanisms to enable the enforcement of U.S. laws against websites operated beyond the reach of the U.S. government, SOPA and PIPA, as they came to be known, empowered authorities to cut off U.S.-based sources of funding for websites in foreign jurisdictions that offered material infringing on copyright or violating intellectual property rights.
Unable to target these websites directly, authorities would instead be empowered to go after the advertising networks they used for revenue, the payment processors they used to receive funds, and even the search engines that guided users to them. Perhaps most alarmingly, the bills included provisions that would force DNS server operators to filter requests made to them in order to restrict access to these websites, transforming them from neutral directories of addresses into tools for moderating content, fundamentally altering the traditional role they had played in the day-to-day operation of the Internet.
Supporters of SOPA and PIPA included the Motion Picture Association of America, the Recording Industry Association of America, and the U.S. Chamber of Commerce, amongst other groups. They had supported a previous bill known as COICA, the Combating Online Infringement and Counterfeits Act, which had been introduced in 2010 and passed the Senate Judiciary Committee unopposed but never made it to a vote on the Senate floor due to strenuous opposition. COICA was rewritten the following year to become PIPA; SOPA was the House’s version of PIPA, introduced six months after PIPA made it to the Senate floor.
The time spent on rewriting and drafting the newer versions of these bills allowed opponents of the bills to plan out and mount an effective campaign against them. Shortly after the first markup hearing by the House Judiciary Committee for SOPA, on November 16, 2011, prominent website operators – most notably Tumblr and Reddit — began displaying prominently placed black banners with the words ‘STOP CENSORSHIP’, informing their users of the dangers of the legislation and imploring them to contact their local representatives to speak out against the bills. The web site https://americancensorship.org was set up by a group of interested third parties, including the Electronic Frontier Foundation, the Free Software Foundation, the Foundation for Participatory Politics, and Mozilla. It served as a focal point for website operators interested in participating in the protests; it provided descriptions of the chilling effects that SOPA and PIPA were expected to have on free speech on the Internet, provided instructions to website operators on how to join the protests and add ‘censored’ logos to their sites, and displayed statistics tracking how many members of Congress had been emailed or received phone calls from Internet users voicing their concerns.
In retrospect, what became known as ‘American Censorship Day’ was the first shot in the battle against SOPA and PIPA, and the element of surprise was instrumental in ensuring that the adversaries of the bill scored a resounding tactical victory. It’s surprisingly easy to emerge from a conflict victorious when your opponent doesn’t actually show up to battle. The lobbyists and lawmakers that supported the passage of SOPA and PIPA appeared to have made the tactical error of dismissing the masses of website operators rising against them as being of little consequence, and misunderstood the events of November 16 as being a self-contained event instead of an opening salvo. There were no follow-on actions planned after American Censorship Day, no future protests to thwart; perhaps the best tactic was to simply dismiss the events of November 16, allow time for the media firestorm to die down, and move forward with efforts to pass the legislation.
On December 22, 2011, a user posted on the Internet message board website Reddit about his intention to move over fifty of his domains away from GoDaddy due to their support of SOPA and PIPA, and suggesting that December 29 be declared ‘Move Your Domain Day’. One of the most controversial provisions of SOPA was a mechanism for rights holders to compel DNS service providers to block access to ‘infringing’ sites via a court order. Since most of the root domain name servers as well as the most popular domain registrars were under the jurisdiction of the United States, this proposal would have effectively placed the global DNS under the jurisdiction of the U.S. Department of Justice. This was naturally opposed by almost every popular domain name registrar; from the point of view of a DNS service provider, it was far more preferable to push for legal responsibility for copyright infringement to be regulated at the endpoints from which content was being served, and to leave the domain name system untouched and blissfully unaware of what sort of content might be lurking behind the addresses that names were resolved to.
GoDaddy was a notable exception to this opposition. On a list of over 100 companies released by the House Judiciary Committee of companies supporting SOPA, GoDaddy was the only domain registrar to be found. Their chief counsel offered testimony in front of that same committee explaining that GoDaddy fully supported the passage of SOPA and saw it as vital for “preserving American ingenuity” and downplaying the concerns raised by SOPA’s opponents as “utterly unfathomable”. On the eve of its passage, GoDaddy went as far as to submit an op-ed article to popular news website Politico explaining in detail their reasons for supporting the bill.
What started as a Reddit post quickly turned into a spontaneous mass movement to boycott GoDaddy. First the post began to trend on Reddit’s main page as more users chimed in, along with former GoDaddy employees offering advice and encouragement. Within 24 hours, the post had caught the attention of Jimmy Wales, the co-founder of Wikipedia, who announced on Twitter his intention to move Wikipedia’s domain names away from GoDaddy in support of the boycott. This in turn attracted attention from the mainstream media, and what had started as a trending Reddit post quickly turned into a breaking story.
With Wales’s endorsement, GoDaddy’s competitors sensed blood in the water and quickly pounced. The DNS registrar Namecheap began a marketing campaign targeted at disaffected GoDaddy customers, decrying GoDaddy’s pro-SOPA stance and offering discounts for GoDaddy customers to move their domains to Namecheap. So many of GoDaddy’s customers took them up on their offer that the mechanism GoDaddy used to confirm the transfer names between registrars was automatically throttled —‘rate limited’— as the rate of domains being moved was high enough that GoDaddy’s software interpreted it as being a malicious attack instead of an unprecedented mass boycott. Namecheap in turn presented this as proof that GoDaddy was blocking attempts to transfer domains — framing it as yet more evidence that GoDaddy was a bad actor that needed to be punished — and managed to wring another news cycle out of the drama, with back-and-forth accusatory blog posts being passed between the two companies.
As tens of thousands of domains were transferred to other registrars in the week before ‘Move Your Domain Day’ on December 29, GoDaddy caved in the face of the boycott, at first announcing that it was no longer actively backing SOPA, and then, as the flood of domain transfers continued, stating clearly that they opposed the Protect IP act and its provisions for DNS filtering. It was a massive achievement for the opponents of SOPA and PIPA; forcing perhaps the largest commercial enterprise supporting the bills to turn heel and reverse their position in a matter of weeks transformed a hodgepodge group of malcontents on Reddit into a cohesive crowd with an impressive victory under their belt. Whereas American Censorship Day could be easily dismissed by SOPA proponents as noise for the sake of noise, with no actual impact on the bill’s chances of becoming law, ‘Move Your Domain Day’ in comparison forced GoDaddy to pay a real, tangible cost and make an embarrassing about-face — all the while pushing the controversy over the bills to the front pages of the mainstream media.
The overwhelming victory not only emboldened and encouraged the anti-SOPA crowd; it set the stage for the events of January 18, 2012. Over 115,000 websites — most prominent amongst them Google, Reddit, Flickr and Wikipedia — engaged in protests against SOPA and PIPA. Some sites went offline partially, others completely blacking out their content, simulating what they believed could happen if the bills were passed and they were forced to implement their content-filtering mechanisms; simultaneously, real-world demonstrations and protests were held in San Francisco, New York City, and Seattle; and millions of voters signed petitions and contacted lawmakers to express their opposition to the bill. The protests of January 18 — from then on referred to as the ‘Web Blackout’ or ‘SOPA Blackout’— landed a fatal blow against both bills, with support for their passage evaporating seemingly overnight:
The impact of the coordinated action was generally considered to be significant. Yochai Benkler of the Berkman Center for Internet & Society stated that the January 18 blackout was “a very strong public demonstration to suggest that what historically was seen as a technical system of rules that only influences the content industry has become something more,” further adding “You’ve got millions of citizens who care enough to act. That’s not trivial.” California House member Darrell Issa called the collective effort an unprecedented means for upsetting a backroom lobbying effort, and the immediate political efficacy of the widespread online protest was characterised in terms of a sleeping giant having awakened and of a new player being in town. One Silicon Valley lobbyist said the content industry had “a lot to learn,” noting that they don’t have grassroots support: “There are no Facebook pages to call your congressman to support PIPA and SOPA.”[…]
– Wikipedia, Retrieved from https://en.wikipedia.org/wiki/Protests_against_SOPA_and_PIPA [18]
By one account, the shift in stated positions on SOPA/PIPA by members of Congress had gone overnight from 80 for and 31 against to 65 for and 101 against[…]
In January 2022, Tiffany Cheng of Techdirt wrote that “The SOPA Blackout not only killed the bill in 2012, but shook Congress so profoundly that no significant copyright legislation has been introduced in the ten years since. Because the Blackout achieved so much progress against the political order in a matter of weeks, this moment in history rewrote what we collectively think is possible in the political realm; in particular among the political set, even though triumphs of this proportion remain elusive, and power is even more entrenched.”
– Wikipedia, Retrieved from https://en.wikipedia.org/wiki/Protests_against_SOPA_and_PIPA [18]
The rejection of SOPA and PIPA marked a watershed moment for Internet activists and the tussle between Internet users and those who wished to govern them. Large commercial interests, lobbyists and lawmakers were all vehemently in favor of the bills and treated their passage as almost a foregone conclusion. No single actor opposing the legislation — neither Wikipedia, nor Namecheap, nor Mozilla, nor the EFF — could have hoped to stand in the way of these interests; instead, it was a loosely coordinated crowd of Internet commenters egging each other on that was able to upend the existing political order, overwhelming their opponents with their collective strength. The proponents of the bills were effectively a static target; their opponents, a guerrilla army, harassing and retreating at will, inflicting a thousand cuts in a war of attrition, insisting that the protocols they had come to rely on be left alone.
What Is To Be Done
I have resisted until now the temptation of putting forth some sort of grand theory of protocols or suggesting a way forward from here. This text deals with protocol development, adoption, and implementation as historical facts, eschewing the theoretical. If your current experience of the Internet works well enough for you, my only suggestion is to keep at it. I believe I have made the case that the Internet comes with its own set of rules; that agreeing to these rules is voluntary, in the sense that no one is forcing you to browse the World Wide Web, but compulsory if you wish to communicate with others who are already following these rules; that if you find yourself in need of ‘more’— in need of more knowledge to consume, in need of more freedom to communicate and disseminate speech, in need of more freedom to transact, to buy and sell — and find yourself blocked by some unseen intermediary, whether it be an authoritarian dictatorship or an unfeeling corporate monolith, then more is yours for the taking. You can use Tor to search for and consume content without a state-owned telecommunications provider blocking you; Bitcoin to engage in transactions outside of the state-sanctioned banking system; stand up your own website if you are banned from sharing prohibited information on macro-intermediary platforms like Instagram and Twitter, or store your data on your own personal server so that the likes of Netflix and Amazon can’t delete your favorite television show out from underneath you. By the same token, if you wish to take on less responsibility — to consume only ‘approved’ content, engage in ‘approved’ commerce, post only ‘approved’ opinions, etc. in exchange for the convenience and benefits that comes with lesser responsibilities, you are certainly free to do that as well, allowing some other party to take on the burden of running, operating and understanding the software that enables you to do so, as long as you are willing to pay their asking price, which is subject to change at their whim, in the same way that a toll operator may raise the price for transiting a road whenever you choose to travel.
I also believe that a few dozen users agreeing to follow this new set of rules to communicate and form commercial or social relationships is an interesting experiment. A few billion users doing the same utterly upends the world our ancestors knew in ways that we are simply not equipped to understand. The ultimate outcome of this re-ordering of the global system of shared rules will necessarily lead to a world with a different relationship between citizen and sovereign than the one we currently inhabit. (Actually, in my opinion, it already has; what I have been describing this whole time is not a distant future but our current Internet reality.) New actors – such as Google, Amazon, Netflix, etc. – will rise up to compete with states in mediating commerce and communications and the spread of information and knowledge; some states will compete with them, some cooperate with them, some will wither and some will thrive. New personalities will rise to compete with their pre-Internet contemporaries; Internet celebrities and prophets will compete with kings and popes for the hearts and minds of Internet commoners. The new rules that govern this system may lead to a structure that feels chaotic and strange, but insofar as the order that it supports is made up of human beings, I expect it to be more familiar than might be expected. After all, the existence of entirely new and novel methods of communication and connection does not necessarily lead to the creation of entirely new and novel human beings or new and novel human desires. We will have chiefs, tribes and fiefdoms connected to each other by protocols, simply because there is a shared desire for them to exist.
As every interaction that can be had over the Internet is destined to eventually find its way there, I believe the offline state will eventually find itself in the business of governing over only what cannot take place online. Its primary task will be safeguarding the infrastructure of the Internet, from the data centers that power it, to the semiconductor fabrication plants that fill those data centers with computers, to the undersea submarine cables and orbital satellites that connect these data centers to users in their homes. This is the penultimate tussle between the state and the user, with the state wielding control of the physical substrate of communications versus the users responsible for the content and delivery of the communications themselves. It is no accident that encryption, particularly end-to-end encryption (E2EE) in messaging protocols, and the overlay networks (Tor, BitTorrent and Bitcoin) are the Internet technologies most frequently targeted by lawmakers and intelligence agencies, typically under the guise of fighting child pornography and terrorist activity, as often and as loudly as they are. These tools are anathema to the state; an Internet that is encrypted from end-to-end is highly resistant to interference, and an overlay network that is open to all participants is highly resistant to intermediation.
I find myself unable to imagine an Internet where encryption is ubiquitous and the use of overlay networks is common, though it seems we are hurtling ever closer to that reality. As we’ve seen, protocol upgrades can take decades to gain traction, and I expect user adoption of these sorts of changes will take a similar amount of time, in fits and starts. I will force myself to refrain from pointing to any one overlay network or protocol upgrade that I expect to see widespread adoption; I think it’s best not to try to predict the future, particularly as engineers and users have wildly differing ideas of what the best solution to any one problem might be. What has convinced me that we are witnessing the next step in the evolution of Internet protocols is not a demonstration of any one tool, but instead the number of different efforts to develop software with end-to-end encryption and peer-to-peer overlays being worked on by completely disconnected groups of software developers with different motivations and incentives. There hasn’t been any one catalyst that I can point to that has kicked off these efforts. If it is true to say that the lifespan of a generation is somewhere around two decades, then we can say that the DNS was made commercially available roughly a generation after ARPANET was first launched; the draft specifications of IPv6 were released a generation after NCP went live; and the migration to HTTPS from HTTP began after a generation of the plain-text web. Perhaps that’s what it takes — a new generation to offer new alternatives to the Internet they grew up with, new solutions to problems that older generations mistook as features.
The reason I find myself unable to imagine an Internet where these changes are adopted is the same reason that an Internet user in the 1990s would have been unable to imagine the Netflix or Facebook of today. To ‘watch television on your computer’ was possible back then; computer graphics cards were sold with television tuners and perfectly capable of displaying broadcast and cable television on the screen of a CRT. Social media, the idea of sharing and communicating the most mundane details of your life with others doing the same was also quite possible and happened frequently, but through endless ‘chain emails’ and in IRC chat rooms. I don’t know what’s in store for this latest iteration of the Internet superstructure we are lurching towards, one where a critical mass of users — perhaps not a majority, but certainly an intolerant minority — decide that end-to-end encryption and peer-to-peer overlay networks are projects worth investing their time and resources into. What I expect is that, once those users fill in these final chinks in the armor of the Internet protocol suite and remove the last locus of control, we will find the users in the middle of the tussle between those that govern the Internet and those whose governing is undone by the Internet will have upended the game board entirely. What comes next for the relationship between protocol and nation-state, between users opting into agreements that cannot be intercepted, filtered, or manipulated by outside actors, compared to our current online existence is difficult for me to imagine. My hope is that it will be, compared to what came before it – and forgive me for closing the text with its title!— something more profound.
Epilogue
I’d like to thank my wife Ivelina, and my daughter Anara, for their patience, understanding and inspiration. This book is dedicated to them.
I was inspired to write this book by reading two others; the first, ‘The Elements Of Networking Style’ by Michael Padlipsky — a cantankerous Internet engineer in the 1980s with strong opinions and sharp wit — was perhaps the most niche text I’ve ever read. Padlipsky was no author, and he had nothing that could be called an audience, but he did have something to say, and that was enough to get him to pick up a pen. The other, Basil J. Moore’s ‘The Horizontalists and Verticalists: The Macroeconomics Of Credit Money’, contains within 2 of its 441 pages the most edifying description of how the supply of money is endogenously credit-driven and demand-determined; it was more succinct and insightful than dozens of economics textbooks that I had force-fed myself in search of similar insights, and I hope somewhere in this text someone may feel the same sort of lightning bolt.
I am indebted to the works of Janet Abbate and Laure DeNardis, who were instrumental in providing the necessary background research for producing this text. This work was produced with theirs in mind, as well as with dozens of RFC documents, oral histories and first-person accounts of protocol development; a full list of these is documented in the bibliography.
I would like to thank James O’Beirne, Saifedean Ammous, Michael Goldstein, Pierre Rochard, and Katherine Champagne, for either in-person discussions or interactions on social media that were instrumental in shaping my thoughts in the conclusion in the final chapter.
Special thanks to David Bowie for the title, prologue and parting words of this text.
Finally: the idea of ‘investing in protocols’ somehow came in vogue as I first started the work of researching this text. If you’ve read it in the hopes of gaining some insight in how to do so, I’m afraid you’re out of luck. Not only do I not have any advice to offer in that regard, I am thoroughly convinced that any attempt to ‘invest’ in a protocol would be either doomed to failure or would lead to success only if by complete accident. I could have told you well before the demise of the walled garden information services like AOL and Prodigy that they were destined to be replaced by the Internet, TCP/IP and the World Wide Web — but for all the people that bought AOL stock at the bottom and sold it at the top, that would have been terrible guidance for an investment. (Little matter, anyway: most of the ‘protocols’ people seem to be ‘investing’ in are HTTPS in a funny hat.)
–
Bibliography
Prologue:
- Bowie, D. (1999). Interview with Jeremy Paxman. BBC Newsnight. Retrieved from https://www.youtube.com/watch?v=FiK7s_0tGsg
- Licklider, J. C. R. (1963). Memo to Members and Affiliates of the Intergalactic Computer Network. Retrieved from https://www.thekurzweillibrary.com/memorandum-for-members-and-affiliates-of-the-intergalactic-computer-network
- Cerf, V. as quoted in Jones, Steve (2003). Encyclopedia of New Media: An Essential Reference to Communication and Technology. 1st ed. Thousand Oaks, London, New Delhi: Sage Publications
Chapter 1:
- Baran, P. (1962). On Distributed Communication Networks. Retrieved from https://www.rand.org/content/dam/rand/pubs/papers/2005/P2626.pdf
- Licklider, J. C. R. (1963). Memo to Members and Affiliates of the Intergalactic Computer Network. Retrieved from https://www.thekurzweillibrary.com/memorandum-for-members-and-affiliates-of-the-intergalactic-computer-network
- Abbate, J. (1999). Inventing the Internet. Cambridge, MA: MIT Press. (pp. 48-49).
- Dickson, P. A. (1968). ARPA network will represent integration on a large scale. Electronics, 30, (pp. 131-134).
- Scantlebury, R., & Bartlett, K. (1967). A Protocol for Use in the NPL Data Communications Network.
- Roberts, L. G. (1967, January). Multiple computer networks and intercomputer communication. In Proceedings of the first ACM symposium on Operating System Principles (pp. 3-1).
- Postel, J. & Reynolds, J. (1987). RFC 1000: The Request For Comments Reference Guide. Request for Comments. https://www.rfc-editor.org/rfc/rfc1000.txt
- Naughton, J. (2015). A brief history of the future. Weidenfeld & Nicolson.
- Hauben, M., & Hauben, R. (1997). Netizens: On the history and impact of Usenet and the Internet. IEEE Computer Society Press.
Chapter 2:
- Russell, A. L. (2014). Open standards and the digital age. Cambridge University Press.
- Systems Network Architecture (2024, March 1st). In Wikipedia. https://en.wikipedia.org/wiki/Systems_Network_Architecture
- Shapiro, C., & Varian, H. R. (1999). Information rules: A strategic guide to the network economy. Harvard Business Press.
- Varian, H. R., Farrell, J., & Shapiro, C. (2004). The economics of information technology: An introduction. Cambridge University Press.
- Clark, D. D., Wroclawski, J., Sollins, K. R., & Braden, R. (2005). Tussle in cyberspace: defining tomorrow’s internet. IEEE/ACM transactions on networking, 13(3), 462-475.
- Day, J. (2007). Patterns In Network Architecture. Pearson Education India.
Chapter 3:
- Russell, A. L. (2014). Open standards and the digital age. Cambridge University Press.
- Callon, R. (1991). RFC 1273: The Recommendation for the IP Next Generation Protocol. Request for Comments. https://www.rfc-editor.org/rfc/rfc1273.txt
- Chapin, L., as quoted in Messmer, E. (1992). Internet Architect Gives Long-Term View. Network World, (pp. 9).
- Clark, D. (1992). A Cloudy Crystal Ball — Visions of the Future. Retrieved on 2024-05-13. (Presentation given at the 24th Internet Engineering Task Force Meeting.)
- Kuerbis, B., & Mueller, M. (2020). The hidden standards war: economic factors affecting IPv6 deployment. Digital Policy, Regulation and Governance, 22(4), 333-361.
Chapter 4:
- Shoch, J. F. (1978). “A Note on Inter-Network Naming, Addressing, and Routing.” Computer Networks, 2(3), 243-247.
- Mockapetris, P. V. (November 1983). “RFC 882: Domain Names - Concepts and Facilities.” Request for Comments. https://www.rfc-editor.org/rfc/rfc882.txt
- Larson, M., & Liu, C. (2001). DNS on Windows 2000. O’Reilly Media, Inc. [Excerpt on HOSTS.TXT challenges]. Retrieved from https://www.oreilly.com/library/view/dns-on-windows/0596005628/ch01s02s01.html
- Mockapetris, P., & Dunlap, K. J. (1988, August). Development of the domain name system. In Symposium proceedings on Communications architectures and protocols (pp. 123-133).
- Schelling, T. C. (1958). “The Strategy of Conflict Prospectus for a Reorientation of Game Theory.” Journal of Conflict Resolution, 2(3), 203-260.
- Vixie, P. (October 2019). “The DNS Wars: Episode IV - A New Hope.” Presentation at NANOG 77. https://pc.nanog.org/static/published/meetings/NANOG77/2033/20191028_Vixie_Keynote_Dns_Wars__v1.pdf
- Szabo, N. (1998). “Secure Property Titles with Owner Authority.” Retrieved from https://nakamotoinstitute.org/secure-property-titles/
- Collier, K. (May 19, 2015). “The Pirate Bay lives on despite Swedish Domain Seizure.” The Daily Dot. Retrieved from https://www.dailydot.com/debug/the-pirate-bay-domain-sweden-seized/
Chapter 5:
- Postel, J. (September 1981). “RFC 823: The DARPA Internet Gateway.” Request for Comments. https://www.rfc-editor.org/rfc/rfc823.txt
- Seamonson, L., & Rosen, E. (April 1984). “RFC 904: Exterior Gateway Protocol Formal Specification.” Request for Comments. https://www.rfc-editor.org/rfc/rfc904.txt
- Van Beijnum, I. (June 2009?). “GGP, EGP and 25 years of BGP: a brief history of internet routing”. Retrieved from https://www.routerfreak.com/ggp-egp-and-25-years-of-bgp-a-brief-history-of-internet-routing/
- Kesan, J. P., & Shah, R. C. (2001). “Fool Us Once Shame on You — Fool Us Twice Shame on Us: What We Can Learn from the Privatizations of the Internet Backbone Network and the Domain Name System.” Washington University Journal of Law & Policy, 7, 55-108. Retrieved from https://openscholarship.wustl.edu/cgi/viewcontent.cgi?article=1399&context=law_lawreview
- Clark, D. D. (2018). Designing an Internet. MIT Press.
- Knieps, G. (2015). Network Economics. Heidelberg, Springer.
- McQuistin, S., and Perkins, C. (2015) Reinterpreting the transport protocol stack to embrace ossification. In: IAB Workshop on Stack Evolution in a Middlebox Internet (SEMI), Zurich, Switzerland, 26-27 Jan 2015.
- Handley, M. (2014). “Why The Internet Only Just Works.” BT Technology Journal, 22(3), 119-129. https://link.springer.com/article/10.1023/B:BTTJ.0000047600.71482.fb
Chapter 6:
- Usenet (2024, March 1st). In Wikipedia. https://en.wikipedia.org/wiki/Usenet
- PCMag Staff . (2008, July 31st). “R.I.P Usenet: 1980-2008.” PCMag. Retrieved from https://www.pcmag.com/archive/rip-usenet-1980-2008-230383
- Alberti, B. (2011). “Internet Gopher: A Bridge To The Web.” MinnPost. Retrieved from https://www.minnpost.com/business/2016/08/rise-and-fall-gopher-protocol/ (Note: URL date is 2016, text is 2011 - this matches your doc)
- Gihring, T. (2016). “The rise and fall of the Gopher protocol.” Internet Archive. https://archive.org/details/internet_gopher_bridge_to_the_web/page/n1/mode/2up
- Frana, P. (2001). “An Interview With Mark McCahill.” Oral History Software History Project of the National Science Foundation. Retrieved from https://conservancy.umn.edu/server/api/core/bitstreams/cf1361cf-b684-410b-8a0e-5841c27ae159/content
- Lee, C. (1999). “Where Have all the Gophers Gone? Why the Web beat Gopher in the Battle for Protocol Mind Share.” University Of Michigan. Retreived from https://ils.unc.edu/callee/gopherpaper.htm
- Lopp, J. (2015). “The Death of Decentralized Email.” Lopp.net. Retrieved from https://blog.lopp.net/death-of-decentralized-email/
Chapter 7:
- Berners-Lee, T. (1989). Information Management: A Proposal. CERN. Retrieved from http://cds.cern.ch/record/369245/files/dd-89-001.pdf
- Bory, P. (2020). The internet myth: From the internet imaginary to network ideologies (p. 169). University of Westminster Press.
- Tusikov, N. (2017). Chokepoints: Global private regulation on the Internet. Univ of California Press.
- Deleuze, G. (1992). Postscript on the Societies of Control. October, 59, 3-7.
- Dixon-Román, E. (2020). Control Societies @ 30: Technopolitical Forces and Ontologies of Difference. Social Text Journal.
- Google. (March 2024). HTTPS Encryption on the Web. Google Transparency Report. Retrieved from https://transparencyreport.google.com/https/overview?hl=en
Chapter 8:
- Tarkoma, S. (2010). Overlay Networks: Toward Information Networking. Auerbach Publications.
- Syverson, P., Dingledine, R., & Mathewson, N. (2004). Tor: The Second-Generation Onion Router. Retrieved from https://svn-archive.torproject.org/svn/projects/design-paper/tor-design.html
- Allen-Robertson, J. (2013). Digital Culture Industry: A History Of Digital Distribution. Springer.
Chapter 9:
- Nakamoto, S. (2008). Bitcoin: A Peer-to-Peer Electronic Cash System. Retrieved from https://bitcoin.org/bitcoin.pdf
- Bonneau, Joseph. (2015). How Long Does It Take for a Bitcoin Transaction to Be Confirmed?. Retrieved from https://www.coincenter.org/education/crypto-regulation-faq/how-long-does-it-take-for-a-bitcoin-transaction-to-be-confirmed/
- Harding, David. (2020). Miner Fees. Retrieved from https://en.bitcoin.it/wiki/Miner_fees
- Bier, J. (2021) The Blocksize War: The Battle Over Who Controls Bitcoin’s Protocol Rules. Independently published.
Chapter 10:
- Ward, Colin. (2004). Anarchism: A Very Short Introduction. Oxford University Press.
- “What Does ICANN Do” (2024). Retreived from https://www.icann.org/resources/pages/welcome-2012-02-25-en
- “Network Protocols And Standards, CCNA Routing and Switching Study Course 1”. (2024) Cisco Press. Retrieved from http://cisco.num.edu.mn/CCNA_R&S1/course/module3/3.2.3.2/3.2.3.2.html.
- W3C. (March 1, 2024)). World Wide Web Consortium (W3C). Retrieved from https://www.w3.org/
- Matthewson, N. (Feb 8, 2015). “Tor design proposals: how we make changes to our protocol”. Retrieved from https://blog.torproject.org/tor-design-proposals-how-we-make-changes-our-protocol/.
- Harrison, D. (Jan 10, 2008). “Index of BitTorrent Enhancement Proposals”. Retrieved from https://www.bittorrent.org/beps/bep_0000.html.
- Rochard, P. (July 8, 2018). “Bitcoin Governance”. Retrieved from https://pierre-rochard.medium.com/bitcoin-governance-37e86299470f.
- Clark, D. D., Wroclawski, J., Sollins, K. R., & Braden, R. (2005). Tussle in cyberspace: defining tomorrow’s internet. IEEE/ACM transactions on networking, 13(3), 462-475.
- Willke. Helmut, as quoted in Jessop, B. (2015). The State: Past, present, future. John Wiley & Son
- Perritt Jr., H. H. (1998). The Internet as a Threat to Sovereignty? Thoughts on the Internet’s Role in Strengthening National and Global Governance. John Marshall Journal of Computer & Information Law, 16(2).
- Pratt, M. L. (2012). Arts of the contact zone. In Negotiating academic literacies (pp. 171-185). Routledge.
- Canetti, E. (1984). Crowds and power. Macmillan.
- Gittlen, S. (1998). Taking the wrong root. Network World. Com. Retrieved from https://web.archive.org/web/20120308200824/http://www.networkworld.com/news/0204postel.html
- Froomkin, A. M. (2000). Wrong Turn in Cyberspace: Using ICANN to Route around the APA and the Constitution. Duke LJ, 50, 17.
- Postel, Jon. Personal communication, January 27, 1998.
- Weinberg, J. (2000). ICANN and the Problem of Legitimacy. Duke LJ, 50, 187.
- Cerf, V. (October 1998). “RFC 2468: I remember IANA” Request for Comments. https://www.rfc-editor.org/rfc/rfc2468.txt
- Protests Against SOPA and PIPA. (March 21st, 2024). In Wikipedia. Retrieved from https://en.wikipedia.org/wiki/Protests_against_SOPA_and_PIPA