The Problem.
In 1974’s East German hit single ‘Du hast den Farbfilm vergessen’ (you forgot the color film), singer Nina Hagen loudly complains about her holiday being ruined by her boyfriend, Michael, who forgot to bring the eponymous color film. The lyrics go,
Now nobody believes us how nice it was here, haha, haha
Nun glaubt uns kein mensch, wie schön’s hier war, haha, haha.Of course, as you’ve probably surmised, the song isn’t really about the color film. It’s a thinly veiled satire aimed at the communist regime by the artist, toying with everything from the regime’s attempts to pass its citizens’ lives as being nice to Western audiences to the drabness of communist Germany.
As with other jetsam in the stream of the public’s consciousness, the song came and went, with few people remembering its context and meaning. I remember listening to it, thanks to a German history lesson. I found Nina Hagen’s performance so enchanting that I played it on loop for a week before promptly forgetting it.
It would have stayed in the drawers of history, gingerly enduring an occasional prod from a wandering history student, if it weren’t thrust back into the limelight in December 2021.
I wasn’t the only one obsessed with the song in my nascent youth. I and many others shared the experience with a post-reunification German chancellor whom you might be familiar with. Except, for her, it was contemporary. Experienced as a nouveau 20-year-old studying physics at Leipzig University. It must have influenced her deeply, for she chose it as one of three songs that would play her out at the end of her 16-year career as chancellor1.
I am not German. Nor am I fluent in the language — a regrettable lapse that I am working to correct. :)
My infatuation with the song was acontextual. I was a passerby pulled in by M. Hagen’s charm; the lyrics didn’t quite land for me as they would for that young student living under the communist regime. Nevertheless, it must have left a deep mark — moving the famously stoic woman to tears decades later.
And I don’t understand why. What did she hear that I didn’t?
Even if someone from that era sits us down and patiently walks us through line-by-line. We will never be able to recreate the experience that moved Merkel to tears. The information contained within their brain — the thought pattern — that’s necessary for constructing and appreciating the necessary context isn’t communicable.
We will never be able to truly understand or feel how the satire hit under the weight of an oppressive communist regime. Their context is transient in the stream of time. Forever just out of reach, even when exquisitely preserved.
This regrettable process of knowledge loss becomes something more acute and dangerous when it comes to our technology. And it shows up in subtle and surprising ways. Here’s one example of large-scale knowledge loss; what if I told you the picture you have in your mind of one of the most famous machines ever made was wrong?
A machine so famous that you’ve seen in film and stills multiple times (and, perhaps, even in person).
A machine so famous that it is synonymous as The Recognizable Object of its class, like the Golden Gate Bridge.
A machine so famous that even people who don’t know its name can still recognize it.
Yes, for those of you who know me, I am a space nerd. And yes, I am talking about the Saturn V.
I have chosen this as an example of the universality of this image and the relative rarity with which the following details are known.
Most people’s exposure to the Saturn V comes from movies, documentaries, model sets, and museums. Save for two instances of film, almost every depiction you have seen of the Saturn V’s first stage engines is incorrect, or more correctly, incomplete. Every single popular representation either omits or fails to correctly depict a critical aspect of the engines,
For example, Apollo 13 (1995);
Assorted Final Fantasy remake (2020),
Rocket gardens (ca. 2022),
Marshall Space Flight Center displays behind Wernher von Braun (1969/70?),
Textbooks2,
Gorgeous illustrations (2022),
And most consequentially, Lego sets,
These images and museum pieces are either based on the genuine article or are the genuine article themselves. But they are not what the Saturn V looked like on the pad. If you could go back in time to stand underneath the mighty Saturn V before it was launched, the F-1 engines would have looked like products of an arts and crafts project gone awry;
This detail is not a triviality. Every pound (or gram) put on the rocket was there for a purpose. In this case, it was to protect the engines from themselves. Before launch, the engines would be brought into the Vehicle Assembly Building (VAB), and layers of Inconel “foil-batting” would be applied to the more complex engine complex surfaces and asbestos blankets for more straightforward surfaces. As this process took thousands of hours, and they were applying asbestos, almost every other display omits this step.
Interestingly, projects around Apollo 11’s 50th that restored the original film showed this detail to audiences for the first time. One of the better shots of this can be seen in First Man. The film’s VFX team and director, Damien Chazelle, restored NASA’s archival engineering footage from the launch and applied CGI to make details more visible, creating a form of Baudrillard-ian hyper-reality,
At first glance, my digression may appear to be pointless and pedantic. And quite fairly, it is pedantic, but it is not pointless.
The Saturn V is one of the most recognizable machines humanity has ever built, and this is one of its most famous parts. When people think of the word rocket, they think of the Saturn V. Billions of people have looked at this rocket and these engines at least once in their life. Countless schoolchildren have been repeatedly exposed to this machine for decades, not just in the US but across the world. It’s difficult to find a curriculum that doesn’t feature this rocket.
It’s iconic. So iconic that it has its own mini-cult of adherents who worship it with near-religious fervor.
And yet, despite all this reproduction and memetic transmission, outside of a few dedicated space fans, we’ve forgotten what a crucial part of it looked like on the launch pad.
We’ve forgotten the asbestos.
Once you pay close attention, this form of forgetting is everywhere. In no particular order, we’ve lost the recipe for roman concrete, the process for creating greek fire, the knowledge and existence of devices like the Antikythera mechanism (before its discovery), odometers (roads had accurate mile markers in the Hellenistic and Roman age, deviating by ±0.2%3), vending machines, primitive steam engines, sophisticated water clocks (with gauges and alarms), etc.
If you had lived concurrently with these inventions, their loss would have been inconceivable. How could anyone forget the recipe for concrete that could last for millennia? Or lifting large stones into a ritual spot without the use of heavy machinery? And yet, we’re the same species that forgot the existence of sophisticated analog computers for millennia before an accidental discovery.
This process of forgetting seems to be like Philip K. Dick’s kipple. A form of entropy baked into us that steadily occurs without us noticing. Forever multiplying unless we consciously fight against it.
I have observed it in my personal projects. I’ve written bits of code that worked and I thought were exceedingly clever in their “simplicity,“ only to come back a year or three later and be faced with inscrutable ancient Greek.
I have also observed it at larger scales within organizations that I have consulted with, ”oh, no one knows how that works anymore, the people who built it have left, and we don’t want to touch it right now.“ And we can witness it writ large with projects like the Saturn V.
Observations.
While examining the problem, two details stood out to me,
Forgetting starts at the edges.
Little details seem to always fall first. They’re the first casualties in our war against entropy. We don’t start out by forgetting the major things; rather, the loss occurs through gradual buildup. Details, like the engine batting, are harmless, but they add up. Over time, the process of forgetting seems to accelerate, and the amount we forget appears to grow as well.
Documentation (usually) doesn’t help.
Documentation doesn’t help. Almost all documentation regarding the Saturn V has been preserved. Yet there were so many changes, so many little details that never made it in, and so much to record that it is simultaneously too much and not enough.4 Documentation is, at best, a tool to help those in the present communicate with others. At worst, it’s a simulacrum of understanding that quickly falls apart on close examination.
Re: Documentation
Some will (rightly) question my observation that documentation doesn’t help. While a broader, more formal study still needs to be performed, the real-world examples I’ve seen firsthand and have read of seem to suggest that documentation rarely (if ever) rises to its intended and stated purpose; documenting a project.
The most emblematic of these cases is the story of FOGBANK. A substance that is the subject of my favorite quote,
The material is classified. Its composition is classified. Its use in the weapon is classified, and the process itself is classified.
After the fall of the Soviet Union, under President Clinton, the US decided to scale back some of its Cold War-era infrastructure in anticipation of a new era of peace. The “end of history.” One of the many things decommissioned during this era was a small plant near Oak Ridge National Laboratory that produced FOGBANK (an aerogel) for the US nuclear arsenal.
At the time, the cuts made sense. The beyond classified material was also beyond expensive to produce. Its factory was even more expensive to maintain. From the perspective of decision-makers at the time, mothballing the factory was the most optimal decision as the production process had been extensively documented. The material had been extensively characterized. They had documentation regarding its properties and (presumably) data from testing. They knew the machines used to make it.
It “just made sense” to remove the physical asset while retaining the knowledge. Surely, they could spin up a new factory if more of the material was required in the future.
In 2000, the National Nuclear Security Administration (NNSA) decided that it would keep the weapons the material had been used in for another 40 years. As the material had a rated lifespan, this decision couldn’t be executed without restarting the production process. A decision was made to allocate $23M to find an alternative and to spin up the factory again. After all, how hard could it be?
They failed.
When they tried to recreate the material following the steps they’d been given, it didn’t display the original properties. Neither did the alternatives. It took nearly $70M and a few more years of experimentation before they finally replicated the material.
Unsurprisingly, the original documentation was lacking — limited both by the practices at the time and what they couldn’t have known. Even the best technology in the ‘80s didn’t detect that the material’s properties were due to a trace impurity and that without this impurity, the process wouldn’t be the same.
The difference was so subtle that it took years and $70M to find the problem. From LANL’s brief explanation;
I used to write research reports for a living. One of the customers I once had was a foundry operated by someone who had spent decades in the industry. While talking to him, he told me something astonishing, when casting something like Aluminum alloy 7075 (AA7075), they would vary the composition of the alloy ever so slightly on a regular basis depending on the day’s humidity, temperature, and casting results.
This practice was surprising to me as I’d always presumed that the alloy composition would be fixed, but on further reading, I learned that there is always some leeway in the mix. For example, the short recipe for AA7075 is,
Aluminum : 87.1 – 91.4% Zinc : 5 – 6% Magnesium : 2.1 – 2.9% Copper : 1.2 – 2% Trace (Silicon, Titanium, : <= 0.5% Chromium, Iron, Manganese etc.)
Like a chef baking a pie, my customer would change the percentage of each element based on what they felt the day’s castings needed. The day-to-day compositions were recorded in a notebook, but the why behind it and the how were a closely guarded secret.
This foundry was by no means an outlier in the industry. I checked with the other handful of foundry customers I had, and they reported similar practices. An entire national security critical industry runs on the concentrated experience of a few hundred to thousand people. Those in glass houses should not cast stones, but their bus factor is worse than the software industry!5
A Solution (?)
In some depictions of Greek myth, the Moirae (or, the fates) would weave destinies into a tapestry that would inform the story of the world. For each life they’d weave into the tapestry, one would spin the thread, the other would measure, and the third would cut. If the tapestry of fate was made out of lives, then the tapestry of our civilization is made out of the individual and collective achievements of humanity. Including what we have achieved in our attempts to solve this problem.
There are as many solutions to this problem as there are lost civilizations. For example, the now extinct Druid order was a strictly oral culture due to their desire to preserve their secrets, and their belief in a strong oral tradition being necessary for mental acuity.6 We remember them through the eyes of Romans, and one Roman in particular, Julius Caesar — who painstakingly recorded his campaigns.7
Counterintuitively, the fact that we forgot the druids doesn’t mean that the oral tradition isn’t somewhat effective. It was, after all, the dominant medium of human knowledge transfer for most of our existence — the following quote is from the book Signs of Orality,
The uncomfortable truth is that nothing we have tried so far, no technology we’ve invented, so far, has allowed us to prevent knowledge loss. We have lost more than 50% of all feature films made before the 1950s, and over 90% of all movies made before 1930.8 We are already unable to access substantial amounts of important data stored on floppy drives,
“But, then again, sometimes it could just be a bad set of disks,” said O’Neill. “I’ve hit a whole run where double-sided, high-density disks don’t work. I can’t get them to read. Sometimes I’m able to recover them quickly; sometimes not. Sometimes the third time works. It’s trial and error. But you have to balance that with what’s on the disk and whether or not it is worth the extra levels of work.”
- Tracking Digital Collections at the Library of Congress, from Donor to Repository
Nothing we have tried in the past two hundred thousand years of our species’ existence has successfully captured and preserved sufficiently complex knowledge for future transmission. Or, rather, nothing that we have tried so far has worked. We have tools that our ancestors didn’t. What if we imagine what a potential solution could look like?
In the spirit of decomposing problems and spherical cows in a vacuum, let’s imagine that we have some magical tool that can create any technology that we can imagine, what would the solution to this problem look like? How would it work?
For me, the answer to this question is “obvious,” if I could magically create any technology that had the potential to solve this problem, then I would recreate braindances from Mike Pondsmith’s Cyberpunk universe.
To quote the in-game text;
Braindance.
As braindance's popularity spread, public opinion divided into two camps: those who saw the new technology as a revelation, and those who saw it as a threat. Some predicted braindance would save humanity as we know it, bringing an end to unnecessary conflict and ushering us into a new utopian age of mutual understanding and respect. After all, the argument went, if you look at yourself from your enemy's point of view, if you understand him deeply – and vice versa – he ceases to be your enemy.
Others, however, foresaw a wave of new addictions, deepening isolation and societal division, even a crisis of identity.
In the end, the market proved both views right – braindance became an everyday product, another form of entertainment, but also a new form of surveillance.
In the Cyberpunk universe, characters with Brain-Computer Interfaces (BCIs) can step into the consciousness of other individuals through “braindances.” A braindance isn’t just a simulation of what the other person saw and experienced at a sensory level, but rather it’s the recording and immersive playback of someone’s consciousness.
As a fan of the Pondsmith’s universe, and sci-fi in general, it is interesting to me that these stories never quite tackle the profound effects such BCIs would have on knowledge work. A braindance is different from simply “downloading knowledge” — a fact that’s never truly explored in these stories. Instead of downloading say, a physicist’s understanding of,
What if you could download and experience Einstein’s consciousness as he conceived, drafted, and created the field equations? What if you could sit in Emmy Noether’s body as she wrote her theorems? Or felt the beautiful desolation as you step out of the LEM in Neil Armstrong’s shoes? Or, experience Swan Lake from the perspective of the ballerinas?
If we could record, store, and transmit the configuration of our neural state as we are in the process of discovering, creating, and problem-fixing our inventions, then we could theoretically eliminate the problem of knowledge loss.
If such a machine existed, it would be a solution to our problems. Which, I believe, gives us an indication of how to solve the problem within the constraints of our current technology.
We don’t have a magical BCI that can jack into our brains and record our neural contexts for posterity (not yet, anyway). However, what if we used the ideal case to generate the next best thing? What if we could induce — through questions, examples, explanations, and sample problems — an approximate neural state in someone else?
We already do this, sub-optimally, in teaching — what if some of those stratagems could be adapted for preserving technology? It won’t be 100% effective. But it could be a potential solution with greater efficacy than current practices.
More concretely, we could shift the role of documentation from describing what something does or how something was done, but rather attempting to teach why the system does X, why choice Y was made, before detailing the how.
If we are truly dedicated, and willing to go a step further, then we could take a page out of the reverse engineering handbook, and have a small “clean room” team that have access to the project notes and documentation, and nothing else. Can they recreate the project from those notes, in that vacuum?
Or, less ambitiously, if given access to the code base/blueprints, can they understand what does what, and successfully add to the project?
As a concrete example, we can turn back to the Saturn V. If we were attempting to preserve this element of the project using this methodology.
F-1 Engine Thermal Protection System.
When firing on the test stand, $ENGINEER expressed concern about the heat being produced by the engines. Most of the engineering of the first stage was dictated by thermal constraints and Milton Rosen’s desire to have a “Big ****ing Rocket.” The original design called for 4 F-1 engines, as seen here,
The addition of one more F-1 increased the amount of thrust available for the vehicle. But one of the bigger problems was that the quartet had a empty space in the middle. People were worried that exhaust gases would accumulate with potentially explosive consequences. Adding a fifth engine and its plum filled the void and directed the heat away from the base of the first stage.
But the engine still got too hot!
The fuel pump delivered 15,471 gallons or 58,560 liters of RP-1 (kerosene) per minute. RP-1 is heavier than kerosene. Assuming 1.02g/ml, we get 995.5kg/s or 2,195lbs/s.
The higher heating value of kerosene is 47 MJ/kg, using that to gain a rough estimate, we arrive at 46,788.5 MJ/s per engine. Or, 233,942.5 MJ/s for all five engines.
That’s nearly five and a half times the amount of energy used by your house in a year. The engines would run for 168s in a nominal launch, or roughly it would be about 63% of the yield of the weapon used at Hiroshima.
Based on measurements of the plume, the peak temperature reached is 2,600˚K/4,220˚F, and it is more than 1.5 miles long.
This is a lot of energy contained in some very hot gases! It has to go somewhere. Ideally, not into our vehicle. But, as the vehicle goes up, due to properties of aerodynamics, the plume starts to climb up the sides of the vehicle9,
There was concern that, at higher altitudes, gases would concentrate around the engines, leading to catastrophic failure from the exterior (a liquid film of fuel protects the interior).
After testing, it was found that inconel and asbestos could be layered to create a shield to protect the engines. Layering was done in A, B, and C steps by personnel in the VAB… [details of the process — note: there isn’t sufficient public information without consulting archivists on the how/why of the precise material choices and foil batting process]
The example is likely subpar as it is a fictional example, but the intent is to preserve the why before explaining how the solution was arrived at. Hopefully, as the reader goes through the example, they can put themselves (imperfectly) in the shoes of the designers and arrive at a more intuitive understanding.
In the future, I will try to document my personal projects in this newsletter using a similar technique in an attempt to practice what I preach. It would be interesting to compare/contrast and see if the results are better at preserving knowledge than the alternatives.
Best,
areoform
When the song came out, I doubt it that anyone would have foreseen that the proto-punk bopper would be played by a military band in full dress at a state function to a Chancellor who listens with tears in her eyes.
There is an argument to be made that the textbook example is correct, as it is a drawing showing different engines, and it is displaying the engines.
This error number was derived, helpfully, by a wikipedian who referenced the distances described in Pliny the Elder’s Natural History, “we will follow in the track of Alexander the Great. Diognetus and Bæton, whose duty it was to ascertain the distances and length of his expeditions, have written that from the Caspian Gates to Hecatompylon, the city of the Parthians, the distance is the number of miles which we have already12 stated; and that from thence to Alexandria,13 of the Arii, which city was founded by the same king, the distance is five hundred and seventy-five miles;”
From the wiki, “section between the cities Hecatompylos and Alexandria Areion, which later became a part of the silk road, was given by Alexander's bematists as 575 Roman miles (529 English miles) long, that is with a deviation of 0.2% from the actual distance (531 English miles).”
Case in point, the need to pull apart and 3D scan the engines to derive a final schematic. See arstechnica’s excellent piece, How NASA brought the monstrous F-1 “moon rocket” engine back to life
Could we build a neural network that takes their logs, recorded variables, and the outputs to generate these recipes on demand? Yes, I suspect so. But that would shift the black box away from wetware to hardware without addressing the underlying problem.
Nagy, Joseph Falaky. "Representations of oral tradition in Medieval Irish Literature." Language & Communication 9, no. 2-3 (1989): 143-158. Accessible via PDF
It is likely that the Gaelic Druids would be horrified to learn that the Romans are the reason why we remember them. Their horror would only be compounded upon learning that their Chronicler-in-Chief is Julius Caesar.
Via the Scorsese Foundation, Group's rescue of old films preserves glimpse into past
Dr. Chris Combs has published a great explanation of the phenomena on Twitter,
Really recommend the book "The Shock of the Old" some great examples of forgotten tech but also long this line of forgetting, there is also the persistence of many old technologies, far beyond their narrative expiry.
https://en.wikipedia.org/wiki/The_Shock_of_the_Old
The chapter on flush pop rivetting in "What engineers know and how they know it: A History of the aircraft industry 1918 to 2002" by ... Damn I forgot..