A certain nuclear power plant had a Windows NT 4.0 machine running as late as 2007. The reason is interesting.
The machine's purpose was to report status of the control rods that mitigate nuclear reactions. Basically, "are the rods inserted, and if so, how many / how far?". I want to emphasize that this was reporting only, NOT control.
The original software was written back in the 80's, when the plant was originally commissioned, for AmigaOS. Of course, it's hard to buy Amigas anymore, and the original one died long ago (nobody remembers when).
So in the mid '90s, the utility purchased an AmigaOS emulator that ran on Windows NT 4.0, which was current at the time. The emulator (IIRC) was developed by a firm in the UK. The firm went out of business sometime in the late '90s. The control rod monitoring software ran under this emulator on top of NT4.
Windows NT 4.0 was the last OS to allow the emulation software direct access to the physical hardware that produced the status signal. Later versions of Windows abstracted the hardware access away, and the monitoring software broke. Because the emulation company had gone belly up, there was no way to fix the incompatibility.
So the utility had a choice: get new hardware/software certified (by NRC?), or keep doing what they were doing with the software (and hardware) that they had. They chose the latter.
So this is how, in 2007, during a tour of the facility, I stumbled across a Pentium 1 system running an AmigaOS emulator on Windows NT 4.0 that was responsible for displaying the status of the control rods of a nuclear power plant.
Spare hardware for this setup was purchased off of eBay and stocked on an adjacent shelf.
To me the funny part of this story is that they used to show you this warning as part of the EULA when installing Windows NT 4, which I remember joking about:
NOTE ON JAVA SUPPORT. THE PRODUCT MAY CONTAIN SUPPORT FOR PROGRAMS WRITTEN IN JAVA. JAVA TECHNOLOGY IS NOT FAULT TOLERANT AND IS NOT DESIGNED, MANUFACTURED, OR INTENDED FOR USE OR RESALE AS ONLINE CONTROL EQUIPMENT IN HAZARDOUS ENVIRONMENTS REQUIRING FAIL-SAFE PERFORMANCE, SUCH AS IN THE OPERATION OF NUCLEAR FACILITIES, AIRCRAFT NAVIGATION OR COMMUNICATION SYSTEMS, AIR TRAFFIC CONTROL, DIRECT LIFE SUPPORT MACHINES, OR WEAPONS SYSTEMS, IN WHICH THE FAILURE OF JAVA TECHNOLOGY COULD LEAD DIRECTLY TO DEATH, PERSONAL INJURY, OR SEVERE PHYSICAL OR ENVIRONMENTAL DAMAGE. Sun Microsystems, Inc. has contractually obligated Microsoft to make this disclaimer.
Also:
The machine's purpose was to report status of the control rods that mitigate nuclear reactions. Basically, "are the rods inserted, and if so, how many / how far?". I want to emphasize that this was reporting only, NOT control.
It would take a whole lot more context to make this somehow comforting. :D
This is a great post. How does someone write software that needs to run for ~50 years where the hardware will need to be replaced with non-equivalent, newer hardware? If I were facing this issue today, I might start with an OS that has excellent emulation. Example: Can I run 32-bit MS Windows 95 via emulation on a variety of current 64-bit OSes, like MS Windows, Linux, AIX, HP-UX, etc. If yes, then we can assume(?) this emulation will remain relatively stable even if we upgrade our hardware later. Maybe I am overthinking the whole problem: Can VMs do exactly what I want today? Will VMs running ancient OSes, such as 32-bit MS Windows 95, continue to be stable/viable in the future? I am unsure.
Make sure that you have as thorough a specification of what the system is supposed to do as you can.
Then define the version control system and the build process, specifying the dependencies, and so on.
Think about any opaque blobs in the system and try to eliminate them so that you have plain text source code so that no tools are needed to read the code.
Make sure that the build process runs entirely locally and never fetches anything from outside.
Simplify everything, use only tools and languages that are well understood and supported.
The real problems are not strictly technical but social: how do you prevent loss of the code, the tools, the specification, how do you maintain the expertise needed to maintain it. How do you ensure that all those things that are obvious to you now are written down in all their gory detail so that your great grandchildren will not apply their new and different preconceived ideas to the system?
Document all this on paper as well as electronic storage, make sure that version ids are recorded on every page as well as being available to the user of the machine or program.
In the industry in which I worked for the last thirty years of my career it was not uncommon to have things come back for repair after fifty years use and to be able to consult the original drawings and bill of materials so that exact replacement parts could be made.
> How does someone write software that needs to run for ~50 years where the hardware will need to be replaced with non-equivalent, newer hardware?
Some ideas:
Write it in a popular language/ecosystem, stick rigidly to well-defined APIs, use commodity hardware, flatten out any malignant cleverness, maintain documentation on why every part does the thing it does, and make the source code readily available.
This is based on working with some very old systems, and each point above is the opposite of something that made life harder.
Unfortunately, “popular language/ecosystem” includes JavaScript and PHP.
I’d go for “universal and timeless”, which has meant C for several decades now. There is, I’d concede, a reasonable chance that Rust obtains that label in our lifetimes.
It's interesting to think about this in the context of parent poster's story about the Amiga emulator in the nuclear power plant.
Or, more broadly, in the context of any situation where the software is closely coupled to a specific operating system and/or hardware.
Having the source code would have helped tremendously, but might not have been sufficient because of the hardware story and the regulatory approval factor.
I'd recommend not having it so dependent on the hardware and choose an OS that is relatively hardware agnostic (e.g. Linux). Also, ensure that the software is open source and if compiled, has an open source toolchain to do so.
Though people often decry it (due to the many, many footguns), writing something in Shell/BASH will make it trivial to move to newer machines.
A friend works for an airline as a flight simulator tech. Their entire software stack, including the compiler and OS, is FAA-certified.
Then their ancient Honeywell(?) mainframes reached end-of-life they scouted for compatible hardware, of which there was none. The cost of certifying new software, plus the time involved, was astronomical. So, after consulting with the FAA, they paid a hardware company to clone the ancient mainframes in modern silicon. The FAA signed off on it, and they had all-new computers - much smaller than the originals - running the old stack.
Custom chips aren't as expensive as you think. About a few million dollars for the design and a hundred thousand chips, way out of reach for a hobbyist, but accessible to large enough companies. Just because it's opaque to us software people doesn't mean it's not a real industry you can buy things from.
Situation: Software-1 on Platform-1, both certified.
Time-evolution: P1 is deprecated, replaced by uncertified P2, but S1 remains certified.. just nowhere certified to run yet.
Solution: There's another software S2 which originally vouched for P1, itself still certified, which can still be used to certify P2.
Counterfactual?: If S1 were deprecated in favor of new S3.. there'd be no plan to certify it!
The problem: None of this actually makes any sense! But we're trying to fake due diligence. Everyone knows the hardware/platforms kinda need to be certified with respect to each other anyway, but if we did it that way it would all be even more expensive an time-consuming.
Doesn't seem crazy to me. Which one do you suppose needs more proof of correctness: a cake recipe, or the oven you bake the cake in? The recipe has to be correct or the cake won't work, but the oven just has to hold a temperature.
Ovens vary greatly. There is no consistency in how even the heat is in the space, the airflow through the space, how much and how fast the temperature varies around the set temperature, etc. These things all do affect how things bake up and often people have to tweak recipes for their specific oven.
In a case where something is safety critical the computer is super important as very subtle errors could cause catastrophic consequences. This is why you get into things like running software on 3+ computers concurrently depending on how fail-safe something has to be.
It could be an FPGA. Given enough gates, and the know-how to program them, you can make an FPGA emulate anything. Speed and efficiency could be better or worse, but if you're targeting old hardware, better is likely.
When Xerox established PARC, they asked the assembled scientists what computer they wanted. The majority view was a DEC PDP-10 KA10, with the BBN memory management unit that let it run Tenex (the ancestor of DEC TOPS-20). Xerox couldn't really buy a competitor's mainframe, so they built MAXC (maximum access computer), which was a complete emulation of the Tenex machines.
>MAXC (maximum access computer), which was a complete emulation of the Tenex machines.
Compuserve also had a software dependency on the PDP-10 and Decsystem 20, and when those machines were no longer available they bought a company making clones so they could continue to manufacture them for themselves. The company they bought, Microsolutions, was Mark Cuban's first startup.
making clones of mainframes (IBM's) had been a big area of intellectual property litigation, but also facing monopolization investigations, IBM had to allow them. They were referred to as "plug compatibles".
Meh, old architectures are relatively simple and you can get 95% of those via fpga softcores (some freely available, some paid) and run them on fpga. The rest can be implemented by a proper ee team.
I mean...there's some precedent. The XKL TOAD ("Ten On A Desk") implements the PDP-10 instruction set, and will happily run TOPS-10/TOPS-20. LCM had one doing just that.
https://en.wikipedia.org/wiki/GE-600_series from the 36-bit era which turned out to be a sweet-spot for Lisp; also "famous" for running Multics (old joke flipped: "which was many of whatever unix is one of")
GE sold it's division to Honeywell, and from there to Group Bull (French) and then to NEC (Japan)
Hardware is more amenable to static analysis than running programs which may receive an arbitrary number of inputs and express an arbitrary number of possible intermediate states.
I am interested in how firmware is treated, since perhaps in the case of these old machines it's small enough to be analyzable or at least cloned bit-for-bit.
Having worked on nuclear plants (as a reactor operator) around that time this doesn't surprise me in the least.
Thats far more advanced than the systems I worked with, one of which reported rod position via resistance measurement on a brushed cylinder (one for angular and one for depth).
Cleaning and calibrating those was a constant maintenance item every time the reactor was shut down.
That must have been a crazy moment for you when you first saw it and a big sigh moment for whomever had to explain it to you the first time. Thanks for sharing this; it made me smile!
That is an advanced question best asked of the folks in charge of the plant. At the time I was just trying to figure out how to explain to the IT auditors why there was no antivirus software on this piece of crap.
Could you skirt around it and just remark "there are no viruses for AmigaOS"
(Probably not literally true. But functionally true in the sense that they were likely transmitted via infected floppy disks, of which there'd be virtually none left in the wild in 2007)
It wasn't the AmigaOS running in the emulator that they were worried about, it was the Windows system running the emulator. (Not that the average IT auditor there could have understood the difference.)
If it were a newer version of NT I'd be concerned about USB media, but being NT 4.0 and not supporting USB it's imminently more capable of being air-gapped than later versions. (I recall a fun Ed Skoudis quote-- "At best, an air gap is a high-latency connection". Evidence Stuxnet.)
Depends. A lot of these audits are quite prescriptive and don't leave much room for actually thinking about the problem. There is often some kind of mechanism for 'this is sufficiently segregated it doesn't matter that it's utterly out of date' but then usually some awkward rules develop that prevent some things from being put into that category. More subtly you can get whether the thing even exists as a thing that the audit cares about, and that often depends on the framing (embedded software is often invisible here but it needs to not look too much like a general-purpose OS even though it often is).
(Also, in my experience, what the auditors think the rules are and what is written down can often be divergent and even contradictory)
It is wild to me that you would use any GPOS for something like this instead of dedicated firmware for the task. This is evoking similar reactions to when I read articles about infrastructure being hacked remotely: Something must have taken a wrong turn with the architecture design for this to happen!
This reminds me of the apocryphal story of the IBM System/360 running a 1410 emulator that ran an IBM 705 simulator that ran a business-critical application.
Which is why we should target VMs directly. Then that VM can be brought forward to new platforms rather than emulating the entire machine, not that emulating the entire machine is bad. One could argue that the emulation stack you outline was made possible by putting in the work to make simple, fully specified and documented hardware.
> A certain nuclear power plant had a Windows NT 4.0 machine running as late as 2007.
To put this in perspective, NT 4 was released in 1996, 11 years before 2007. That’s roughly the same as someone running Windows 10 today, which a lot of people still do. The Pentium 1 was only 14 years old at most. These parts of the system shouldn’t be surprising.
The AmigaOS and software from the 80s are of course the older more interesting bits, but for critical infrastructure and safety, shouldn’t we expect and design for our hardware and software to last a long time, and not try to keep up with tech fads every decade? I feel like the main problem isn’t hardware or software that’s old, but that there wasn’t a longevity plan.
It’s always been an interesting question how to write software that will last a hundred years. Maybe with AI this is the first time in history that planning to port & upgrade to new hardware every 5-10 years seems totally reasonable.
> That’s roughly the same as someone running Windows 10 today
While true the situation is different. Hardware and software moved a lot faster in the 90's and early 2000's. Everything has matured, is far more standardized, and moves more slowly now. Hardware from 11 years ago when Win10 was released isn't that different from today's hardware. Same goes for Win10 and Win11.
There might be a seed of truth there, but FWIW, I’m not very convinced by that. CPU speed development has slowed down a bit for sure, but software development is going faster than ever. GitHub and NPM and Python 3 and Rust all didn’t exist in 2007. CUDA was first released that year. LLMs weren’t being used. At the start of 2007, the latest Windows version was Windows 5, one more than Windows 4. Almost same as today. There isn’t a Windows 12 yet, while Windows 6 was released in 2007. So the pace of major Windows versions might be slightly slower. OTOH, the OS is much bigger than it used to be and it gets more frequent updates and point releases now than it did in 2007.
lol. I know a ... factory, that had a BBC model B (with a rather complicated wiring loom) still doing a job around that time.
The IT supplier at the same factory went to a museum in Cambs. around late '90s, early '00s to ask if they could buy an exhibit because something had failed locally. The museum gave them the part.
That was just aerospace and nothing fancy like your nuke plant!
Was this status indicator simple vertical bars that were black and white and displayed on an old CRT Monitor?
Almost looks like black and white terminal vertical bars (like Alsamixer)?
I'm curious, how did the hardware present itself? It couldn't have been an Amiga Zorro card since it would've been impossible to get that into a PC. Did it connect over a serial or parallel port?
You don't need protected memory if your program is the only thing running. Plenty of microcontrollers have no protected memory and get used in safety critical applications.
I wonder if there is push in the public sector for open source software/hardware for cases like this.
I completely get the decisions over time here, but it's unsettling having a relevant piece of software (reports are important too!) working on with parts from ebay, in 50 years time they might be gone.
I've dreamt of governments hiring devs with promises of full time OSS work. It _might_ get some people over the lower salaries of government work. But I overheard a state CIO once say "We configure, we don't create".
That's to say nothing about opinions of open source that rival early 2000s Microsoft.
For old ass critical software and hardware, the org can always have a backlog of hardware and contingency plans (paying someone to fix the hardware, paying someone to create a fully certified and modern solution..) for when they start running out of parts.
How on earth are these massive pieces of infrastructure relying on software they don't have the code to? i'm just utterly confused how these things happen.
This is crazy. Why on earth wouldn’t the respective government spend money to modernize a critical infrastructure like a nuclear power plant?
Safety and reliability come from understanding the system. Something that's old but that you know everything about is far safer than something new. This risk is that the people who know move on, or the sources of replacement parts stop working, or that other things around the system change. Then the risk curve inverts and you find the old system more of a liability than a asset.
Almost always people choose to update things either too early or too late. Knowing when to do something is hard.
That’s pure negligence. It’s not like the people that have the know how disappear suddenly . It’s the responsibility of the management to make sure the knowledge transferred to new generation.
I guess that kind of thinking is what led to the collapse of ancient civilisations like why even bother to improve anything and let everything decay and die out
Basically everything collapses as far as it is allowed to, and the collapse is slowed only by:
- profit, if a system generates it, people will keep the system running.
- dilligence, usually driven by personal idealism, and unrewarded.
- legal or financial penalties, or insurance costs.
- it has collapsed to a temporarily stable state for now.
For another comment I was looking at the Grenfell Tower fire in the UK around 2017 and the people who lived there had been raising fire risks for years and the landlord (management organization) and the council had been ignoring them, and ignoring the fire department. The companies which replaced the cladding on the tower with flammable cladding were choosing the cheaper flammable option and pointing fingers as well. And during the fire, the fire service had never dealt with such a big fire and didn't have a truck with a long ladder, didn't have extended-breathing gear, had radio problems, water pressure problems from the local water company (who deny that).
This kind of backstory is typical for disasters, and for IT outages - and I've taken to believing that if something "should work" but wasn't tested recently then you should expect that it doesn't work. This is a common saying in backups (you need to test restoring), but it seems to apply everywhere. Backup internet connections that don't have enough bandwidth for the company to keep running. Disaster recovery sites that share resources with the production site. 'Disaster recovery' that recovered into a remote site, but they couldn't do CAD work remotely over their slow connection so it was still disastrous. Multiple power feeds but they were cross-wired so one failing still functionall took down everything.
If knowledge transfer "should be happening" but it's not critical to a job, or it's not profitable, or it's not legally required and audited, then it isn't happening. Subject to the dilligent idealist employee mentioned above who are temporary in the long view. Management's involvement seems not to arrange the larger system to effectively shoulder responsibility, but rather find ways to avoid personal blame while cutting costs beyond the point where everything that needs to happen keeps happening.
You can only say such things with hindsight. In the moment, nobody knows what the future will hold. Take the space program in the US for instance. In 1969 we landed a man on the Moon. We'd repeat that several times until 1972. At that point Nixon made a speech which implied various unpleasant things about space's future, but he'd be out of office, disgraced, a couple of years later, so how big a deal could it have been anyhow?
I think almost nobody from that era would have believed you if you told them that more than 50 years later not a single human would have traveled beyond low Earth orbit, and that we'd now be struggling to recreate what we did in 1969. Furthermore a significant chunk of the world doesn't even believe we landed on the Moon anymore, no doubt in part because of this apparent anachronism.
It's not like there was any sort of collective or even singular decision to just let things die out. I mean sure Nixon did his thing intentionally, but even as a President in the golden years of the US, he was still just one man.
It’s the responsibility of the management to make sure the knowledge transferred to new generation.
That's just lazy thinking, and probably a sign you've never been a manager. For all the responsibility to lie with management you'd have to be leading people who enthusiastically do what they're asked to do, raise problems, document things, follow processes, and make sure everything is handed over to the next person when they leave.
That's not how people are. They'll have times when they're unmotivated, underpaid, grumpy, over-worked so they miss things, etc. That's when the organisational tech debt starts piling up, and there's nothing a manager can do to stop it except manage it as best they can even though they're under the same pressure, with the same lack of motivation and crap pay as everyone else.
It's so easy to think you can fix it just by keeping on top of it and micromanaging where it starts to show up. It doesn't work that way.
Their 100 y/o hardware is built from stuff that's still produced though. We still make relays (not identical ones but ones that perform the same function), wires, and mechanical levers. The system's behavior is not encoded in the relays but in the connections between them. If you had a CPU made of separate transistors you could replace burned out transistors with any future transistors but if you had one made of chips you'd need those exact chips that were out of production.
Replacing the relays in old telephone switches is a problem because they use so many weirdly specific types of relays - but those systems are only found in museums now. I believe track control uses fairly ordinary DPDT, etc., relay designs.
But similar reasoning is why Canadian trams are worse than other countries, and it's not "good experienced engineering". In this nearly hour long rant by NotJustBikes[1] the section I've linked is about the technology and how the transit authority will not move on from 100 year old track technology which makes everything worse for customers. There are junctions where tram drivers have to stop and look at the track to make sure the junction is correctly set because the single-point switching is less reliable than modern designs. Or the driver has to stop the tram and get out to use a stick to move the track and change the junction because they aren't electrified. And then pass through junctions as slowly as 10km/h because the old track design has higher risk of derailment than modern designs.
The designs for the electrified track switches were made by one company which was sold, re-sold, and then burned down and the designs lost, and there isn't anywhere else to buy them because nowhere else uses such old track design anymore, so not only is Toronto struggling to buy parts to continue electrifying the manual junctions, it's struggling to custom-make enough replacement parts to keep the aging electric switches working faster than they fail.
And they were using trolley poles to connect to the overhead power lines instead of Pantographs until 2017, 50-100 years after everywhere else, which meant a) they couldn't get enough power through them to run air conditioning, and b) they are more likely to fall off the power line and the tram stops and the driver has to get out and reconnect them, and c) their newer trams now have to have custom dual-power connections to go through older parts of the network, and d) the carbon shoes on the trolley poles wear out quicker which means the trams are dirtier and taken out of service for maintenance more often in the wet which is when people want to use trams more.
I understand that way of thinking for a small business computer system. Not for a critical infrastructure. Also if people actually thought about like that we would still be in stone age, like why invent something new or use a new technology? Keep on using rocks to crack nuts and kill animals
I think you can imagine the difference between “I did some work and made this thing that is clearly better, but we might have a few kinks to work out” and “I did some work, and if I did it perfectly, it’s just as good as the old thing, but we might have a few kinks to work out.”
For non-perishable things - i.e technologies, ideas, books, institutions - the expected remaining lifespan is roughly proportional to how long they have already survived
Time acts as a filter: what has already withstood a long stretch of disorder is more robust, so the longer it lasts, the longer it is expected to last.
The old nuclear software running on mainframes have passed the test of time. The new software, has yet to.
Nassim Taleb talks about this at length in his book, Antifragile.
> Why on earth wouldn’t the respective government spend money to modernize a critical infrastructure like a nuclear power plant?
Same reason they don't modernize health care infrastructure. It's too expensive, people die if they get it wrong and they get voted out of office for their troubles.
Now you know why "nuclear power may have been dangerous in the past, but with modern technology it's completely safe and Chernobyl will never happen again" is not a good argument.
Chernobyl didn’t happen because of technological blunder or even human negligence. It happened because poverty caused by communism in Soviet Union had to use old outdated (not the modern nuclear technology at that time) graphite tips in carbon rods because it was cheap and also kept it as a secret from the facility operators giving them a false sense of trust on the kill switch.
Chernobyl is the perfect example to not use outdated old technology. Had they used the modern western technology at that time to upgrade their nuclear reactors the kill switch would have prevented the whole disaster instead of increasing the neutrons make the core a nuclear bomb.
Next time educate yourself with facts before making a fool out of yourself
Wikipedia's pages on Chernobyl say that construction started in 1972 and the problem with the control rods causing an initial spike of power was discovered in 1983 ?
Chernobyl was very much not "old", it was a more modern design than the PWRs which are built today. Chernobyl Unit 4 was also basically brand new, only 3 years old at the time of the disaster. The RBMK design dates from the 60s, whereas the PWR is from the 40s.
>So this is how, in 2007, during a tour of the facility, I stumbled across a Pentium 1 system running an AmigaOS emulator on Windows NT 4.0 that was responsible for displaying the status of the control rods of a nuclear power plant.
Vernor Vinge's A Deepness in the Sky depicts a human society thousands of years in the future, in which pretty much all software has already been written; it's just a matter of finding it. So programmer-archaeologists search archives and run code on emulators in emulators in emulators as far back as needed. <https://web.archive.org/web/20231114211656/http://www.gareth...>
(Heck, recently I migrated a VM to its third hypervisor. It began as a physical machine a quarter century ago.)
The machine's purpose was to report status of the control rods that mitigate nuclear reactions. Basically, "are the rods inserted, and if so, how many / how far?". I want to emphasize that this was reporting only, NOT control.
The original software was written back in the 80's, when the plant was originally commissioned, for AmigaOS. Of course, it's hard to buy Amigas anymore, and the original one died long ago (nobody remembers when).
So in the mid '90s, the utility purchased an AmigaOS emulator that ran on Windows NT 4.0, which was current at the time. The emulator (IIRC) was developed by a firm in the UK. The firm went out of business sometime in the late '90s. The control rod monitoring software ran under this emulator on top of NT4.
Windows NT 4.0 was the last OS to allow the emulation software direct access to the physical hardware that produced the status signal. Later versions of Windows abstracted the hardware access away, and the monitoring software broke. Because the emulation company had gone belly up, there was no way to fix the incompatibility.
So the utility had a choice: get new hardware/software certified (by NRC?), or keep doing what they were doing with the software (and hardware) that they had. They chose the latter.
So this is how, in 2007, during a tour of the facility, I stumbled across a Pentium 1 system running an AmigaOS emulator on Windows NT 4.0 that was responsible for displaying the status of the control rods of a nuclear power plant.
Spare hardware for this setup was purchased off of eBay and stocked on an adjacent shelf.