On a Saturday morning I ran apt upgrade on one of my two servers and rebooted it. It answered pings within the minute and then refused every connection for the rest of the day.
I run two machines. The first holds the infrastructure I am not prepared to lose and has its own UPS, because where I live the power goes out on a schedule. The second is nominally a sandbox, which undersells what had accumulated on it: seven virtual machines, among them the DNS resolver every device in the building points at, the file server with the family archive, and the reverse proxy that puts a real HTTPS certificate in front of half a dozen internal tools. It had been up for thirty-two days. This is the one in the story.
The lab is a working instrument. It is where the operational judgement I am hired for stays in practice: nobody schedules these incidents, and the recovery is mine.
Two faults were waiting, and both were ordinary. What I keep turning over is the shape of the morning. I ran a thorough investigation, most of the digging done with Claude Code, every hypothesis raised and cleared in a sensible order. The cause was a port on my router that I had plugged the server into myself, earlier that same morning.
Everything either of us examined sat inside a boundary I had drawn when I stated the problem. An investigation searches the world it was handed, and that world is drawn by whoever framed the question, usually before they knew enough to frame it well.
A dormant fault hides behind whatever moved last
The machine booted, answered pings, and refused everything else: SSH, the web interface, cluster membership. One port stayed open, and only one. Port 111.
That single open port was the machine telling me how far it had got. Port 111 belongs to rpcbind, and systemd holds that socket open itself from very early in the boot, before the program behind it ever runs. Everything else needs the boot to have finished. The boot had not finished.
I forced myself to unplug the thing and carry it to my desk so I could put a monitor and a keyboard on it. Four seconds after the reboot the screen said everything: it could not reach a disk that was no longer there, had given up waiting, and had dropped into emergency mode, the state where Linux starts almost nothing and waits for a human.
Hence the responses on port 111, and nothing else.
The disk was absent because on 27 June I had reformatted those drives from exFAT to ZFS and never removed the old entry from fstab, the list of storage the machine must have before it will consider itself booted. fstab is read at boot and never again, so the mistake sat there for thirty-two days without producing a single symptom. Then it fired on the very next reboot.
That is the general shape of it, well past fstab: a fault that can only surface on restart will always surface next to whatever prompted the restart. Reboots follow upgrades. So the upgrade wore the blame for two hours, on evidence that amounted to it was the most recent thing to move.
The line needed one word. nofail tells the system that a missing disk is disappointing but survivable. Without it, a drive that has been unplugged, or has merely woken up slowly, takes all seven virtual machines to a black screen. I added the word, rebooted, and the machine came up.
The second fault, which I created by investigating the first
That should have been the end of it. It was roughly the middle.
Both servers sit in a Proxmox cluster, which means they share a small distributed filesystem holding the configuration for every virtual machine, and it has to stay identical on both. Reading it worked. Writing to it hung forever, on both machines at once. Nothing could start, because starting anything requires writing a lock file first.
The logs were precise about what was happening and silent about why. One node reported seventeen pending updates and declared the pair in sync. The other said it was waiting for updates, and then said nothing else ever again. Neither of them retried.
I did not know it yet, but I had caused this myself, twenty minutes earlier, by carrying the server to my desk and plugging it back in somewhere else. The act of investigating the first fault produced the second one. Nothing in the timeline made that visible, because moving a machine does not feel like changing it.
Three hours of correct answers
I worked it with Claude Code. Some hypotheses were mine, some were its; it ran the commands, read the output back, and pushed on results rather than accepting the first reassuring one. Over about three hours we checked and cleared all of this:
- Cluster membership: correct on both nodes, exactly two members, no stale entries.
- Quorum: healthy, three votes of three.
- The link between them: up in both directions, zero transmission errors, zero retries.
- Packet size: a full-size ping crossed the link and came back.
- Clocks: synchronised to within half a second.
- The cluster config file: byte-identical on both nodes, same checksum.
- The database behind the shared filesystem: passed its integrity check.
- The same database, deleted and rebuilt from empty so it could resync from scratch. It failed identically.
Every answer was the right answer
Each of those checks was a reasonable thing to run, and each came back clean. I restarted the services individually, then together, then rebooted both machines, then carried hardware across the flat a second time to get a screen onto the other node.
The absurdity took me most of the morning to notice and none of it to resolve: the cluster was in perfect health and completely unusable. Two members, quorum held, clocks aligned, config identical, database structurally sound, link up with a zero in every error column. A system that agrees with itself about everything except whether it can write a file.
Then I had to be in town for the rest of the day, and it stayed broken while I was out.
A stranger’s forum thread
I came back to it in the evening, out of ideas, and did the thing I should have done at hour one: went and read what other people with this symptom had found. A forum thread described my failure exactly, on someone else’s cluster. Their cause was a bad cable. The cable still carried traffic. It was making the network card fall back from a gigabit to a tenth of that.
Two commands, one on each node. The healthy one reported
1000Mb/s. The one on my desk reported 100Mb/s.
When I moved the machine that morning I had plugged it into the nearest free port, which is a port I already knew was bad. It negotiates 100 Mb and nothing else, which is why it normally carries an IP desk phone, a device that needs about a thousandth of what it is being given and has never once complained. I solved the monitor problem, manufactured a far worse one with my own hands, and then spent three hours asking why the software was broken.
If you are here because your cluster says it is waiting for updates from a node that insists it already sent them, and every check you can think of comes back green: run ethtool on both ends and compare the speed. If one of them says 100Mb/s, stop reading and go and move the cable.
A failure that passes every test you thought to run
A dead link fails loudly, because is it reachable is the first question anybody thinks to ask and the only question a dead link can fail. A link that is merely slow passes that question and every follow-up in the same family.
- ping fails, so the first check anyone runs already names the problem
- cluster membership drops and both logs say why
- the interface reports itself down
- every monitoring system ever built notices within one poll
- you are holding a cable inside five minutes
- ping succeeds, so every reachability check passes
- membership forms, quorum holds, both nodes agree they are fine
- the interface reports itself up with zero errors: it negotiated a legal speed and is honouring it
- small control messages cross all day; the batch that matters never lands
- nothing in the failure names the network, so you debug software instead
From the network card’s point of view nothing had gone wrong, and that is the whole trap. My two nodes exchanged their small control messages perfectly all day: hello, I am here, you are the leader, send me the updates. The batch of updates itself never arrived. Same wire, different packet sizes, opposite outcomes.
The specific form is rare and the general form is everywhere: a component that has degraded rather than failed will pass the tests written for failure. The tests were right about the failure they were designed to catch. They encode a model of how the thing breaks, and the thing found another way.
[max]There is no fixing the port. A network engineer came out, tried, and left it as it was: a single port on a consumer router can degrade and stay degraded, and nobody replaces a working router over one of them. You label it and you route around it.
The box is drawn by whoever frames the problem
Claude Code ran a real investigation that morning. It proposed checks I would not have thought of in that order, read output properly, and refused to treat a green result as an answer. What it could not do was suspect a router.
The reading was one command away the entire time. ethtool would have said 100Mb/s at any point after 10am. But the frame held a cluster, a shared filesystem, two nodes, their logs and the question I had typed, and a router appeared in none of them. You run the commands your frame gives you a reason to run.
That frame was mine. I built it when I described the problem: two nodes, a cluster, a filesystem that will not write. Everything inside it was searched thoroughly and correctly, by both of us. The cause was outside it, one physical layer down and one device across, in a piece of hardware that emits no logs and belongs to no system either of us was looking at.
So this is think outside the box in the flat, literal sense rather than the motivational-poster one. The box is a real object. It has an edge, the edge was drawn by whoever stated the problem, and it was drawn at the moment of least knowledge, before anyone knew where the fault was. Reasoning inside it can be flawless and still never arrive, because the answer is not in there to be found.
Which is the part that survives with the AI taken out of the story. A colleague you hand a bug to, a contractor you hand a brief to, a team you hand a quarter’s objective to, and a model you hand a prompt to are all in the same position: they will search the world you gave them, and they will search it better than you expect. Nobody inside the box is positioned to notice its edge. That job stays with the person who drew it. It is not a reasoning problem, and no larger context window fixes it, because the missing piece was never written down anywhere to be read.
Two things I do differently. When something distributed hangs, I check the physical layer before I touch a service; it costs one command, and I spent a morning proving a great deal of innocent software innocent. And at the start of any investigation I now answer one question out loud, into the session: what changed physically, or outside this system, since it last worked. I had the fact about that port the whole time. It was in my head, and a fact in your head is not in the investigation.
The port is still there, still degraded, still carrying a phone. What I do not have is any way of being told when it happens again: a link that renegotiates to 100 Mb looks identical to a healthy one on every dashboard I own, and I have not yet written the check that would catch it.