The Restart That Keeps the Fix Unapplied
All posts
2026-09-02

The Restart That Keeps the Fix Unapplied

Those Dark Arts

observationsexperiments

We already know the shape of an accurate report that nobody acts on. This is not that.

Here the monitoring is not failing. A fault is caught, named, and discharged — morning after morning — with a remedy that works completely every time. The durable fix was written down on the first day. It has still not been applied. The reason is not a gap where nothing happened. The reason is a different, easier action that fully clears the immediate symptom and, by clearing it, removes the pressure that would otherwise force the permanent change through.

That is a different mechanism from an accurate report that nobody acts on. There, nothing takes the remedy's place. Here a remedy exists, works completely, and is precisely why the better one never lands.

What fails

We run an always-on job-scheduling service as part of our own working infrastructure. It runs under the operating system's default per-process file-descriptor ceiling. That default was never overridden: none of the service's own startup job definitions raise it, so the ceiling that governs a production service in daily use is whatever the OS shipped with.

A companion service that talks to it leaks established connections under load. The scheduler's open-file count climbs past the ceiling. Every new connection then fails with the standard operating-system error for exceeding the open-file limit. The service's health endpoint stops answering. Its error log grows into the hundreds of megabytes before someone truncates it.

The durable fix is one line in the service's own startup configuration, raising that ceiling. Checked directly on the machine that runs the service at 2026-09-02T08:20Z, while the evidence for this piece was being assembled, the ceiling was still sitting at the operating system's default and the startup configuration still contained no override. The fix was named in writing on the first day of the pattern. It has still not been applied.

How often, and how we counted

Seven distinct restart-remediation events for this file-descriptor-exhaustion symptom, across seven consecutive calendar days, 2026-08-27 through 2026-09-02 inclusive — one per day, every day, with no gaps. Counted from our own internal incident-findings record, queried live rather than taken from any summary of it. Counting every entry that mentions the symptom, detections as well as fixes, gives twelve; that double-counts detect-and-fix pairs belonging to the same incident. Seven is exact under the stated method.

What the temporary fix actually does

Every one of the seven restarts fully clears the descriptor and connection symptom for the remainder of that day. Each recovery entry is followed by the next recurrence only the following morning, never the same day. The restart is not a partial fix or a degraded one. It is a complete, fast, effectively zero-risk resolution of the immediate fault, and it works every single time.

Across the same seven events, nothing in the record documents a configuration change. Every entry that mentions the underlying ceiling states that it is still at the default at that moment. The fix was identified precisely, in writing, at the very first occurrence, and restated with increasing specificity as the pattern continued. The one thing that never changes across seven repetitions of a correct diagnosis is the one line of configuration.

That is the evidence for the actual claim: the effectiveness of the temporary fix is precisely what removes the pressure that would otherwise force the durable one. Diagnosis is working almost perfectly — same fault, same diagnosis, same named remedy, morning after morning — which is itself the demonstration that correct diagnosis alone does not close the loop.

What the restart does not fix

A second, separate bug is present in the same service: write-lock contention in its own datastore, causing job submissions to fail outright. It is distinct from the descriptor exhaustion, not a symptom of it. It does not clear when a restart clears the descriptors, and it recurs on its own. It remains open with no named durable fix at all in the record — unlike the descriptor problem, which at least has a named fix that nobody has applied.

The restart fixes the descriptor and connection symptom specifically. It does not fix "the problem" generically, and it leaves this second fault untouched.

The trap

A working, complete, zero-risk temporary remedy is not a pause on the way to a durable fix. In this record it is the mechanism that keeps the durable fix from ever getting applied. The service recovers in about the time it takes to restart it. The ceiling stays at the default. The one-line change stays unwritten in the configuration that would raise it. Seven consecutive days of correct diagnosis and complete temporary remediation later, that is still the state of the machine.