Every Call Died at Exactly Ten Minutes

The dashboards were green. Call volume was normal, error rates were flat, nothing paged. And calls were dying at ten minutes.

Not around ten minutes. At ten minutes.

This post is about how a system can lie to you about its own health, and why a failure that happens on a schedule is telling you something very specific.

The telemetry was wrong, not the users

The first thing that made this hard is that nothing looked broken.

When a call ended, we logged it as a call ending. The media session closed, our handler ran, and the event went into the pipeline the same way it did when someone actually hung up. There was no error to catch, because from the application's point of view nothing had gone wrong — a session that was open was now closed, which is what happens to every call eventually.

So the graph said: users are hanging up. And there is no alert you can write for "users hung up," because that is the normal case.

This is the part worth internalising. A monitoring gap is not always a missing metric. Sometimes the metric exists, fires correctly, and is categorically wrong — a system failure recorded as a user's choice. Those are much harder to see, because the number isn't zero and it isn't spiking. It's just quietly mislabelled.

What actually surfaced it was the shape of the distribution. Real hangups spread out. People leave after thirty seconds, or four minutes, or twenty. When you plot call duration and see a wall at exactly one value, you are not looking at human behaviour any more.

Regularity is a fingerprint

Here is the heuristic I now reach for first:

Networks fail irregularly. Timers fail on schedule.

Packet loss, congestion, a flaky carrier, a bad handoff between towers — all of these produce messy, variable failures. They cluster, they correlate with load, they vary by region and device. What they do not do is happen at precisely the same offset from session start, every single time, for everybody.

A failure that is exactly regular has a clock behind it. So the question stops being "what's broken on the network" and becomes "what expires at this duration."

That reframing is most of the work. Once you're looking for a timer, the candidate list is short and finite, because the durations in a real-time stack are mostly specified in public documents rather than chosen at random. You can enumerate them.

For a WebRTC call going through a relay, ten minutes is not an arbitrary number. TURN allocations and channel bindings have a defined lifetime in that neighbourhood, and they have to be refreshed by the client to stay alive. If a refresh doesn't happen, the relay tears the binding down on schedule — precisely, predictably, and without telling your application anything except that the media stopped.

That matched the fingerprint exactly. Which gave me a suspect, but not yet proof.

Looking below your own logs

The problem with the suspect is that all of it happens inside the third-party media library. Our logs sat above that boundary. We could see that media stopped; we could not see what the library was doing on the wire, and reasoning about it from the outside was guesswork.

The library did have internal tracing. But turning it on is not free — in a real-time media path it is extremely chatty, it costs CPU in a loop that cannot afford to miss deadlines, and it produces volumes of output you do not want flowing out of production.

It also could not be reproduced anywhere else. That is worth stating plainly, because it is the actual reason this took as long as it did: the bug needed real relay infrastructure, real session lifetimes, and ten uninterrupted minutes. A local reproduction attempt tells you nothing if the thing you're testing only misbehaves against a real TURN server after a ten-minute allocation lifetime elapses.

So: tracing on, in production, scoped as tightly as I could make it. Narrow, short-lived, and targeted at the sessions I already expected to fail.

That is a deliberate tradeoff and I would make it again. When the symptom is invisible from outside a boundary, you either go inside the boundary or you keep guessing. Guessing is cheaper right up until it isn't.

One timer, shared

The trace showed what was happening: the refresh wasn't being sent for every channel that needed one.

The credential-refresh timer was scoped globally rather than per channel. One timer, one expiry, one schedule — for what should have been an independent lifecycle per channel. The second channel inherited the first channel's expiry, so its own refresh never came due on its own terms. The relay did exactly what the protocol says to do with a binding nobody refreshed, and tore it down.

The bug is a single-line-of-reasoning mistake, not an exotic one: an object's lifecycle timer was attached to the wrong scope. That's it. A resource that is per-channel had its refresh managed as though it were per-connection.

I want to be honest about how mundane the fix is relative to how long it took to find, because that ratio is the actual lesson. The hard part was never the patch. It was getting from "users are hanging up" to "look inside the library at channel binding refresh," which required disbelieving our own telemetry first.

We shipped it as a vendored patch rather than waiting for the fix to travel upstream and come back through a release. When a bug is cutting live sessions, the cost of carrying a local patch is much lower than the cost of waiting.

What I take from it

Suspect your instrumentation before you suspect your users. If a graph is telling you people behave with clockwork regularity, people are not behaving with clockwork regularity. Something is filing a failure under the wrong category.

Exact regularity means a clock. Before you go looking at networks, enumerate every timer in the stack that expires near the number you're seeing. Most of them are in public specifications, so this is a finite list you can actually work through.

Your logs stop at your boundary. A third-party library is a place where your visibility ends, and when the symptom is invisible from outside it, the only honest options are to go inside or to keep guessing.

Ask what scope a lifecycle belongs to. Anything with a refresh, a lease, a keepalive or an expiry has a scope, and getting that scope wrong produces failures that look like infrastructure problems and are not. It's worth checking that question explicitly whenever you touch code that manages a resource's lifetime.