Book a 30 min Demo

How Relay traced a critical API failure in under 3 minutes

R
3.0 Relay

Relay operates API gateway and networking infrastructure for platforms where request reliability is not a nice-to-have. Their systems sit between clients and the services they depend on, which means when something fails inside Relay's stack, the failure is visible immediately and everywhere. There is no graceful degradation. There is just a broken integration and a clock running.

The Problem

Relay's engineering team had solid alerting in place. They knew within seconds when something was wrong. What they did not have was a fast, reliable way to figure out where wrong was coming from. Their API gateway infrastructure sat in front of dozens of upstream services, and a failure presenting at the gateway level could originate from almost anywhere behind it. Tracing that back to a root cause required manual investigation across logs, traces, and service documentation that was rarely current enough to be trusted.

The average time from alert to identified root cause was sitting at just over 18 minutes. For a networking layer handling live traffic, 18 minutes is a long time. It meant prolonged degradation for any downstream client during an active incident, and it meant engineers were spending the majority of that window doing archaeology rather than remediation.

The deeper issue was structural. Relay's service graph had grown considerably as their platform scaled, and the internal documentation describing how services connected to each other had not kept pace. When a new engineer got paged for an incident, they were navigating a system they only partially understood, leaning on whoever else was available to fill in the gaps. That dependency on institutional knowledge was a reliability risk in itself.

How Relay Used STRAND

Relay deployed STRAND primarily to solve the gap between alert and diagnosis. The live dependency map gave their engineers an immediate view of the full service graph behind the gateway, including which upstream services a given request path touched and what each of those services depended on further back in the chain. For a team whose job is understanding why traffic is behaving unexpectedly, that map changed the starting point of every incident investigation.

The shift became concrete during an incident roughly six weeks after deployment. A critical API failure began surfacing at the gateway layer with no obvious cause. In previous incidents of similar complexity, the team would have spent the first fifteen to twenty minutes working backward through logs and tracing spans manually, cross-referencing service documentation to build a picture of what was connected to what. This time, an engineer opened STRAND, located the affected path in the dependency map, and traced the failure back to a degraded upstream authentication service within three minutes of the alert firing. Remediation started immediately.

That incident became the internal reference point for how Relay's team talked about the before and after. Three minutes from alert to root cause, on an incident that would previously have taken the better part of twenty minutes just to locate. The difference was not a faster engineer. It was having the system already mapped and queryable at the moment it was needed.

Relay also adopted STRAND's health monitoring across their full upstream dependency graph, giving the team visibility into service degradation before it reached the gateway layer and became visible to clients. Several potential incidents were caught and addressed before any traffic was affected.

3 min
To root cause on critical failure
83%
Reduction in time to diagnosis
12+
Incidents caught before client impact

The Results

The 3-minute trace on that critical API failure was not a one-time result. It reflected a repeatable change in how Relay's team approaches every incident. Mean time to diagnosis dropped by 83 percent across the quarter following deployment. Engineers stopped spending the first stretch of every incident reconstructing system topology and started spending it on the actual problem.

Proactive health monitoring across Relay's upstream dependency graph produced a secondary result the team had not fully anticipated. Over the following months, more than twelve incidents were identified and resolved before any degradation reached the gateway layer. Clients never saw them. For a networking layer where client-visible failures carry direct SLA implications, that number had meaningful business impact beyond the engineering metrics.

New engineers joining the team now onboard against a live, accurate map of the system they are responsible for operating. The institutional knowledge problem that had made incident response dependent on who was available did not fully disappear, but it became substantially less acute. Any engineer on the team can open STRAND and understand the system clearly enough to begin investigating an incident independently.

Looking Ahead

Relay's focus going forward is on using STRAND's dependency visibility to get ahead of failures before they require any response at all. The health monitoring they have in place across their upstream graph is already producing that outcome in patches. The next step is extending that coverage across the full request surface and building pre-deploy impact analysis into their standard release workflow, so that changes to any service in the chain get scoped against the full downstream picture before they ship.

"Three minutes to root cause on a live API failure used to sound like an unrealistic benchmark. Now it is just what happens when you have the system mapped. The incident that used to take us twenty minutes to locate took one engineer three minutes to trace end to end. That is the whole story." Staff Engineer, Relay
Kinecta Wyre