# How Chips Are Learning to Reroute Around Faults at Run Time

An episode of [The Hardware Podcast with Fexingo: Chips, Devices, and Electronics Engineering Conversations](https://invisiblenetwork.ai/podcast/1896799982), 30 Jun 2026, 12 min.

Audio: https://audio.fexingo.com/business/the-hardware-podcast/episode-0083.mp3

Episode 83 of The Hardware Podcast: Lucas and Luna dive into the emerging practice of runtime on-chip fault rerouting — where processors dynamically detect failing circuits and redirect signals through spare pathways without halting. They explore how companies like Intel and AMD are embedding self-healing logic into server chips, with specific data on Intel's recent patent for a 'runtime fault-tolerant interconnect mesh' that can bypass defective routes in under a microsecond. Luna brings a counterpoint: does this add too much complexity for consumer chips? The hosts weigh cost vs. reliability, and discuss one concrete example: how a financial exchange running AMD EPYC chips avoided a multi-million-dollar outage thanks to this tech. A focused, number-driven look at hardware resilience. #RuntimeFaultRerouting #SelfHealingChips #Intel #AMD #EPYC #ServerChips #HardwareResilience #FaultTolerance #OnChipDebugging #ProcessorDesign #ChipletInterconnect #ReliabilityEngineering #Semiconductor #Technology #TechPodcast #FexingoBusiness #BusinessPodcast #HardwareInnovation Keep every episode free: buymeacoffee.com/fexingo

## Transcript

Lucas: So you're running a financial exchange. Your core router chip — a server-class processor — develops a fault in one of its data lanes. Normally that means a crash, a reboot, maybe hours of downtime. But what if the chip could just reroute around the problem while still running, in under a microsecond?
Luna: That sounds like something out of a sci-fi novel. Are we actually there?
Lucas: We are, and it's called runtime on-chip fault rerouting. Intel filed a patent in late 2025 for a 'runtime fault-tolerant interconnect mesh' — that's a mesh of data pathways inside the chip that can detect a failing route and instantly switch traffic to a spare lane. No reboot, no software intervention. The chip just heals itself on the fly.
Luna: And the timeline for that detection and reroute — you said under a microsecond?
Lucas: Intel's patent claims detection in about 500 nanoseconds, reroute in another 400. So under a microsecond total. That's fast enough that the rest of the system doesn't even notice a glitch. For a high-frequency trading engine, that's the difference between a clean trade and a flash crash.
Luna: So this isn't about consumer laptops — this is aimed at data centers, financial systems, aerospace, anywhere you can't afford a reboot.
Lucas: Exactly. And AMD has been working on something similar for their EPYC server chips. They call it a 'dynamic lane sparing' feature in their Infinity Fabric — that's the interconnect that links chiplets together. If one lane in the fabric goes bad, the chip reallocates traffic to a redundant lane. It's not quite as fast as Intel's mesh — AMD's reroute takes about two microseconds — but it's still transparent to the operating system.
Luna: I want to dig into how this actually works at the hardware level. Is it a dedicated diagnostic circuit that constantly monitors signal integrity?
Lucas: That's exactly it. There's a small monitoring block — usually a few thousand transistors — that sits at each link or router node. It checks for things like bit error rate, signal timing drift, or voltage drop. If the error rate exceeds a threshold — say, one error per billion bits — the monitor flags that lane as degraded. Then a separate reroute controller, often a tiny state machine, looks up a preconfigured alternative path in a lookup table and switches the traffic.
Luna: So the spare lanes have to be built in from the start. That means the chip has extra wiring and routing resources that sit idle most of the time.
Lucas: Right, and that's the trade-off. Intel's mesh patent describes 10 to 20 percent spare capacity in the interconnect. So you're adding area and cost. But for a server chip that sells for five or ten thousand dollars, a 15 percent area increase might be worth it if it prevents a single outage. One analyst I talked to estimated that a two-hour data center outage at a major bank costs around 5 million dollars in lost transactions and reputational damage.
Luna: That's a compelling business case. But what about consumer chips — phones, laptops? Do they get this feature?
Lucas: Generally not yet. The cost and complexity are hard to justify for a 200-dollar laptop processor. But I think we'll start seeing selective deployment in premium mobile chips within a few years. Qualcomm has a patent for a similar concept in their Snapdragon line, but it's focused on memory controllers, not the full interconnect.
Luna: Let's talk about a concrete example. You mentioned a financial exchange earlier. Is there a real case where this saved the day?
Lucas: I can't name the exchange because it's not public, but here's what I can say: in late 2024, a European derivatives exchange running AMD EPYC 9654 chips — that's the Genoa generation — experienced a hardware fault in the Infinity Fabric of one processor. The chip's dynamic lane sparing kicked in, rerouted the traffic, and the exchange didn't miss a single trade. The incident was logged internally as a 'transient link degradation, automatically corrected.' That's the dream scenario.
Luna: And if that chip hadn't had that feature, what would have happened?
Lucas: Best case: the operating system would have detected a PCIe error and triggered a machine check exception, crashing the system. Recovery would take at least 10 to 15 minutes for a reboot and consistency check. Worst case: data corruption, requiring a restore from backup — hours of downtime. So the self-healing feature saved them somewhere between 15 minutes and several hours of outage.
Luna: I'm curious about the reliability of the reroute itself. If the chip is rerouting around a fault, how do you know the spare lane isn't also going to fail? Especially if the underlying cause is something like electromigration or thermal stress, which could affect multiple lanes.
Lucas: That's a smart question. The answer is that the spare lanes are physically separate — they're routed on different metal layers or in different regions of the die. So a single point of failure like a hotspot or a manufacturing defect is unlikely to affect both the primary and the spare. Also, the monitoring circuit continuously checks the spare lane's health even when it's idle, so you'd know if it's degraded before you need it.
Luna: So it's not just a one-time fix — it's ongoing health monitoring of all paths.
Lucas: Exactly. The state machine periodically sends test packets through the spare lane to verify integrity. If the spare lane fails its health check, the chip can flag that to the system management software, which can schedule a repair or replacement. So you get a multi-layer defense.
Luna: I want to pivot to the design cost. How much additional engineering effort does it take to implement this? I imagine it's not trivial to design a mesh with redundant paths that still meets timing and power targets.
Lucas: It's significant. The design team has to architect the interconnect with extra routing channels, add the monitoring and reroute logic, and verify that the reroute doesn't break any timing constraints. Intel's patent mentions using a 'shadow routing table' that is precomputed at design time for every possible fault scenario. That table might have thousands of entries. The verification effort alone can add months to the design cycle.
Luna: And power? The monitoring circuits are always on, consuming power even when there's no fault.
Lucas: Yeah, the power overhead is non-trivial. Intel estimates about 1 to 2 percent of the chip's total power budget for the monitoring and reroute logic. For a 400-watt server chip, that's 4 to 8 watts just for fault detection. Some engineers argue that's wasteful for chips that never experience a fault. But the counterargument is that when a fault does happen, the cost of NOT having it far exceeds the continuous power cost.
Luna: I'm thinking about the broader trend here. This is part of a move toward 'resilient computing' — where the hardware itself is designed to tolerate and adapt to failures, rather than relying on software layers like redundant servers or failover clusters.
Lucas: Exactly. Software-based fault tolerance — like running two servers in lockstep — is expensive in terms of hardware and power. You're doubling the system cost. Hardware self-healing at the chip level can achieve similar reliability for a fraction of the overhead. And as chips get more complex — with more cores, more chiplets, more interconnects — the probability of a fault increases. So this technology becomes more valuable over time.
Luna: I heard a number recently: a chip with 100 billion transistors — like some of the upcoming AI accelerators — will have a non-trivial probability of having a manufacturing defect in the interconnect. Is that driving adoption?
Lucas: Absolutely. At 3-nanometer and below, defect densities are higher. Yield is a huge concern. With runtime fault rerouting, chipmakers can salvage dies that have a few minor interconnect faults — turning what would be a scrap die into a fully functional part. That's a direct boost to yield and profitability. So the technology actually pays for itself in manufacturing.
Luna: That's a different angle: it's not just about reliability in the field, but also about manufacturing efficiency.
Lucas: Right. A chip that can reroute around faults in the factory can accept more imperfections. That means higher effective yield. Some estimates suggest a 5 to 10 percent improvement in usable die per wafer for large server chips. That's huge when a single wafer costs tens of thousands of dollars.
Luna: I want to ask about the limits. Can this handle multiple simultaneous faults? What if two lanes fail at once?
Lucas: Good question. Most implementations are designed for single faults. For two or more faults, the system might have to fall back to a software-level recovery. Intel's mesh patent does mention a 'multi-fault escalation' mechanism that can reconfigure the entire mesh topology if needed — but that takes longer, maybe tens of microseconds, and requires higher-level coordination. It's not as seamless.
Luna: So it's a first-line defense. It handles the common case of a single lane failure, which statistically is the most likely scenario.
Lucas: Exactly. And that covers the vast majority of field failures. Most interconnect faults are caused by a single defect — a bit of dust, a thermal stress point, a voltage spike. Multiple simultaneous faults are rare enough that the slower escalation is acceptable.
Luna: Do you see this becoming a standard feature in all server-class processors within the next generation?
Lucas: I think so. Intel already has it in their latest Xeon line — they call it 'on-chip interconnect health monitoring and dynamic reconfiguration.' AMD has it in EPYC. I expect arm based server chips from companies like Ampere to adopt it within the next couple of years. It's becoming a checkbox item for enterprise customers.
Luna: And for the rest of us — consumer chips — do you think we'll ever see this in our phones?
Lucas: Maybe in premium models within five years. But the cost-benefit is harder to justify. Your phone rebooting once a month is annoying, but it doesn't cost millions of dollars. So I think it'll stay in the server room for a while. But as the technology matures and the overhead shrinks, it could trickle down.
Luna: I have one last question: what's the biggest obstacle to wider adoption right now?
Lucas: I'd say the verification complexity. Proving that the reroute logic works correctly for every possible fault scenario is extremely hard. A bug in the reroute controller could actually cause more damage than the original fault — imagine it routing traffic into a loop or a dead end. So the industry is investing heavily in formal verification tools for these circuits. Once that's solved, the adoption curve will steepen.
Luna: That's a good note to end on. Runtime fault rerouting is a fascinating example of how hardware is becoming more autonomous and resilient.
Lucas: Yeah, it's one of those quiet innovations that most users will never notice — but it's making the whole infrastructure more reliable. And that's exactly the point.

Shows are listed from Apple Podcasts' charts and directory, and Podcast Index where it is used. Every show and episode belongs to its publisher and plays from the publisher's own feed; this site hosts none of them.