# How Chips Are Using On-Chip Neural Networks for Real-Time Inference

An episode of [The Hardware Podcast with Fexingo: Chips, Devices, and Electronics Engineering Conversations](https://invisiblenetwork.ai/podcast/1896799982), 13 Jul 2026, 8:15.

Audio: https://audio.fexingo.com/business/the-hardware-podcast/episode-0108.mp3

This episode of The Hardware Podcast explores the emerging trend of integrating neural network accelerators directly onto chips for real-time inference at the edge. Lucas and Luna discuss how on-chip neural networks enable low-latency AI processing without cloud dependency, using the example of Syntiant's NDP120 chip that achieves 99% accuracy in keyword spotting while consuming under a milliwatt of power. They examine the architecture trade-offs between digital and analog compute-in-memory approaches, and how companies like Mythic and GreenWaves Technologies are pushing the boundaries of ultra-low-power inference for wearables and IoT devices. The conversation also touches on the implications for privacy and the future of distributed intelligence. #OnChipNeuralNetworks #EdgeAI #TinyML #Syntiant #Mythic #GreenWavesTechnology #ComputeInMemory #KeywordSpotting #UltraLowPower #Wearables #IoT #AIAccelerator #NeuralNetwork #RealTimeInference #Technology #Hardware #FexingoBusiness #BusinessPodcast Keep every episode free: buymeacoffee.com/fexingo

## Transcript

Lucas: You know the phrase 'there's no cloud, it's just someone else's computer.' Well, what if the computer doing all the heavy lifting is smaller than your fingernail and sits right next to the sensor?
Luna: You're talking about on-chip neural networks — doing inference locally instead of shipping data to the cloud.
Lucas: Exactly. And it's not just a lab curiosity anymore. There are production chips today that can run a small neural network on a few milliwatts, even microwatts, and they're showing up in everything from smart earbuds to industrial vibration sensors.
Luna: Before we dive into the architecture, quick honest thing — we keep this show ad-free and listener-supported. A handful of people chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds us making this many episodes. So if the show's been useful to you, that's where to find it.
Lucas: Yeah, and we really appreciate that. So let's talk about one concrete example: the Syntiant NDP120. It's a neural decision processor that does keyword spotting — think 'Hey Siri' or 'Alexa' — but it consumes under a milliwatt. That's roughly one-thousandth of what a typical smartphone SoC uses for the same task.
Luna: And it's not just about power. Latency matters too. If you have to send an audio clip to the cloud and wait for a response, you get a half-second delay. On-chip inference is essentially instantaneous.
Lucas: Right. The NDP120 achieves 99 percent accuracy on the standard Google Speech Commands dataset while drawing less than 140 microwatts in active operation. That's a big deal for battery-powered devices that need to always be listening.
Luna: So how do they do it? I mean, a typical neural network requires multiply-accumulate operations — lots of them. How do you fit that on a tiny chip?
Lucas: There are two broad approaches. One is to design a dedicated digital accelerator with highly optimized multiply-accumulate units, often using reduced precision — 8-bit integers instead of 32-bit floats. That's what Syntiant does. The other approach is analog compute-in-memory, where the weights are stored in non-volatile memory cells like RRAM or flash, and the multiplication happens physically using Ohm's law.
Luna: Analog compute-in-memory — that's fascinating. Instead of shuttling data between memory and a compute unit, you do the math right where the weights live.
Lucas: Exactly. Mythic is a company pushing that approach. Their M1076 chip uses analog compute-in-memory with flash-based synapses. It can perform matrix-vector multiplications in a single step, dramatically reducing energy. They claim up to 35 trillion operations per second per watt. For comparison, a typical GPU might get around 10 trillion operations per watt, and that's at much higher power.
Luna: But analog has its own challenges — noise, temperature variation, manufacturing tolerances. How do they maintain accuracy?
Lucas: That's the trade-off. Analog compute-in-memory is inherently less precise than digital. But for inference tasks like image classification or keyword spotting, you don't need floating-point precision. The networks are trained to be robust to small errors. Mythic backs that up with error correction and calibration circuits on-chip. They claim their chips achieve accuracy within 1 percent of a full digital implementation.
Luna: So it's a classic engineering trade-off: precision versus energy efficiency.
Lucas: Right. And that trade-off is driving a whole ecosystem of startups. GreenWaves Technologies, based in France, makes the GAP9 chip — a risc v based processor with a hardware neural network accelerator. It targets hearables and smart sensors. They publish benchmarks showing 10 times better energy efficiency than a Cortex-M7 running the same neural network.
Luna: I keep coming back to the use case though. Who actually needs on-chip neural networks? Is this just a solution in search of a problem?
Lucas: I'd argue the opposite. The problem is already here. Every smart home device, every wearable, every industrial sensor that wants to do audio or visual recognition currently faces a choice: either stream everything to the cloud, which costs bandwidth and raises privacy concerns, or use a low-power microcontroller that can't run complex models. On-chip neural networks fill that gap.
Luna: Privacy is a big one. If your smart speaker processes voice commands locally, the audio never leaves the device. That eliminates a whole class of privacy risks.
Lucas: Exactly. And it's not just voice. Think about a security camera that can detect a person versus a moving branch. With on-chip inference, it can decide locally whether to send an alert or record video, rather than streaming everything to a server. That saves bandwidth and storage.
Luna: So what's the bottleneck? Why aren't we seeing these chips everywhere yet?
Lucas: Part of it is the software toolchain. It's not enough to build a great hardware accelerator. You need a compiler that can take a model trained in TensorFlow or PyTorch and map it efficiently onto the chip. Companies like Syntiant and Mythic have invested heavily in their SDKs. Syntiant's Neural Network Studio lets you train a model online and deploy it directly. But it's still not as seamless as running inference on a CPU or GPU.
Luna: And there's the chicken and egg problem: engineers are familiar with traditional MCUs, but not with these specialized NPUs. There's a learning curve.
Lucas: Right. But the industry is converging on some common ground. The risc v ecosystem is helping, because you can have a standard core and a custom accelerator side by side. ARM is also getting into the game with its Ethos NPU series. So the tooling is improving fast.
Luna: Let's talk numbers. Can you give a sense of scale? How many of these chips are shipping?
Lucas: Syntiant alone has shipped over 100 million chips as of last year. Most of them go into consumer audio products — earbuds, hearing aids, smart speakers. That's a real volume. Analog compute-in-memory from Mythic is earlier stage, but they have design wins in industrial predictive maintenance and smart retail.
Luna: So the market is real. And it's growing. What's the next frontier?
Lucas: I think the next push is toward larger models — not just keyword spotting, but natural language understanding and visual scene analysis on very low power. There's research on spiking neural networks, which are even more energy-efficient because they only fire when events happen. But that's further out.
Luna: And on the architecture side, there's work on in-sensor processing — putting the neural network right on the image sensor or microphone diaphragm, so the data never even leaves the transducer.
Lucas: That's the ultimate edge computing. Companies like Prophesee are doing that with event-based vision sensors that only output changes in the scene, then process them with a tiny on-chip network. It's incredibly efficient for applications like drone obstacle avoidance or industrial safety.
Luna: So the pattern is clear: move intelligence as close to the data source as possible. And the chip industry is building the pieces to make that happen.
Lucas: Exactly. The on-chip neural network is one of those enabling technologies that quietly changes what's possible. A few years from now, we'll take for granted that our devices understand us locally, without any cloud round-trip.
Luna: And that's a shift that touches everything from privacy to power consumption. It's a fascinating moment in hardware.
Lucas: I'd say it's the beginning of truly distributed intelligence.

Shows are listed from Apple Podcasts' charts and directory, and Podcast Index where it is used. Every show and episode belongs to its publisher and plays from the publisher's own feed; this site hosts none of them.