Edge-AI silicon is not one kind of part. It runs from a standalone accelerator that sits beside a host, through application processors that fold a neural engine onto the same die, down to microcontrollers small enough to spend most of their life asleep. These parts overlap on a spec sheet and behave nothing alike in a product. A number like TOPS, total operations per second, makes them look comparable on a distributor page; in a design they answer different questions. The choice starts with the model and the power budget, not with the tier a vendor files the part under.

What the chip has to do narrows it quickly. A camera running object detection at thirty frames a second needs sustained throughput and a memory path wide enough to keep the arithmetic fed without stalling. A battery sensor that wakes a few times a minute to classify a sound needs microamps at idle and does not care about frame rate at all. The same word, inference, covers both jobs, and almost nothing else about the two designs is shared. A part chosen for one is usually the wrong starting point for the other, so the range below is organised by where the part sits in the system, not by how fast it runs.

So what follows is a map, not a ranking.

When a dedicated part earns its place

A general-purpose processor can run a small model in software, and for an occasional inference that is often the right answer: no extra part, and nothing new to keep in supply. The question is what it gives up to do it, namely the cycles it owes the rest of the application and a current draw that climbs with every layer it evaluates. Working out where a dedicated accelerator earns its place over a faster general processor turns on duty cycle far more than peak rate. A part that classifies once a second can usually stay on its host and spend the time between inferences doing real work; one that runs a network continuously, frame after frame, is the case that pays for dedicated hardware. The line between those two patterns is where the decision sits, and a faster CPU does not move it, because the problem is the energy and time spent doing the work, not the raw speed of doing it once. A low duty cycle also rarely repays the fixed cost of adding a chip, since the board area and the part to source exist whether the accelerator runs once an hour or ten times a second.

Accelerators that attach to a host

When a host is already chosen and carries the rest of the system, a separate accelerator can take only the neural work and leave everything else where it is. The host keeps the operating system, the application, and the interfaces; the accelerator holds the model and a fast path into it, usually over PCIe, USB, or M.2. The trade is one more part to place, route, and keep in stock, set against a host that no longer stalls on every frame it has to push through the network. On a design that already carries a capable processor for other reasons, this is often the lowest-risk way to add vision.

Vision is where the split pays off most, because the data rate is high and the network is heavy. Hailo-8 handling the vision inference next to a conventional processor lets a camera pipeline run detection or segmentation at real frame rates without loading that work onto the host, and it does so inside a power envelope a GPU of similar throughput would not hold. What it asks back is a PCIe or M.2 lane on the carrier board and a thermal path that stays adequate when the model runs flat out for hours, not the few seconds a bench demo lasts. The accelerator is only as useful as the host's ability to feed it frames, so the link between the two is part of the design rather than an afterthought.

Processors with the NPU on board

The other way to get neural throughput is to choose a processor that already carries it. An application processor with an onboard NPU runs Linux, drives the cameras and displays, and keeps the model on the same die, so the bill of materials does not grow and there is no host-to-accelerator link to lay out. The catch is that the NPU is whatever the vendor built into the part, and its tools and supported operators decide how much of a given model lands on the accelerator rather than falling back to the CPU at a fraction of the speed. Two parts with the same TOPS can post very different real throughput once the unsupported layers are counted.

RK3588 as an edge-AI processor with an onboard NPU brings a high core count and a capable neural unit for a box that can spend a few watts and wants a lot of compute for the money. It suits a design running a full Linux stack that wants headroom for several cameras or a heavier model. Its toolchain asks for more bring-up time than a part from a larger vendor support operation, and that effort, not the unit price, is the real cost behind the attractive headline figure.

i.MX 8M Plus, which folds an NPU into the application processor, trades raw throughput for a long industrial supply commitment, a wide operating-temperature range, and the documentation and lifecycle guarantees that come with that class of part. On a product meant to ship for ten years into an industrial socket, the promise that the chip will still be made and supported is often worth more than a higher frame rate nobody was short of.

TDA4VM in automotive and machine-vision edge inference sits toward the functional-safety and multi-sensor end, carrying hardware blocks for the vision front end and the signal processing that a generic NPU does not. That makes it a strong fit for a driver-assistance or robotics design pulling several cameras and a radar together, and a weak fit for a single-camera product that pays for the complexity without using it.

Choosing among the three is less about the TOPS figure than about which support, lifecycle, on-die blocks, and toolchain maturity fit the product. The part with the largest number on the slide is rarely the one that reaches production; the one whose ecosystem matches the team usually is.

When the MCU is the AI

At the low end the model and the controller are the same chip. There is no host, no Linux, no separate accelerator. The part wakes, runs a small network, acts on the result, and goes back to sleep, often on a coin cell that has to last for years. The model has to fit in a few hundred kilobytes of on-chip memory and run in real time on a core measured in tens or low hundreds of megahertz, which rules out anything heavy and rules in keyword spotting, simple gesture and vision tasks, and signal classification. The toolchains for these parts expect a model trained with that memory limit in mind, so the constraint shapes the network long before it reaches the chip.

The parts that do this are not interchangeable, and what separates them is energy and integration rather than raw speed. MAX78000 for ultra-low-power neural inference puts a dedicated convolutional accelerator beside the microcontroller core, so a small network runs at a fraction of the energy the core would spend doing the same multiply-accumulate work itself; on a device that infers thousands of times a day off a battery, that energy-per-inference number sets the battery life, and it is not the figure printed on the front of the data sheet. K210 as a low-cost RISC-V AI MCU takes the value end, pairing a neural unit with RISC-V cores at a price that suits a high-volume consumer build where the model is light and the unit cost dominates the bill; its tooling is thinner, which is the trade made for that price. NDP120 for always-on voice neural decision making is narrower again, built to listen for wake words and sound events at an idle current low enough to sit on permanently beside a larger system that stays asleep until the small part hears something worth waking it for; a general MCU running the same model would flatten the battery just keeping itself awake to listen. ESP32-S3 running light inference on its vector instructions is the pragmatic case rather than the purpose-built one: not a dedicated AI chip, but a connected microcontroller many teams already have on the board, with enough vector math to run a light model without adding a part at all, which makes it the cheapest neural inference in the range when the model is small enough to fit. Where any one of these fits the model, it takes a whole tier of parts off the board.

The constraint on all of them is memory and clock, and it arrives sooner than the headline suggests. A model that runs in the lab on a development kit with room to spare can still be too large for the production part once the real input buffers, the communication stack, and the rest of the firmware are sharing the same few hundred kilobytes. Sizing that early, before the part is locked, saves a painful redesign late in the schedule.

What actually decides it

The model and the duty cycle set the floor. Everything above that floor is supply and tools, and that is where most of these choices are won or lost, long after the benchmark has been forgotten.

A part with no second source is a single point of failure in the bill of materials, however good its numbers look in isolation. On a product with a long life the question is not only whether the part performs now, but whether it can still be bought in volume in three years, and whether a pin-compatible alternative exists if it cannot. That is a sourcing decision as much as an engineering one, and it is easier to answer while the schematic is still open.

A thin toolchain costs real time at bring-up. A part that needs hand-tuning to get a model onto its NPU, or that quietly falls back to the CPU for operators it does not support, can erase the throughput advantage that made it attractive, and none of that surfaces until the model is chosen and the schedule is already set. The maturity of the conversion tools and the size of the community around them are worth as much as the silicon when a real deadline is involved.

The TOPS figure is the easiest thing to compare across parts and the least likely to decide whether the design ships. The part that wins is usually the one whose power, supply, and tools fit the product, sitting somewhere in the middle of the range rather than at the top of it.