Groq

Groq designed its own inference chip instead of renting someone else's. The LPU keeps model weights in on-die SRAM, roughly 500MB per chip with no HBM at all, which removes the memory bandwidth wall that caps how fast a GPU can decode tokens one at a time. The practical effect is generation speeds several times a GPU serving identical weights, and the advantage is largest on smaller models where the whole thing fits across a rack's SRAM.

The use case is anything with something waiting on the other end. Voice pipelines, where the whole round trip has a few hundred milliseconds before it feels broken. Agent loops that fire eight or ten model calls before a user sees a single word. High-volume classification and routing, where $0.05 per million input tokens makes per-request cost round to nothing. Teams running these workloads on GPU-backed APIs usually discover that latency, not model quality, is what is killing the product.

Two things should keep it out of a critical path. The catalogue is open weights only, so anything needing frontier reasoning has to route somewhere else anyway. And the corporate picture is genuinely unsettled: the Nvidia deal removed the founder, the president and any architectural exclusivity, and the FTC signalled in January 2026 that it would look at this class of licence-plus-hire arrangement as a merger in disguise. The API is live, performance is unchanged and no end-of-service date exists, but that is a description of today rather than a guarantee about next year.