Cerebras pipelines trillion‑parameter models across wafer‑scale chips
Cerebras’ WSE‑3T wafers use 16 GB per layer and linear‑attention tricks to fit large models across multiple chips, delivering frontier‑level benchmark scores via an RDMA‑based pipeline.
Cerebras explains that each WSE‑3T wafer provides 44 GB of on‑chip SRAM, enough to host a single 16 GB dense or linear‑attention layer. Because trillion‑parameter frontier models require terabytes of weights, the system distributes layers across several wafers, keeping one dense layer per wafer and reserving the remaining SRAM for its key‑value (KV) cache. Linear‑attention layers avoid a KV cache by storing only a small, fixed‑size state per request, allowing two such layers (2 × 16 GB = 32 GB) on a wafer.
The Cerebras compiler maps model layers to wafer regions, and the runtime routes token activations between wafers over RDMA‑enabled Ethernet, incurring roughly 2 µs per hop. By keeping the pipeline full—different requests occupying different stages—the architecture avoids idle time and achieves low‑latency inference, scoring 88.3 % on Terminal‑Bench 2.1 and 91.2 % on BrowseComp, comparable to leading proprietary models.
Developers can access the system today via Cerebras’ free cloud API, but workloads must respect the 16 GB‑per‑layer memory constraint and accept the linear‑attention trade‑off for cache‑free layers when scaling across wafers.
Our take. Engineers can run trillion‑parameter models with frontier‑level performance by partitioning them into 16 GB layers and leveraging linear‑attention, but must orchestrate a multi‑wafer pipeline rather than rely on a single chip.
- cerebras.aiShared Discussion