Why does AI need so much memory to answer a question?
Type a short question into a chatbot: Why is the sky blue?
For a moment, nothing happens. Then the answer begins to appear, one word at a time.
Behind that pause, an industrial process has started. The question is divided into tokens, in other words units that may represent a word, part of a word or punctuation mark, and converted into numbers. The system passes those numbers through a large language model containing billions of parameters, or weights.
Put simply, AI needs so much memory because it must store the model’s weights and the working data created while processing a prompt. Generating each new token requires those numbers to be accessed quickly, often across multiple AI chips.
The model is not searching its parameters for a stored answer, nor is it relearning the subject. It is applying numerical patterns acquired during training to calculate which token is most likely to come next.
Perhaps the first token is The. Once it has been selected, the system combines it with the original question and calculates another set of probabilities. Then it does the same for sky, appears, blue and every word that follows.
What looks like writing is an extraordinarily fast loop of data retrieval, calculation and prediction.
AI memory capacity and bandwidth are different problems
A model with 100 billion parameters contains 100 billion numerical values. Depending on the precision used to store them, those values can occupy tens or hundreds of gigabytes. They cannot all fit inside the small but extremely fast memory built directly into a conventional processor.
Most are therefore stored in high-bandwidth memory, or HBM, positioned close to graphics processing units (GPUs) or other AI accelerators.
This creates two separate constraints. Memory capacity determines whether the model and its working data will fit. Memory bandwidth determines how quickly that data can reach the processor.
Think of the memory as a warehouse. It must be large enough to store all the ingredients, but its loading doors must also be wide enough to supply the kitchen. An enormous warehouse with narrow doors still leaves the chefs waiting.
During the first stage of an answer, known as prefill, the computer processes the prompt and any earlier conversation. Many prompt tokens can be handled in parallel, allowing the processor to make relatively full use of its calculating power.
The second stage, decode, produces the answer token by token. Each new token depends on what came before it, making the process largely sequential. The model cannot generate the tenth word until it has generated the ninth.
For a dense model, most or all of its weights may need to be read during every decoding step. Mixture-of-experts models activate only selected parts of the network for each token, reducing the amount of data and computation involved, although they introduce additional routing challenges.
This is how a short answer can require an enormous model to be read repeatedly.
Why AI inference can be limited by memory
Modern AI accelerators are exceptionally good at matrix multiplication, the mathematical operation at the centre of neural networks. Their processing cores can perform vast numbers of calculations simultaneously.
But those cores are useful only when data arrives in time. During token-by-token generation, an accelerator can spend more time waiting for model weights than performing arithmetic. Engineers describe this as a memory-bandwidth-bound workload: a faster processor achieves little unless the memory system can keep up.
Moving data also consumes energy. Retrieving a value from nearby on-chip memory is generally less costly than fetching it from HBM or another processor. The precise difference varies between systems, but distance comes with a price.
Repeated across billions of values, millions of prompts and thousands of simultaneous users, that price appears in electricity consumption, cooling requirements and the cost of every generated token.
The growing answer creates another burden. Transformer models preserve information derived from earlier tokens in a structure called the key-value cache, usually shortened to KV cache. This prevents parts of the previous sequence from having to be calculated again for every new token.
The cache speeds up generation but consumes memory. As the chatbot explains how sunlight is scattered by gases in the atmosphere, each additional token expands its working record of the conversation. Long prompts, reasoning models and autonomous AI agents can make that record particularly large.
When many people use the service simultaneously, every active sequence requires its own KV cache.
Large AI models extend across multiple chips
Large models are commonly divided across multiple accelerators because they cannot fit, or run efficiently, on a single device. Intermediate results must then travel between processors, servers and sometimes entire racks.
Every transfer adds delay and consumes power. Copper electrical connections remain effective over short distances, but carrying more data through them at higher speeds becomes increasingly energy-intensive.
Engineers are pursuing three broad remedies. One is to bring memory and computation closer together, reducing the distance model data must travel. Another is to move less data: quantisation uses fewer bits to represent weights, while newer techniques compress or selectively manage the KV cache.
The third is to improve the roads between chips. Optical interconnects can carry data using light for part of its journey, potentially increasing bandwidth and reducing the energy required to move each bit.
These approaches can overlap. The emerging AI computer is not simply a faster chip, but a system designed around where information is stored and how it moves.
Chip companies are taking different routes around the bottleneck
Olix is not alone in redesigning AI hardware around that movement, although its potential rivals approach the problem differently.
Groq places a large amount of static random-access memory, or SRAM, on its processor and uses compiler-controlled scheduling to make data movement predictable. Cerebras integrates computation, on-chip memory and connections across a wafer-scale processor, replacing a collection of conventional chips with one unusually large piece of silicon.
D-Matrix takes another route. Its digital in-memory computing architecture is designed to perform inference calculations close to where data is stored, reducing the traffic between separate memory and processing components.
Lightmatter and Celestial AI concentrate primarily on the connections around processors and memory. Both are developing photonic interconnect technology intended to move data through large AI systems more efficiently. As MTN explored in The race to replace AI’s copper connections, they form part of a wider effort to prevent electrical links from becoming a constraint on AI computing.
These systems are not direct equivalents. Some redesign the processor, some reduce the distance between calculation and memory, and others rebuild the network connecting them. Together, they show how competition in AI hardware is shifting from raw calculating power towards the architecture of data movement.
Olix proposes an optical AI production line
That broader engineering contest provides the context for the $312m Series B financing round raised by British chip start-up Olix on 3 August. The round valued the two-year-old company at $3.3 billion and attracted investors including Arm, Hudson River Trading and Netflix co-founder Reed Hastings.
Olix argues that using the same general-purpose accelerator for every stage of AI inference is inefficient. Its proposed X-1 platform distributes a model across many specialised chips, allowing different stages of token production to run on hardware designed for their particular demands.
The company compares the system to a factory production line. Instead of one machine repeatedly changing tools, the work travels from one dedicated station to another.
Olix plans to connect those stations through what it calls a “slow and wide” optical interconnect. Data would travel directly between chips using light rather than relying entirely on copper connections, with processors, networking and compiler designed together at rack scale.
Its first chip, DX-1, is intended for the decoding stage. Olix says portions of the model will be held in fast on-chip SRAM distributed through the system, avoiding HBM and the advanced chip packaging currently in tight supply.
For models with 100 billion parameters, the company claims DX-1 could deliver more than 10,000 tokens per second per user while achieving higher output-token throughput per watt than general-purpose chips running large batches.
The commercial test will be how much output an entire rack can produce while serving many users simultaneously. Buyers will also want to know the total power consumption, cost per token and how readily the platform can accommodate model architectures that continue to change. Until working systems are tested under realistic conditions, Olix’s proposed advantage remains unproven.
Its funding nevertheless reflects a broader wager: AI inference may be becoming large and expensive enough to justify building different machines for different stages of a model’s work.
The hidden cost of a simple AI answer
For the chatbot user, years of chip and networking development may translate into little more than a shorter pause before the first word and a faster stream of text.
For AI providers, memory capacity determines which models they can run. Bandwidth affects how quickly those models respond. Interconnects influence how efficiently the system can expand across racks. Together, they shape the hardware, electricity and cooling required to produce every answer.
By the time the chatbot has explained why the sky is blue, the machine may have moved many times more data than the finished answer contains.
A chatbot appears to think and write. Inside the data centre, much of the work is traffic.
Liked this article? You can support our independent journalism via our page on Buy Me a Coffee. It helps keep MoveTheNeedle.news focused on depth, not clicks.