Three public figures do a lot of work in the way we describe this market, and we would rather readers understand exactly what each measures than take any of them on trust. One is a count of tokens crossing a marketplace. One is a benchmark of cost per token on distributed hardware. One is an older count of tokens processed inside a single company. None of them is our number. All three are worth reading slowly.
The demand figure: 250 trillion a month
The largest independent routing marketplace publishes its own throughput, and that series is the cleanest public read on inference demand anyone has. It now runs at about 250 trillion tokens a month, which is three quadrillion a year, from about 0.4 trillion a month two years earlier. The growth rate implied is roughly five-fold every six months, or 625 times over two years.
What it says is narrow and strong. This is not a forecast and not a survey: it is one marketplace's own published throughput at its own published rate, and it counts tokens that were actually served by somebody, at a price somebody paid.
What it does not say: that the rate holds. Nothing compounds five-fold every six months forever, and a marketplace's share of a growing market moves for reasons that have nothing to do with the market. Token counts also inflate as models reason at length before answering and as context windows grow, so part of any multiple is tokens per answer rising rather than answers rising. We use the figure for one purpose: to establish that the demand clock runs in months while the supply clock, grid connections, runs in years.
The NVIDIA figure: 52.8% and 76.1%
On 17 March 2026 the NVIDIA Technical Blog published results from a benchmark run with Comcast on what they called an AI grid: the same small voice model served on four RTX PRO 6000 cards at distributed points of presence, compared against serving it from a centralised cluster. The reported reduction in cost per token was 52.8% at baseline load and 76.1% at burst.
What the figure says is narrow and useful. For a small model, on modest hardware, serving inference close to where requests arrive cost roughly half as much per token as serving it centrally under steady load, and roughly a quarter as much under peak load. The gap widens at burst because a central cluster has to be provisioned for the sum of everyone's peaks and sits partly idle the rest of the time; distributed nodes absorb their own local peaks, so less provisioned capacity goes unused. The arithmetic behind that is utilisation, not magic.
What the figure does not say matters just as much. It says nothing about large models. It says nothing about Indian power prices or about the hosting arrangements we use. It was run on a workstation-class card, not on the node we cost our mills on. And it describes one operator's network under one traffic pattern; another operator with flatter traffic would see a smaller burst gap.
We cite it because it is the clearest published statement, from the vendor with the least incentive to make it, that distributing inference lowers the cost per token rather than raising it. We do not cite it as our result. Our promise to buyers is sovereignty first, then a comparison: quality and total cost, compared on their own workload before they decide. We publish no number for any saving, and we do not quote this benchmark as one.
The older figure: 9.7 trillion to 480 trillion
At the Google I/O keynote on 20 May 2025, Sundar Pichai said that Google was processing over 480 trillion tokens a month across its products and APIs, up from 9.7 trillion a month a year earlier: a factor of roughly fifty in twelve months.
We keep the citation and no longer lead with it. It is one company's internal count, most of it not sold to anyone, and a year-on-year multiple from a single vendor is a weaker instrument than a marketplace's continuous published series. It remains useful as corroboration: two independent counts, on different bases, both say the same thing about the direction and the order of magnitude.
Why placement beats parallelism
The reason distribution works for inference and fails for training is bandwidth, and the numbers are not close.
| Link | Bandwidth | Where |
|---|---|---|
| NVLink or Infinity Fabric | 600 to 1,800 GB/s | Inside one mill |
| InfiniBand NDR | about 50 GB/s | Inside one hall |
| City fibre at 1 Gbps | about 0.125 GB/s | Between mills |
Inside a mill, the accelerators share a fabric that moves hundreds of gigabytes a second. Between two buildings on city fibre, the link moves an eighth of a gigabyte a second: four orders of magnitude less. Any job that needs its parts to talk to each other mid-step has to stay inside one box.
| Workload | Across buildings? |
|---|---|
| Batch inference, generation, rendering, embeddings | Fine. Only inputs and outputs cross the link; a 1 Gbps link carries 7 to 9 TB a day |
| LoRA fine-tune, about 40 MB per step | Marginal. Viable with gradient accumulation |
| Full fine-tune of a 7B model, about 14 GB synced per step | Not viable. 112 seconds per step on the link alone |
14 GB per step ÷ 0.125 GB/s = 112 s of synchronisation per step
0.125 GB/s × 86,400 s ≈ 10.8 TB per day raw; 7 to 9 TB at realistic utilisationSo the product is scoped to what physics allows. Many independent jobs can be placed across the fleet. One tightly coupled job stays inside one machine, because no scheduler can fix the case where it would have to cross two buildings. Placement, not parallelism. The mill is the unit of contiguity: real fabric inside, city fibre between, and a boundary any technical buyer can verify.
That is also why the memory in one mill matters more than the count of machines. A mill holds 1,536 GB, enough for a 671-billion-parameter model, so the class of job that has to stay in one box mostly fits in one box.
What cost per token depends on in our model
Given that the job fits in the box, the cost per token from a mill is set by five things, in roughly this order.
Utilisation. A mill at 60% of sellable capacity, our modelled steady state, spreads its fixed costs over three-fifths of the hours; the same mill at 30% costs twice as much per token. Nothing else in the model moves the number as far. This is why the after-hours cooling question matters so much: a mill that cannot run overnight cannot reach 60%.
The design choice. The engine can choose the silicon for a room as well as the layout. What that choice does to the delivered cost per token is part of our cost model, it is not yet measured, and we print no figure for it. It only matters at fleet scale: choosing silicon per room is only rational across many rooms.
Power. We model a blended ₹9.0 per kWh with a band of ₹8 to ₹11 by state, reimbursed to the host at actuals at the host's own tariff. It is a real cost and it is never one flat number fleet-wide.
Silicon, measured as dollars per terabyte per second of memory bandwidth, because for inference the bandwidth is what serves tokens. A ten-kilowatt mill lands at $289,400, and the design is card-agnostic inside a frozen envelope, so the number tracks the market rather than a part.
The host floor, which is fixed per mill per year and is the reason a building says yes.
What the cost per token does not depend on is the GPU-hour list price. The IndiaAI compute portal lists H100 SXM capacity at ₹153 per GPU-hour on demand and ₹117 reserved, and any buyer will quote those numbers at us. They are the right price for shared, resident, allocation-gated capacity. They are not the price of a machine a buyer can name in a contract, in a building the buyer chose, that nobody else touches. What we offer is a comparison on delivered output: quality and total cost on the buyer's own workload, set against what the buyer runs today, not an hourly rate.
What we will publish
The published figures say that distributing inference lowers cost per token when the work fits in one box. They do not say by how much it will for our fleet, on our silicon, at our power prices. That number can only be measured on real work, installation by installation, and it is what a comparison of quality and total cost on a buyer's own workload rests on. Until it is, the honest position is the one above: a demand series that explains the urgency, a benchmark that points the right way, and a physics table that says where the line is.