Private vs public AI cost estimator

Public APIs charge by the token. Private AI charges a flat rate for the machine, regardless of how much you generate.

• The public figure is the month's requests at each model's list price per million tokens, split across the models you pick. The private figure is the machines the busiest hour needs, at their on-demand or purchase price, plus electricity for a bought machine and the engineer time to keep them running.
• The verdict is the difference between the two. Below the volume where they cross, the API is cheaper because you pay only for what you use; above it, the machines' fixed cost is spread over enough requests to win.
• Machines busy a small share of the time are the usual reason private AI costs more: a rented machine costs the same at 3 a.m. as at noon.

An API bills every token sent and received at the provider's price per million. A private model runs on a machine you pay for by the hour or once; it costs the same whether it's busy or idle, until the work needs a second machine.

The tool calculates public API costs from each model's list price per million input and output tokens, for 30 days of your daily requests. The private cost counts the machines your busiest hour needs, from published measurements of how fast that class of machine serves a model that size, plus the people who run them.

1. Monthly input tokens = daily requests × 30 × average input tokens; the same for output tokens.
2. Public API cost = input tokens ÷ 1,000,000 × input price + output tokens ÷ 1,000,000 × output price, for each model's share of the requests (batch prices for work within the day, where offered).
3. Memory a request needs = weights + the context cache (2 × layers × key-value heads × head size × 2 bytes for every token of prompt and answer) + a tenth for the runtime.
4. Machine time per request = prompt tokens ÷ the machine's prompt rate + answer tokens ÷ its answer rate, both scaled from a published measurement by memory bandwidth and model size.
5. Machines = busiest-hour requests a second × machine time per request, rounded up, plus a spare if you keep one.
6. One reply alone = memory bandwidth of the GPUs holding the model ÷ its active weights (the most it could manage) × the share of that measured runs reached: 25% to 55% for mixture-of-experts models, 38% to 66% for dense ones.
7. Private AI cost = machines × monthly price (hourly price × 730, or the purchase ÷ months) + electricity + engineer hours × hourly cost.
8. Difference = public API cost − private AI cost: above zero, hosting privately saves money.

Assumptions

• A month is 30 days of requests, as the brief has it; machines bill 730 hours a month (24 × 365 ÷ 12), as cloud providers do.
• List prices: each provider's published price per million tokens and each cloud's on-demand price an hour, Linux, without reserved or committed-use discounts. Machine prices in another currency are converted at the European Central Bank's reference rates.
• Speed: estimated from measurements, never from a vendor's claim. NVIDIA's own NIM measurements of Llama models on H100, H200, L40S and A100 GPUs, at four prompt and answer lengths up to 5,000 tokens, are the basis for NVIDIA machines; llama.cpp's on an M2 Ultra for Apple; AMD's on a Ryzen AI Max+ 395 for that chip. A mixture-of-experts model counts as total^0.7 × active^0.3 of its parameters when many requests share the machine, which lands within 0.7 to 1.2 times of NVIDIA's TensorRT-LLM figures across models.
• The machine picker offers only machines with the memory for the model at the chosen precision and request length. A machine that can't hold it gives way to the cheapest one of the same kind that can, and the tool says so.
• "Someone is waiting" means every reply streams at 20 tokens a second or faster and starts within 2 seconds, which is how the interactive measurements were picked; "within the day" uses each measurement's busiest load and spreads the requests over 24 hours.
• One request at a time on the AMD chip is the only kind AMD publishes, so those estimates are cautious: serving several at once usually raises a machine's total.
• Engineer time is an example to replace: the site's example salary for the country with its employer costs, over 2,080 hours a year. Setup, the software around the model, and testing it on your work aren't counted.
• The brief's figures are one choice away: "Your own price per million tokens" starts at its $5 in and $15 out, and its flat $750 a month is close to an AWS g5.xlarge on demand ($1.01 × 730 = $734), a 24 GB machine.

Open-weight models and the memory they need

Every open-weight model the tool can run, with its licence and the memory one request of 1,500 tokens needs at each precision it offers. Weights are the developer's own release where there is one, else parameters × 2 bytes (16-bit), 1 byte (8-bit) or half a byte (4-bit).

Machines and their prices

Every machine the tool can price, with its memory, its memory bandwidth (which sets how fast it writes) and its price.

The measurements behind the estimate

Each estimate of a whole machine scales from one of these runs: for NVIDIA, the prompt and answer rates fitted to NIM 1.8.0's published rows, for replies someone waits on and for background work. One reply's speed comes from the single-request runs below them.

Both examples are the tool's own output for the inputs shown.

Private server capacity is fixed. If your 10,000 daily requests all happen in one hour, a single machine will throttle. You would need to scale up your infrastructure, raising the private cost.

• Bursts: the busiest-hour share sizes the machines. If a tenth of a day's work can arrive in a few minutes, set the share higher or keep a spare machine.
• Your own speed: serving software, settings, batch sizes and prompts all change throughput. The estimate scales from published measurements; run a load test of your own prompts on the machine before buying, and enter its figure.
• Model quality: a private model costs less only if it does the job as well. Test the open-weight model and the API model on a few hundred of your real cases first.
• Memory: a model runs only where it fits. A 70B-parameter model needs about 35 GB at 4-bit for its weights alone, more than a 24 GB GPU holds; the tool checks the fit, weights and context cache together, for every model and machine.
• Prompt caching and discounts: providers charge less for repeated prompt text and for committed spend; clouds charge less for reserved machines. Enter your own prices where you have them.
• Setup and the software around it: the private cost is running cost only. Building the application, deploying the model and testing it come on top, as they do for an API.
• Prices move: model prices change and new GPUs arrive every few months. Every price here carries the date it was checked.

To weigh the running cost against the hours the work replaces, use the AI automation payback calculator.