What an AI Feature Really Costs

A founder shows me an AI feature that users love. Then I ask what it costs to serve, and the room goes quiet. They had modeled the cost of building it, not the cost of every single use, forever. For AI, that second number is the one that decides whether the business works. The demo is free. The bill arrives later, every time someone presses the button.

Why AI economics are different

Traditional software costs a lot to build and almost nothing to run one more time. Once a web app exists, serving the millionth user costs a sliver more than serving the thousandth. AI inverts part of that. Every query runs a model, every model run burns compute, and that compute has a real, recurring price. A feature that is delightful at a hundred users can be ruinous at a million if nobody checked the unit economics.

In AI, the dangerous line in the budget is not what it costs to build. It is what it costs each time someone uses it.

The hidden parts of the bill

When teams estimate cost, they usually count the obvious thing: the price of a single model call. Then reality adds the parts nobody put in the spreadsheet.

• Input and output tokens: you pay for the prompt you send and the answer you get, and long context or verbose outputs multiply both.
• Retries and retries-of-retries: failed or low-quality responses that get regenerated quietly double or triple real cost.
• Retrieval and embedding: RAG systems pay to embed queries and search a vector store on top of the model call.
• Multi-step chains: an agent that makes five model calls to answer one question costs five times as much as it looks.
• Idle infrastructure: self-hosted models mean paying for GPUs whether or not anyone is using them.

How to estimate cost per request

Before you ship, build a simple model of what one typical interaction costs, then multiply by expected volume. The arithmetic is not hard, and it is the single most clarifying exercise a team can do.

A back-of-the-envelope method

• Measure the average input and output tokens for a real interaction, not a toy one.
• Multiply by the model’s per-token price to get the cost of a single clean call.
• Add the hidden parts: retries, retrieval, and any extra steps in the chain.
• Multiply by your realistic monthly request volume to get the true running cost.
• Compare that number to the revenue or value each request generates.

If the cost per request is larger than the value per request, you do not have a feature. You have a liability that grows with adoption. Finding that out before launch is a gift.

The levers that decide profitability

The good news is that per-request cost is highly controllable. A handful of design decisions routinely cut it by an order of magnitude without hurting the experience.

• Model size: the smallest model that clears the quality bar is almost always the right one, and often a fraction of the price.
• Caching: never pay twice for the same answer, since many requests repeat or overlap.
• Routing: send easy requests to a cheap model and reserve the expensive one for genuinely hard work.
• Prompt discipline: trim bloated system prompts and context, because every token is a line item.
• Output limits: cap response length when a short answer is enough.

When to self-host versus use an API

At low and moderate volume, paying per call through an API is usually cheaper and far simpler, because you pay only for what you use. As volume grows and becomes predictable, self-hosting an open model on your own hardware can win, since a busy GPU running cheap open-weight models can beat per-token API pricing. The crossover depends on your traffic, and the mistake is assuming one answer fits every stage of growth.

Frequently asked questions

How do I calculate the cost of an LLM feature?

Measure the average input and output tokens for a real interaction, multiply by the model’s per-token price, add the cost of retries, retrieval, and any extra steps, then multiply by your expected request volume. Compare that figure to the value each request produces.

Why do my AI costs scale faster than my user count?

Because cost is driven by requests, not users, and heavy users make many requests each. Longer conversations, bigger context, retries, and multi-step chains all multiply the per-user cost, so spend can climb faster than headcount.

What is the cheapest way to reduce AI inference costs?

Start with the free wins: use the smallest model that meets the bar, cache repeated answers, and trim bloated prompts. These three usually cut cost dramatically before you need anything more sophisticated like routing or self-hosting.

Is it cheaper to self-host a model or use an API?

At low or spiky volume, an API is usually cheaper and simpler because you pay only for what you use. At high, steady volume, self-hosting an open model can be cheaper per request. The right choice shifts as your traffic grows, so revisit it over time.

Key takeaways

• AI inverts software economics: the recurring per-request cost, not the build, decides viability.
• The real bill includes tokens, retries, retrieval, and multi-step chains, not just one model call.
• Estimate cost per request and multiply by volume before you ship, not after.
• Model size, caching, and routing can cut per-request cost by an order of magnitude.
• The API versus self-host answer changes as your traffic grows, so revisit it.

None of this means AI features are too costly to build. It means the unit cost is a first-class design decision, not an afterthought. Model it on day one, or the success of the feature becomes the thing that sinks you. Getting those unit economics right from the start is a big part of the applied-AI work we do at Vesper Labs.