A client called us last year, excited about an AI customer service bot they’d built. The prototype was impressive. Accurate answers, natural tone, handled edge cases well. They’d estimated the running cost at about $200/month based on the API pricing page.
Three months into production, their monthly AI bill was $4,700. Nobody had done the math on what happens when 3,000 customers a month each have multi-turn conversations that stuff the context window with product documentation.
This is the cost conversation that most AI projects skip. And it’s the one that determines whether your AI project is sustainable or gets quietly shut down six months after launch.
The components of AI cost
The sticker price for AI models (the per-token cost from OpenAI, Anthropic, Google, etc.) is real, but it’s only one piece. Here’s what actually shows up on the bill when you run AI in production.
API costs (the obvious one)
Every time your AI processes text, you’re paying for tokens. Input tokens (what you send to the model) and output tokens (what the model generates). Output tokens typically cost 3-5x more than input tokens.
For a simple chatbot that handles short questions, this might be $0.01-0.05 per conversation. That’s cheap. But costs multiply fast when you add context.
A RAG-based knowledge assistant that pulls in relevant documentation with each query? Now you’re sending 2,000-10,000 tokens of context with every message, on top of the conversation itself. A customer service bot with access to the customer’s order history, your product catalog, and your return policy? You’re easily hitting 15,000+ input tokens per response.
Here’s a concrete example. Say you’re running a customer support bot using Claude Sonnet:
- Average input per interaction: 8,000 tokens (conversation + context + system prompt)
- Average output per interaction: 500 tokens
- Cost per interaction: roughly $0.03-0.04
- 100 interactions per day: ~$3-4/day, or about $90-120/month
That’s manageable. But if you’re handling 1,000 interactions per day, you’re looking at $900-1,200/month just in API costs. And that’s for a mid-tier model. GPT-4o or Claude Opus can be 5-10x more expensive per token.
Infrastructure costs
Your AI doesn’t run in a vacuum. It needs somewhere to live.
If you’re using RAG, you need a vector database (Pinecone, Weaviate, Qdrant, or similar). Free tiers work fine for prototypes; production workloads run $70-500/month depending on how much data you’re indexing and how many queries you’re running.
Then there’s the orchestration layer: a server or cloud function that receives the user’s request, queries the vector database, assembles the context, calls the AI API, and returns the response. A small cloud instance handles low traffic for $20-50/month. High-traffic applications need auto-scaling infrastructure, which can hit $200-1,000/month.
Throw in a database for conversation history and session management ($20-100/month) and monitoring tools to track output quality ($20-50/month for basics), and you start to see how these line items accumulate. None of them are expensive individually. A typical small-scale production AI deployment runs $150-400/month in infrastructure, not counting the AI API itself.
Maintenance costs (the sneaky one)
This is where budgets fall apart. Building an AI application is a project. Running an AI application is a responsibility.
Models get updated. OpenAI releases a new version, and suddenly your carefully tuned prompts produce different results. You need to test, adjust, and sometimes rewrite them. This happens a few times a year per model provider.
Your data changes. You add new products, update policies, hire new people, change processes. The AI’s knowledge base needs to keep up. If you’re using RAG, that means re-indexing documents. If you’re using fine-tuned models (less common for small businesses, but it happens), that means retraining.
Prompts drift. What worked perfectly in March might produce subtly worse results by June because the underlying model changed, or because your use case evolved, or because you fed it different data. Somebody needs to monitor output quality and tune the prompts when things slip.
We tell clients to budget 10-20% of the initial build cost per month for ongoing maintenance. If your AI project cost $15,000 to build, expect $1,500-3,000/month in maintenance once you add infrastructure, API costs, and the human time to keep it running well.
The hidden cost: somebody needs to own this
I want to be direct about this one, because it’s the cost that most businesses don’t anticipate.
Your AI application needs an owner. Not a committee. Not “the IT department.” A person who understands how it works, monitors its output, responds when it breaks, and makes decisions about when to update it.
For a small business, this might be a few hours per week from someone who’s already on your team. For a larger deployment, it could be a significant chunk of someone’s role.
This isn’t optional. An AI application without an owner is an AI application that slowly degrades until someone notices it’s been giving customers wrong information for three weeks.
Real numbers from real deployments
We’ll be honest: cost varies wildly depending on what you’re building and how much traffic it handles. But here are rough monthly ranges from deployments we’ve worked on or reviewed.
An internal knowledge assistant for a team of 20-50 people, used moderately throughout the day, typically runs $150-400/month. API costs are modest because internal usage is lower volume and the infrastructure stays simple.
Customer-facing chatbots are more expensive: $300-1,200/month for 500-2,000 conversations. API costs are the biggest line item, and you need better monitoring because errors affect customers directly.
Document processing pipelines (200-500 documents/month) land around $200-800/month, depending heavily on document length and complexity. The main ongoing cost is handling edge cases and maintaining extraction accuracy as document formats change.
AI-assisted email systems processing 1,000-5,000 emails/month run $400-1,500/month. High volume drives up API costs, and human review time is an additional expense that’s easy to underestimate. You still need people approving outbound messages, at least for the first few months.
Custom AI agents with multiple tool integrations are the most expensive to run: $800-3,000/month. More tools means more points of failure, which means more maintenance. This is where the “somebody needs to own this” cost really shows up.
None of these numbers include the initial build cost. They’re what you pay to keep the lights on every month.
How to forecast and control spend
If you’re planning an AI deployment, here’s how to avoid the surprise we described at the top of this article.
Start by estimating the per-interaction cost before you build anything. How many tokens will a typical interaction use (input + output)? Multiply by the cost per token for your chosen model. Multiply by your expected daily volume. That gives you a monthly API cost estimate. Then double it, because your estimates will be low.
Not every task needs GPT-4o or Claude Opus. For classification, routing, and simple extraction, smaller models (GPT-4o-mini, Claude Haiku, Gemini Flash) cost a fraction and perform well enough. Save the expensive models for tasks that actually need the reasoning capability. Using the right model for each task is one of the easiest ways to cut costs in half.
Cache aggressively. If the same questions come up repeatedly (and in most business applications, they do), cache the responses. A cached response costs zero tokens. Even partial caching, like storing the RAG retrieval results, can make a real dent.
Set spending limits on every API. Every major provider lets you cap monthly spend. Use this. It’s better to have the service stop responding when you hit your budget than to discover a $12,000 bill because a bug caused infinite retry loops over a weekend.
Build dashboards that show token usage per interaction, cost per day, and trends over time. Tools like LangSmith, Helicone, or Portkey make this straightforward. If costs are creeping up, you want to know why before the bill arrives, not after.
And review your model choices quarterly. The AI market moves fast. The model that was the best price-performance option six months ago might not be today. Prices tend to drop, new options appear, and a quick review of “are we using the right model for each task” can save meaningful money.
The part nobody tells you
The actual dollar cost of running AI in production is usually manageable for businesses that plan for it. What catches people off guard is the attention cost. Somebody needs to think about this system regularly. Check the output quality. Review the costs. Update the knowledge base. Fix the things that break.
That’s not a technology problem. It’s an organizational one. And the businesses that succeed with AI in production aren’t the ones with the biggest budgets. They’re the ones that treated the AI application like what it is: a system that needs care, feeding, and occasional repair, just like any other part of their operations.
If you’re budgeting for an AI project, add up the API costs, the infrastructure, and the maintenance. Then add a line item for “someone’s time to keep this running.” That last line item is usually the biggest one, and it’s the one most teams forget.