TL;DR
Adding AI to a live product is an architecture problem, not a model-picking one. The gateway, the build ladder, evals, and how to ship without a rebuild.
TL;DR: Adding AI to an existing product is an architecture and evaluation problem, not a model-picking problem. Do not rebuild. Route every model call through a thin gateway service so you can swap models, cap spend, and log everything from one place. Climb the build ladder in order: exhaust prompt engineering, then RAG over your own data, then fine-tuning, and self-hosting last. Write the evaluation harness before you ship the feature, because if you cannot measure the output you cannot ship it. Wrap the whole thing in guardrails, tracing, a cost budget, and a graceful fallback for when the model fails. Ship one thin, high-value slice behind a feature flag, prove it with evals, then expand.
Your board wants an AI feature. Your users are asking for one. And you already have a live product with real customers, real data, and a codebase you cannot afford to destabilize. The pressure is to bolt something on fast. The trap is doing it in a way you spend the next year unwinding.
Here is the reframe that saves you: the hard part of adding AI to an existing product is almost never the model. It is where the AI layer sits in your architecture, how you feed it your own data, and how you prove the output is good enough to put in front of a paying customer. Teams spend weeks agonizing over which model and almost no time on those three things, which is backwards.
We add AI features to existing products for a living, on codebases we did not write, and the same playbook holds every time. This is that playbook: the architecture, the build decisions, and the non-negotiables that separate a demo from something you can ship.
The first architectural decision is the one most teams skip, and it is the one that pays off the longest. Do not scatter direct API calls to a model provider across your codebase. Route every AI call through a thin internal service: an LLM gateway.
A gateway is a small layer between your application and whatever model providers you use. Every AI call goes through one entry point, which gives you five things you will need within weeks: the ability to switch models without touching application code, per-request cost and token logging, automatic failover when a provider errors, spend caps that return an error instead of a surprise bill, and one place to enforce guardrails. Open-source options like LiteLLM normalize 100-plus providers behind one interface; hosted options add observability and caching.
The gateway also decides where the AI logic lives. Put shared model logic behind its own service when several parts of your product consume it or it needs to scale independently of your web tier. Embed it directly only for latency-critical or offline paths. And make model calls asynchronous whenever the task tolerates it. A synchronous call to a slow model ties up a web worker for the whole request, so anything measured in seconds (bulk classification, summarization, document processing) belongs on a queue with the result delivered by poll or callback.
There is a strong consensus on the order to reach for techniques, and skipping steps is where budgets die. Climb this ladder and stop at the first rung that works.
1. Prompt engineering first. Exhaust it before anything else. A well-designed prompt on a mid-tier model beats a sloppy prompt on a frontier model, and most single-step tasks are solved here with structured inputs and outputs and a few examples. This rung is free and ships in a day.
2. RAG when the model needs your data. If the feature needs facts the model was not trained on, or facts that change (your docs, your prices, your customer records), retrieve them at query time and put them in the prompt. This is the rung most product AI features actually need.
3. Fine-tuning only after the first two fall short. Fine-tuning bakes behavior into the model's weights. It fits a stable, well-defined task where you need a specific format or voice, or lower per-call cost at high volume, and you have quality labeled data. It does not teach the model fresh facts, and most teams that fine-tune early regret it. RAG beats fine-tuning for small or changing datasets.
4. Self-hosting last. Run your own open-weight model only when data residency, rate limits, or cost at very high volume force it. Below serious sustained volume, a hosted API is cheaper and far less work.
The fork that matters: RAG retrieves knowledge at inference, fine-tuning bakes it into weights. For most features bolted onto an existing product, retrieval over your own data is the answer. Our RAG vs fine-tuning breakdown goes deeper on that choice, and what RAG is covers the basics.
The value of AI in an existing product usually comes from your own data, which means RAG over your database and documents. Three things decide whether that works.
Retrieval quality beats model choice. The single highest-impact technique in 2026 is contextual retrieval: before you embed and index each chunk, prepend a short snippet describing where that chunk sits in its document. Anthropic measured this cutting retrieval-failure rates by around a third on its own, and roughly two-thirds when combined with keyword search and reranking. It needs no new database or architecture. Pair it with hybrid search, which combines vector similarity with keyword matching (BM25) and metadata filtering, because pure vector search misses exact terms like product codes and error strings. For where to store the embeddings, most teams should start with the database they already run before adopting a dedicated vector database.
Keep the index in sync with the source of truth. The quiet failure mode of bolt-on RAG is the split between your application database and your vector index. Delete a record and its embedding lingers as a "ghost document" that retrieval still surfaces. Do not solve this with nightly full re-indexing. Stream row-level changes from your database (change-data-capture) and re-embed only what changed, and skip unchanged chunks with content hashing.
Treat freshness as an SLA. Semantic similarity has no sense of time, so a stale document scores just as high as a current one, and quality degrades silently with no error and no latency change. Monitor it. And never mix embeddings from two different embedding models in one index, because upgrading the embedding model shifts the vector geometry and quietly corrupts retrieval. If you change embedding models, re-embed everything. This is the kind of decision our enterprise RAG architecture guide covers in depth.
You do not pick one model. You route between a cheap, fast, small model and an expensive, capable frontier one, and send each request to the cheapest model that can handle it. A small model handling the routine 70 percent of traffic, with a frontier model reserved for the hard cases, cuts cost sharply with no visible quality drop. Build this as a routing rule or a cascade that escalates on failure, and put it in the gateway.
Two levers matter for the bill. Prompt caching is the biggest: the providers now cache a stable prompt prefix and charge roughly a tenth of the input price on cache hits, so structure your prompts with the fixed system content first. And choose the smallest model that passes your evals, because speed costs money, capability costs latency, and verbosity costs both. For the API-versus-self-host question, a hosted API wins until you have steady, high sustained volume or a hard data-residency requirement; the break-even is a volume calculation, not a philosophy. If you are choosing a backend to host this layer, our FastAPI vs Node.js for AI backends comparison is a useful companion.
A demo needs none of the following. A feature in front of paying customers needs all of them. This is the gap between the two.
Evaluation, before you ship. This is the number one non-negotiable. Write the eval harness before you build the feature, and review the evals, not the demo. Build a golden dataset of real examples from your product (start with about 30 and keep adding until no new failure modes appear), and score outputs pass or fail rather than on a vague 1-to-5 scale. When you use a model as a judge, align it to a human reviewer until they agree. The blunt version: if you cannot write the eval, you cannot ship the AI feature. Run evals offline to catch regressions before deploy, and online to catch drift after.
Guardrails, in the architecture. Prompt injection is still the top entry on the OWASP list of LLM risks, and it cannot be prompted away. It needs structure: separate instructions from data, validate outputs, give the model least-privilege access to tools, add a guard model for high-stakes flows, and log everything. Constrain generation to a schema at the API level rather than just asking for JSON. Assume the model will be manipulated and design so the blast radius is small.
Observability, from day one. You cannot debug what you cannot see, and standard monitoring is blind to AI failures. Instrument model calls with tracing (the OpenTelemetry GenAI conventions are the standard) and track cost per request, latency at p95 and p99, eval scores, error and retry rates, and any injection or PII hits. The failure to do this shows up as an agent that quietly retried a tool for 90 seconds or ran a retrieval step 14 times in one conversation, invisible until a customer complains. A dedicated observability tool makes these visible; choosing one is its own decision.
Latency budgets and cost controls. Anchor your latency targets on p95 and p99, not the average, because the tail is where users feel it and providers diverge most. Stream tokens for anything interactive, because users perceive lag past roughly half a second. And put hard spend caps in the gateway, because retries and fallback logic silently multiply cost until they show up in the monthly bill.
Graceful fallback. Model APIs go down; 2026 has already seen multi-hour outages from major providers stall production workloads. Single-provider dependency is now a real production risk. Your gateway should reroute on errors, downgrade to a cheaper model on rate limits, and, when everything fails, serve a last-known-good cached response with a note rather than a spinner. Degrade capability instead of failing outright.
Bolting AI onto an existing product means sending your product and customer data to a model, and that carries obligations you should settle before you prototype, not after.
The major API providers do not train on your API data by default, and both Anthropic and OpenAI offer zero-data-retention arrangements on request, though these carry exceptions (batch endpoints, file storage, and certain models can be excluded), so read the terms for the exact endpoints you use. For health data you need a signed business associate agreement, and you should keep regulated data out of places that are cached separately, like schema definitions. If data cannot leave your environment at all, running the model through your own cloud account (a hyperscaler's managed model service, or a self-hosted open-weight model) keeps the provider out of the data path.
There is also a compliance clock. The EU AI Act's transparency obligations take effect on August 2, 2026, and as a company adding AI to a product you are a deployer, which means you must disclose when users are interacting with AI or viewing AI-generated content. Build that disclosure in from the start rather than retrofitting it.
The same failures recur when teams bolt AI onto a live product.
The demo-to-production gap. A feature that scores well on a clean internal test set drops sharply on real, messy production input, because real users bring abbreviations, typos, and broken data your test set did not. Build your eval set from real production samples, not invented ones.
Data that is not ready. The most common reason these projects fail has nothing to do with AI. It is data scattered across spreadsheets, legacy databases, and unindexed drives that retrieval cannot use. Getting your data reachable is often the real project.
Over-agentifying. Agent reliability degrades as you add tools and becomes unreliable past a few dozen. Most features do not need an autonomous agent; they need a fixed workflow of a model plus tools on a predictable path. Reach for a real agent only when you cannot hardcode the path but can still verify the result. Our guide to AI agents and the agent framework comparison cover when that line is worth crossing.
No kill switch. Ship AI features behind a feature flag with a kill switch that takes effect in seconds, not a redeploy. Treat the model as a configuration value, roll out from a small percentage of traffic with sticky user assignment, and watch your metrics before widening. A public 2025 incident where an AI agent deleted a production database is the version of this lesson nobody wants to learn firsthand.
Model-upgrade regressions. A provider updates a model and your carefully tuned feature quietly gets worse. Pin model versions, and validate a new version against your eval harness and a shadow run before you switch.
Put it together and the order is what keeps you from rebuilding.
None of this requires touching the rest of your product. The AI layer sits beside what you already run, behind a flag, measured before it ships.
A few things we hold to on every engagement.
The eval harness is the deliverable, not the demo. Anyone can produce an impressive demo in an afternoon. The teams that ship AI that survives contact with real users are the ones who wrote the evaluation first and treated a passing demo with suspicion until the numbers backed it up.
Start smaller than feels satisfying. One workflow, one thin slice, proven end to end, beats a broad feature that half-works everywhere. The broad version is the one that erodes trust and gets pulled.
The boring infrastructure is the moat. The gateway, the evals, the tracing, the fallback, the kill switch. None of it demos well, and all of it is what separates a feature you can stand behind from one that produces a bad answer you cannot explain. We wire it in from the first commit, the same way we wire in CI/CD, because retrofitting it after an incident is the expensive path.
Add the AI as a separate layer beside your current system, not inside it. Route model calls through a thin gateway service, retrieve your own data with RAG at query time, and ship the feature behind a flag. Because the AI layer is decoupled and feature-flagged, you can build, test, and roll it back without touching the rest of your product. Start with one thin, high-value workflow rather than a broad feature.
Climb the ladder in order. Start with prompt engineering and exhaust it. Move to RAG when the feature needs your own data or facts that change. Fine-tune only after prompting and RAG fall short, when you need a specific format or lower cost at high volume with quality labeled data. Self-host a model last. RAG beats fine-tuning for small or changing datasets, and most product features never need to go past RAG.
An LLM gateway is a thin internal service that every model call passes through. You need one because it gives you model-swapping without code changes, per-request cost and token logging, spend caps, automatic failover between providers, and a single place to enforce guardrails. Adding it early is far cheaper than untangling direct provider calls scattered across your codebase later.
Write an evaluation harness before you build the feature. Collect a golden dataset of real examples from your product, starting around 30 and growing until no new failure modes appear, and score outputs pass or fail against clear criteria. Run it offline to catch regressions before deploy and online to catch drift afterward. If you cannot write an eval for a feature, you are not ready to ship it.
It can be, with the right setup. The major API providers do not train on your API data by default and offer zero-data-retention options on request, though those carry endpoint exceptions you should read. For regulated data, sign the required agreements (such as a BAA for health data) or run the model inside your own cloud account so data never leaves your environment. From August 2, 2026, EU rules also require disclosing to users when they interact with AI.
Route requests to the cheapest model that can handle them, reserving a frontier model for hard cases, and do it in the gateway. Use prompt caching, which charges roughly a tenth of the input price on cache hits, by putting stable content at the start of your prompts. Put hard spend caps in the gateway, and watch retry and fallback logic, which silently multiply cost. Pick the smallest model that passes your evals.
The frequent ones: testing on clean data that hides how the feature fails on messy production input, unready data scattered across systems that retrieval cannot use, building an autonomous agent where a fixed workflow would do, shipping with no kill switch or feature flag, and getting silently broken by a provider's model upgrade. Most trace back to skipping evaluation and observability.
Not necessarily. Start with the database you already run. Postgres with the pgvector extension, or the built-in vector search in MongoDB or Redis, handles retrieval for most products up to tens of millions of vectors, and keeps embeddings next to your application data with no sync pipeline. Adopt a dedicated vector database only when scale, latency, or hybrid-search quality genuinely outgrows your existing system.
The teams that add AI well are not the ones with the best model. They are the ones who treated it as an architecture problem: a decoupled layer, a gateway, retrieval over their own data, and an evaluation harness that had to pass before anything shipped. That is boring, and it is exactly why it works.
MarsDevs builds AI-powered features into existing products for startups and scale-ups, on codebases we did not write, with the gateway, evals, and observability wired in from day one. If you have a live product and a mandate to add AI without destabilizing what already works, talk to our engineering team or book a call. We will help you ship the thin slice first and prove it before you scale.
Co-Founder, MarsDevs
Vishvajit started MarsDevs in 2019 to help founders turn ideas into production-grade software. With deep expertise in AI, cloud architecture, and product engineering, he has led the delivery of 80+ software products for clients in 12+ countries.
Get more guides like this
Join founders, CTOs, and engineering leaders who receive our engineering insights weekly. No spam, just actionable technical content.
Partner with our team to design, build, and scale your next product.
Let’s Talk
