The platform underneath the AI feature

The situation

A Series A SaaS company had shipped an AI feature fast, and it worked. Customers liked it. It was in the sales deck.

Then the questions started arriving. Finance wanted to know why the inference line had tripled and which customers were driving it — nobody could answer either. Engineering wanted to know why p95 latency had doubled on Tuesdays. Sales had two enterprise deals stalled on a security questionnaire asking about PII handling and data residency, and the honest answer was "we send it to a model and hope."

Nothing was broken, exactly. That was the problem. It worked well enough that nobody had built anything underneath it.

The constraint

Three things ruled out the obvious approach.

They could not stop shipping product. The team was seven engineers and the roadmap was a fundraising commitment. Any plan that started with "pause feature work for a month" was dead on arrival, regardless of how correct it was.

The AI code was load-bearing and undocumented. Prompts lived inline in application code. Model selection was a hard-coded string in three separate files. There was no test that would tell you if a prompt change made output worse — which meant nobody was confident enough to touch it, which meant it kept getting worse.

They were nervous about lock-in but couldn't afford to act on it. The CTO had a well-founded worry about building the company on a single vendor's API, and no capacity to do anything about it. That tension was going to keep resurfacing until someone resolved it structurally.

So the work had to be additive. Build the platform around the feature rather than rewriting it, and make the migration path a property of the architecture rather than a project.

What I did

A gateway, so the application stops caring which model it's talking to.

The first move was to put an inference gateway between the application and Amazon Bedrock. The application now calls one stable interface. Behind it: routing by task type, automatic failover across models and regions, per-tenant rate limits, structured request and response logging, and — the thing finance actually wanted — token counting tagged by customer and by feature.

That last part sounds mundane and it was the highest-value hour of the engagement. Within a week they could see that eleven percent of customers were generating sixty percent of inference cost, and that most of it came from one feature nobody had thought of as expensive.

Retrieval moved onto managed infrastructure.

The RAG implementation was a set of Python scripts someone ran by hand after a content update. I moved it onto Bedrock Knowledge Bases with an OpenSearch Serverless vector store, with ingestion running as scheduled jobs and re-indexing triggered by content changes rather than by memory.

Guardrails, because the enterprise deals needed an answer, not a promise.

Bedrock Guardrails for PII redaction and content policy, applied at the gateway so it couldn't be bypassed by a new code path. Combined with regional isolation, this turned the security questionnaire from a conversation into a document.

Prompts became code.

Prompts and model configuration moved out of application source into a versioned repository, with an automated evaluation suite running against a held-out set on every change. A prompt edit is now a pull request that either passes the eval gate or doesn't. This was the change the engineering team cared about most — not because it was clever, but because it meant they could finally change prompts without fear.

Throughput was sized rather than guessed.

Provisioned Throughput for the predictable baseline, on-demand for burst. Batch and long-running generation moved to Step Functions with streaming responses served through API Gateway and Lambda, which took the slow requests out of the synchronous path entirely — most of the Tuesday latency problem turned out to be a weekly batch job competing with interactive traffic.

All of it in Terraform, with a deliberate exit.

The whole platform is defined in Terraform with per-environment separation. And behind the same gateway interface, I built a self-hosted vLLM deployment on EKS. It is not in the serving path today. It exists so that staying on Bedrock is a commercial decision, reviewed annually, rather than a technical constraint discovered too late.

The outcome

TODO(anshuman): replace these with your real figures once the build is done.

• Cost per request down [[N%]], and — more usefully — attributable to a customer and a feature for the first time
• p95 latency down [[N%]], most of it from moving batch work off the synchronous path
• Prompt and model changes became reviewable pull requests with an evaluation gate; the team ships prompt improvements weekly now instead of quarterly
• Guardrails and regional isolation unblocked [[N]] enterprise deals that had stalled on data handling
• Zero product roadmap time lost — no feature work was paused for the migration

What I'd do differently

I built the gateway before I built the cost dashboard, because the gateway felt like the architecturally important piece. It was the wrong order. The cost data was what got the CFO into the room, and having it a week earlier would have made every subsequent conversation easier.

The general lesson, which I keep relearning: on an engagement where you need political capital, ship the thing that makes someone's number legible before you ship the thing that makes the architecture correct.