Adding AI to your existing APIs
The cleanest AI integration is one your callers never notice. If an endpoint already returns a category, a summary or a priority, a model can produce that value without the contract changing at all. The interesting problems are not about the model; they are about latency, retries and what happens when the provider is slow.
Where does the model actually go?
Behind your own endpoint, never in front of it. Your application calls your API as it always did; your API calls the model. That keeps the provider an implementation detail, which matters more than it sounds: swapping models becomes a configuration change rather than a client release, and every consumer of that endpoint is insulated from it.
Calling a provider's SDK directly from application code is the mistake that costs the most later. It scatters API keys, retry logic and prompt text across the codebase, and it makes changing model impossible without touching every call site. Given how fast this market moves, that abstraction pays for itself inside a year.
The practical shape is a thin internal service or module with one job: take a typed input, return a typed output, and own everything about how the model was called.
Should it be synchronous or queued?
This is the decision that determines whether the integration holds up, and the honest default is queued. Model calls take hundreds of milliseconds to several seconds, and they occasionally take much longer for reasons entirely outside your control. Putting that inside a request a user is waiting on ties your response time to a third party's worst day.
Synchronous is right when the answer is the point of the request and the user is expecting to wait: a search that returns ranked results, a form field being completed. Even then it needs a timeout well below your own, and a defined answer for what to return when the timeout fires.
Queued is right for anything that enriches a record rather than answering a question. Classification, summarisation, extraction, tagging. Accept the request, return immediately, do the work, write the result back. Callers who need the enriched value poll or subscribe rather than block.
| Pattern | Use when | Watch out for |
|---|---|---|
| Synchronous call | The answer is the response | Timeout must be lower than your caller's; needs a fallback value |
| Queued job | Enriching a record | Callers need a way to know it finished |
| Streaming response | Long text a human reads as it arrives | Harder to cache, harder to validate before sending |
| Batch overnight | Whole-table processing | One prompt change can re-bill the entire corpus |
What breaks that would not break a normal endpoint?
Non-determinism, first. The same input can produce a different output, which breaks the assumption most API tests rest on. Retries are no longer free: retrying a failed call can produce a different answer than the first attempt would have, so anything that writes a result needs an idempotency key and a decision about whether the first or last answer wins.
Then output shape. A model asked for JSON will occasionally return JSON wrapped in prose, or JSON with an extra field, or almost-JSON. Validate against a schema before the value goes anywhere near your database, and treat a validation failure as a retryable error rather than a crash. Providers offering structured output modes reduce this considerably and are worth using where available.
And rate limits, which behave unlike your own infrastructure. You will be throttled at the moment your traffic peaks, because that is when everyone else's does too. Exponential backoff with jitter, a queue that can absorb the delay, and a cap on attempts so a throttled provider does not become an infinite loop.
How do you keep it fast?
Cache aggressively, on a hash of the normalised input. A surprising share of production requests are identical or near-identical, and a cache hit costs nothing and returns instantly. This is the single largest performance and cost lever available, and it is skipped more often than any other.
Right-size the model per call rather than per system. Most requests in any workflow are easy; route those to a small fast model and escalate only genuinely hard cases. Latency and cost both improve by a multiple rather than a percentage.
Trim the prompt. Everything you send is transmitted, processed and billed on every call, so a document or a full conversation history in the prompt is a latency multiplier as well as a cost one. Summarise history rather than resending it.
What does the Node.js side look like in practice?
A module exporting typed functions, one per task, each owning its own prompt, model choice, schema validation and cache key. Callers pass a typed object and receive a typed object; nothing about the provider leaks out. In TypeScript, validating the model's response with the same schema library you already use for request validation keeps the boundary honest.
Keep prompts in version control alongside the code, not in a database or a dashboard. A prompt is behaviour, so it belongs in a pull request where a change can be reviewed and rolled back like any other. Version them explicitly, because a prompt change alters output for everything downstream and you will want to know which version produced a given stored result.
Never call a model from a serverless function with a short execution limit unless the call is genuinely fast and the timeout is set deliberately below it. That combination fails silently and intermittently, which is the hardest kind of fault to diagnose.
Common questions
- Can AI be added to an existing API without changing its contract?
- Usually yes. If the endpoint already returns a value that a model can produce, such as a category, a priority or a summary, the model goes behind your own endpoint and callers see no change. Keeping the provider behind your own interface also means swapping models later is a configuration change rather than a client release.
- Should AI calls be synchronous or queued?
- Queue them unless the model's answer is the point of the request. Model calls take from hundreds of milliseconds to several seconds and occasionally far longer, so a synchronous call ties your response time to a third party's worst day. Enrichment work such as classification or summarisation should accept the request, return immediately, and write the result back.
- How do you handle retries when model output is not deterministic?
- With an idempotency key and an explicit decision about whether the first or last answer wins. Retrying a model call can produce a different answer than the original attempt would have, so any operation that writes a result needs to be safe to repeat. Cap attempts too, or a throttled provider turns into an infinite loop.
- What if the model returns malformed JSON?
- Validate every response against a schema before it reaches your database, and treat a validation failure as a retryable error rather than an exception. Models occasionally return JSON wrapped in prose or with unexpected fields. Providers with structured output modes reduce this substantially and are worth using where available.
