The best way to host an AI feature as a function

To host an AI feature as a function, deploy it through a dedicated AI function runtime: a purpose-built layer that takes your prompt, agent, or multi-agent workflow and wraps it in versioning, evaluation gates, model fallback, and structured observability, then exposes it as a single callable endpoint your application invokes by name and version. Every other approach (inline LLM calls, custom microservices, generic serverless) forces you to assemble that operational shell yourself, one production incident at a time.

Why doesn't "just call the model API" work in production?

Most teams start here: an inline LLM call inside the route handler that needs it. The prototype works. Then the feature ships and the gaps appear, not in the model, but in the hosting layer around it.

Someone edits the system prompt to fix an edge case and silently regresses three others. There's no record of what the prompt said last week. A provider outage at 11pm takes the feature down completely. The team debates which model version is actually running in production. The monthly bill is three times what anyone expected, but no one can tell which function is responsible.

RAND Corporation researchers, based on interviews with 65 experienced AI practitioners, found that more than 80% of AI projects fail to reach meaningful production deployment, twice the failure rate of IT projects that do not involve AI. The failure is almost never the model. It is the operational layer: versioning, evaluation, fallback, and observability, the things teams either skip in the rush to ship or rebuild from scratch in the sprint after the incident.

What are the four ways to host an AI feature?

There are four realistic approaches when moving an AI feature from prototype to production. Each trades infrastructure control for engineering overhead in different ways.

Inline in the application

The LLM call lives inside the same service that handles the user request. Fastest to build, zero extra infrastructure.

What you give up: Everything that makes a feature operational over time. Versioning is git comments or nothing. Rollback is a full redeploy. Observability is whatever logging you remember to add. When the provider goes down, your endpoint returns a 500. This approach is right for a proof of concept. It breaks when the feature starts to matter.

Custom microservice

The AI feature gets its own service, its own deploy pipeline, and a clean API contract. Standard microservices instinct applied to AI.

What you give up: You own every line of the operational shell from scratch. Isolation does not give you prompt versioning, eval gates, cost-per-call tracing, or automatic fallback. For a single AI function, that is a quarter of engineering to wrap a few thousand tokens of behavior.

Generic serverless (Lambda, Cloud Run)

Wrap the function as a serverless handler. Elastic scaling, pay-per-invocation, no servers to manage.

What you give up: Serverless handles compute, not AI function lifecycle. AWS Lambda's 15-minute execution limit and state-free invocation model are mismatches for multi-step agent workflows. You still own every byte of the versioning, evaluation, and fallback logic. You have moved the undifferentiated problem to a different infra layer.

AI function runtime

You define the function in fn.yml: inputs, outputs, model, eval threshold, fallback chain. The runtime owns the operational shell: immutable versioning, eval gates that block bad deploys, automatic model fallback, and structured per-run tracing. Your application calls it by name.

What you give up: Direct control over the execution infrastructure (which is generally the thing you did not want to maintain anyway).

What does deploying an AI feature as a function look like?

With fnAI, the function definition is a declarative fn.yml:

name: classify
version: 3
model: claude-sonnet-4-6

input:
  ticket: string

output:
  category: string
  priority: enum[low, high, critical]
  confidence: number

eval:
  dataset: ./tickets.jsonl
  threshold: 0.95

fallback:
  - claude-haiku-4-5

Deploy it, and the eval gate runs automatically:

fn deploy classify --env production
# Running eval: 120 cases against threshold 0.95...
# classify@v3: score 0.963  (v2 was 0.961)
# Deployed. Endpoint ready: fn run classify@v3

The eval gate is not optional ceremony. If the score drops below 0.95, the deploy exits non-zero and nothing ships. A prompt regression that would have slipped past a code review fails the pipeline instead.

Call it from your application exactly as you would call any function:

import { fn } from "@fnlabs/sdk"

const result = await fn.run("classify@v3", { ticket: body.text })
// result.payload.category: "billing"
// result.payload.priority: "high"
// result.meta.latency_ms: 142
// result.meta.cost_usd: 0.003
// result.meta.trace_id: "8f3c..."

The interface contract (typed input in, structured output with status, payload, and a trace ID) does not change when you iterate the implementation. classify@v3 is immutable. When classify@v4 ships and passes eval, callers pinned to @v3 stay on v3; unpinned callers pick up v4 automatically.

What does the runtime actually change in practice?

The operational primitives show up in specific situations that every team eventually hits.

Prompt regression in CI: The eval gate exits non-zero before it reaches production. Without one, the regression surfaces in user complaints.

Provider outage: The fallback chain routes to claude-haiku-4-5 automatically, degrading gracefully rather than going offline. Without fallback, the feature is down until the primary provider recovers.

Unexpected cost spike: Every run record includes the model, token count, and dollar cost. You can see exactly which function version is driving spend. Without per-call tracing, cost attribution is a spreadsheet exercise after the fact.

Audit of last Tuesday's behavior: Every run has a trace ID linking inputs, outputs, latency, and the function version. What did the model actually do for this user? A trace lookup, not a log investigation.

One honest caveat

fnAI is pre-launch. The commands above show the shape of the product. We are onboarding design partners, not claiming numbers we have not earned. The framework (versioning, eval gate, fallback, structured tracing) is the right one regardless of which runtime you use to implement it. The question to ask of any hosting approach (including building a custom microservice) is: how much engineering does it cost to add these four primitives, and who maintains them when the model provider updates their API?

Your AI, as a function call. That is the goal. A purpose-built runtime is how you get there without rebuilding the operational shell for every feature you ship.

Start hosting AI features as functions

Ready to stop rebuilding the operational shell from scratch? Join the fnAI waitlist. We are onboarding design partners now and would love to hear what you are building.

FAQ

What is the best way to host an AI feature as a function?

Use a dedicated AI function runtime: a layer that hosts your prompt, agent, or multi-agent workflow as a versioned, evaluated, callable function. It provides versioning, eval gates, model fallback, and observability so your application calls the feature by name instead of managing an LLM API directly.

What is an AI function runtime?

An AI function runtime is a purpose-built hosting layer for AI features. You define the function in a configuration file (fn.yml), and the runtime wraps it with immutable versioning, evaluation gates, automatic model fallback, and structured per-run tracing, then exposes a stable callable endpoint. It is distinct from generic serverless, which handles compute but not AI function lifecycle.

Why not use serverless functions like Lambda or Cloud Run for AI features?

Serverless handles elastic compute and pay-per-invocation billing but not AI-specific lifecycle concerns: prompt versioning, quality evaluation, model fallback, or cost-per-call attribution. A purpose-built AI function runtime solves the lifecycle problem on top of whatever compute it runs on.

What is the difference between a custom microservice and an AI function runtime?

A custom microservice gives you isolation and an independent deploy pipeline, but not AI-specific primitives (versioning, eval gates, fallback, and tracing) unless you build them yourself. An AI function runtime provides those primitives so you write the behavior and the runtime owns the operational shell.

How do I prevent a prompt change from regressing my AI feature in production?

Define an eval dataset (real inputs with expected outputs) and a quality threshold. An AI function runtime runs eval on every deploy and exits non-zero if the score drops below threshold. The regression fails the deploy pipeline before it reaches production instead of surfacing as a user complaint.

What happens to my AI feature when my model provider goes down?

An AI function runtime with a fallback chain configured routes automatically to the next model when the primary provider is unavailable. Without fallback, an inline LLM call returns a provider error your application handles ad hoc, typically returning an error to the user.