LLM Frameworks: What’s Missing in Production

LLM frameworks are build-time libraries that reduce boilerplate when calling language models — handling retrieval pipelines, multi-step orchestration, structured output validation, or TypeScript streaming — but they do not version, evaluate, or trace what runs in production. That gap is where prompt regressions and ownership failures live.

Part of our guide to LLMOps.

What do LLM frameworks actually do?

The ecosystem has converged on five distinct jobs. Most production systems need more than one.

Retrieval (RAG). These frameworks ingest documents, build vector indexes, retrieve relevant chunks, and inject context into model calls. The engineering surface is substantial: chunking strategy, embedding model, retrieval ranking, reranking, citation extraction — all before the model sees the query. The most established Python RAG frameworks cover ingestion through response synthesis, with managed cloud tiers for document parsing and indexing.

Orchestration. Frameworks here sequence model calls, route between tools, manage state across steps, and handle retries and conditional branches. Both Python and TypeScript implementations are widely used, with broad integration catalogs covering models, vector stores, and external APIs. High-level abstractions — chains, graphs, agent loops — compress orchestration code during prototyping. The same abstractions make production failures harder to trace and reproduce.

Prompt optimization. A Python-only category. One framework treats prompts as parameters: you define the input → output signature and a scoring metric; the optimizer searches for instructions and few-shot examples that improve that metric on a representative dataset. Useful when you have labeled examples and a measurable quality target — rarely where teams start.

Structured output. Libraries that wrap provider SDK calls with schema validation and automatic retry. Python structured-output libraries validate against Pydantic models; in TypeScript, Zod schemas serve the same role. If the model returns an invalid field or missing required data, the library retries with the validation error. Narrow scope; composes cleanly with any other framework.

TypeScript AI SDKs. Full-stack TypeScript libraries that unify model calls, tool invocation, response streaming, and client-side state management — keeping model logic in the same codebase as the UI. Strong for chat interfaces and streaming responses; for complex retrieval or prompt optimization, a specialist library still does the heavy lifting.

Most production systems combine two or three of these. A retrieval framework supplies context; a TypeScript SDK handles the streaming interface; a structured-output library validates the response before it writes to the database. That composition works — but it is also where the ownership problem compounds.

What do LLM frameworks not cover in production?

Here is what none of the framework README files addresses.

Behavioral versioning

When you update a prompt buried inside an orchestration chain, or when a dependency update changes how tools are invoked, there is no behavioral artifact. The code changes; the running behavior is not hashed, tagged, or linked to a passing test. You are one npm update or pip install --upgrade away from a silent regression, with no diff to show what changed in model behavior.

A typed contract at the boundary

Frameworks help you call models. They do not enforce what callers send in or what downstream services must receive. A retrieval pipeline will start embedding a null query and spend tokens doing it. Schema validation happens inside the framework when you add a structured-output library — not at the function boundary where another service calls your agent, where rejection costs nothing.

Eval tied to the artifact

You can write pytest suites or Jest tests against your framework code, and you should. But those tests run against the code on the branch right now. If the eval run and the deploy are not linked to the same versioned artifact, you are trusting that nothing changed between the last green test and the last push. That trust is usually misplaced after a model upgrade or a dependency bump. (See what an LLM eval actually measures for the scoring mechanics.)

Trace-level observability

High-abstraction orchestration frameworks hide intermediate state by design — that is the abstraction's point. When an agent calls three tools and retrieves from two indexes, the trace that shows which retrieval step returned the wrong context, or which tool call returned an unexpected status, does not exist unless you wired instrumentation through each layer yourself. Non-trivial, and usually skipped until the first production incident. The LLM observability guide covers what that instrumentation layer needs to capture.

Ownership

When the system prompt lives inside an orchestration file last modified six months ago, who owns it? In practice: whoever last touched it. When it regresses at 2am, the on-call engineer finds no changelog, no rollback path, and no failing test that isolates the cause.

The production layer that works with any framework

Each gap has a concrete fix.

Typed input/output contracts catch bad calls before a single token is spent. Define exactly what the caller can send and what the agent must return. Reject at the boundary.

A versioned, hashed artifact makes "what you tested is what runs" verifiable. When the artifact changes, the version changes. The URL and keys stay stable — callers never update their code — but the running logic is exactly the artifact that passed eval.

Eval tied to the artifact runs a scored dataset against the candidate version before it ships. Score drops surface before users see them. A human reviews the findings and decides whether to deploy. The LLM evaluation platform post explains how scoring and trust thresholds work end-to-end.

Structured traces at every step — retrieved chunks, tool calls, intermediate responses — make failures reproducible rather than mysterious.

This is the layer Functional AI provides. Your agent logic stays in whatever framework fits the job; Functional AI wraps it as an Agentic Function with a typed contract defined in function.yaml:

# function.yaml
slug: classify-support-ticket
description: Classifies a support ticket by category and urgency
input:
  type: object
  properties:
    ticket_text:
      type: string
      maxLength: 4000
  required: [ticket_text]
output:
  type: object
  properties:
    category:
      type: string
      enum: [billing, technical, account, feature_request]
    urgency:
      type: string
      enum: [low, medium, high, critical]
    reasoning:
      type: string
  required: [category, urgency]

Initialize, deploy, and call it:

fn init
fn deploy classify-support-ticket

fn run classify-support-ticket \
  --key fnk_… \
  --input '{"ticket_text": "My account is locked and I have a demo in 2 hours"}'

{
  "status": "success",
  "output": {
    "category": "account",
    "urgency": "critical",
    "reasoning": "Account lockout with imminent business impact warrants critical urgency."
  },
  "latency_ms": 142,
  "cost_usd": 0.003
}

The framework powering the logic — retrieval pipeline, structured-output library, or orchestration graph — is invisible to the caller. The typed contract is what the caller sees, and what Functional AI enforces.

Catching regressions before they ship

Before deploying a new version, fn eval runs your scored dataset against the candidate:

fn eval classify-support-ticket --dataset support-tickets-v3 --scores 4

Illustrative output:

Experiment 42 complete
  Dataset: support-tickets-v3 (200 rows)
  category_accuracy:    0.97  (+0.01 vs v11)
  urgency_accuracy:     0.91  (-0.06 vs v11)  ← regression flagged
  reasoning_coherence:  0.89  (stable)
  p50_latency_ms:       138
Exit 1 — 1 score below threshold

That urgency drop — 0.97 → 0.91 after a model version update — would have shipped silently with framework code alone. With fn eval in the pipeline, a human sees the finding before users do. fn eval exits non-zero so your CI pipeline picks it up; a packaged CI gate that blocks the deploy ships after launch.

If a call does fail in production, you flag it into a test case in two clicks from the trace view, and it joins the next eval run. The failure dataset grows from real production calls, not hypotheticals.

Framework selection is a build decision; production safety is not

Teams that treat framework selection as the whole decision ship fast and debug slowly. The questions that actually matter in production are not "which framework?" but:

1. What is the typed contract? — exactly what inputs does this agent accept, what outputs does it promise, and where does bad input get rejected?
2. What exact artifact did you evaluate? — is the thing running in production the thing that passed your scored test?
3. When something goes wrong, how do you find it? — does the trace show the full execution path, including retrieval steps and tool calls, or only the final output?

Choose your framework for the build problem it solves. Then get started with Functional AI to add evaluation, runtime, and observability on top — without changing a line of your framework code. If you want the team that built the platform to build your agent with you, talk to an engineer.

FAQ

What is an LLM framework? An LLM framework is a build-time library that reduces boilerplate when building applications on top of language models. Different frameworks address different jobs: retrieval and RAG pipelines, multi-step agent orchestration, structured output validation, or TypeScript streaming. Frameworks do not manage runtime versioning, eval, or production tracing — that layer lives on top.

Do I need an LLM framework to build an agent? Not always. A provider SDK handles direct model calls. A framework becomes useful once you find yourself repeating the same retrieval, retry, or tool-invocation logic across multiple functions. Start with the SDK, add a framework when you hit a concrete maintenance problem, and keep the production layer separate regardless of which framework you choose.

Which LLM framework is best for TypeScript? TypeScript engineers have two main paths: a TypeScript-native AI SDK for streaming interfaces and UI state management, or structured-output libraries with Zod schemas for validated model responses. For complex agents with tool-calling and multi-step logic, TypeScript-first orchestration frameworks handle the sequencing. The choice depends on whether your bottleneck is retrieval quality, orchestration complexity, or output type safety.

Why do prompt regressions happen even with tests? LLM frameworks do not link your tests to the production artifact. A model upgrade, a dependency update, or a one-line prompt change can silently alter behavior after your last test run. Tying eval to the exact versioned artifact — so the thing you scored is the thing that ships — closes this gap.

What is the difference between an LLM framework and an LLM ops platform? An LLM framework is a build-time library: it helps you call models, retrieve context, and validate output. An LLM ops platform covers evaluation, runtime, and observability: scoring outputs against a dataset before deploy, running the artifact under a typed contract, tracing every production call, and rolling back without a redeploy. Both belong in the stack; they operate at different layers.