The AI layer

How a model is chosen: three tiers, one resolver

Everything model-shaped in the kit — the chat box, the agents, the embeddings behind search — goes through a single pair of functions that resolve a client. They are the only readers of the provider configuration tables, and the only place in the codebase where a stored credential is decrypted. Feature code never imports a vendor SDK, never queries those tables and never sees a key: it asks for a client and calls it.

Chat resolves in this order, stopping at the first tier that can answer; embeddings take a shorter chain of their own:

Both platform tiers of the chat chain are one function that the resolver, the readiness endpoint and the model-assignment screen all read, so the thing the settings page tells you is ready is by construction the thing the runtime will pick.

Workers AI as the zero-key default, and what it costs

Because both environment configurations declare the Workers AI binding, chat and agents work on a freshly cloned workspace with nothing configured — no account signup, no key, no credit card in a form before you can see whether the thing is any good.

The chat floor is @cf/meta/llama-3.3-70b-instruct-fp8-fast, and embeddings default to @cf/baai/bge-m3 at 1024 dimensions. The embedding width is a column type, not a setting: a different width is a new table and a re-embed of every chunk, which is exactly why the original text of every document is kept.

Supported providers: Anthropic, OpenAI and compatible APIs

The shipped adapters cover Anthropic, an Anthropic-compatible mode for services that speak the same wire format behind a bearer token, OpenAI, an OpenAI-compatible mode that works against any local server, and Workers AI. A tenant's key is encrypted at rest and the API only ever reports whether a credential exists, never the credential. Before saving a provider you can test it: a ten-token completion or a single embedding, through the same client builders the runtime uses, so a passing test means the real path works.

On top of the client sits a small toolkit — prompt caching breakpoints, a forced structured tool call that validates its result against a schema and retries once with the validation errors fed back, the tool loop that is the agent engine, and the streaming chat. All of it is written against a client interface rather than a vendor type, which is what lets the test suite drive the whole AI layer with a scripted fake and assert on the exact system prompt, tools and settings a route sent.

Per-tenant request defaults are injected where the client is built, never at the call sites. Extended thinking is off by default and sent explicitly rather than left unset, because a reasoning model would otherwise bill for thinking that a chat surface throws away. Every provider failure is normalised into one error type with a small set of codes, its message run through a redactor — a vendor's error body can echo the key it just rejected — and rendered to the user as a sentence, never as the raw response.

Prompts live in code; a tenant override is a row

Every system prompt lives in a registry in the source — a key, a title, a description, its variables and its default text. Nothing hard-codes a prompt string into a route or an agent. A tenant that wants to change one gets a row keyed by that prompt, and reverting is deleting the row.

Placeholders are filled at resolve time, and an unknown placeholder is left visible in the output rather than silently blanked, so a typo shows up in the first response instead of quietly removing half the instruction. Adding a prompt is one registry entry and no migration; the per-agent model assignments key off the same registry.

Chat: persisted conversations, streamed to the browser

Conversations and messages are ordinary rows. Ownership is enforced as a filter on every query, so another member's thread — an administrator's included — is a 404 rather than a permission error. Creating a conversation resolves the client first, so a tenant with no provider gets its 503 before any row exists, and the provider and model are frozen onto the conversation.

Sending a message does everything that can fail as ordinary JSON before the stream opens: resolve the client, resolve the prompt, load the recent turns, insert the user's message. Only then does it start emitting frames.

Inside the stream there is one more subtlety worth knowing: the request's database client is closed the moment the response object is returned, which is before the stream body runs. So everything written during the stream — the assistant message, the conversation's timestamp, the auto-generated title, the usage row — happens on a second client opened for the stream and closed with it, and all of it is awaited rather than deferred.

Agents run on durable workflows; routes only enqueue

An agent is a registry entry: a key, an input and output schema shared with the client, a prompt key, and a function to run. Starting one is a request that validates the input, inserts a queued row, creates a durable workflow instance whose id is the row id, and answers 202. No route ever calls an agent's run function.

The workflow is three steps — claim, execute, finish — each opening and closing its own database client. The claim is the row itself: an update that only matches a queued or running status and returns the row. Every terminal write carries the same condition, so a settled run is never rewritten and a retried step simply re-claims. Exclusivity is a partial unique index over a slot each run holds: one per organisation for an exclusive agent, one per caller-chosen key, or none at all for an agent that may run many times at once. A second request for a taken slot gets the existing run back rather than a race.

Progress is durable: each step, tool call and chunk of text becomes a numbered event row, and a nudge over the websocket tells any open viewer to re-read. Cancellation is cooperative — a request marks the row and the run checks between turns — and reads reconcile, asking the workflow engine about any run that still claims to be active and settling it if the engine has never heard of it. There is no sweeper cron, because reconcile-on-read covers the same ground at the moment somebody actually cares.

Agents at volume: batches, fan-out, budgets and policies

A non-exclusive agent can be started in batches of up to a hundred inputs in one request, which becomes one workflow call, and inside a run it can fan out over a list of items a few at a time. Each item gets its own namespace for the run's durable keys, so a retried step replays the items that already finished instead of paying for them twice.

Every model call and decision a run makes writes its own usage row carrying the run's id, as the call returns — so a cancelled or failed run still accounts for every turn it paid for, and a run page can say what that run cost. A budget per run, per agent and per organisation per day is checked before each call, and a run that reaches one stops with a reason a person can act on.

An agent can also declare a policy for its tool loop: how many turns, tool calls and seconds of active time it may use, and how many failing calls in a row it tolerates. The limits count across retries, and time spent waiting on a person does not count. When a run runs out, the policy decides: make one forced call for a final answer over what it has found so far, or fail with the reason. A separate authorisation hook can refuse a tool call by rule before it runs or before anyone is asked to approve it.

Changing an agent is checked against its history before it ships: a replay script runs the agent's current code over its settled runs, answering every recorded decision, effect and human answer from the database, and reports which outputs would change — read-only, with no model calls unless asked for.

The three knowledge tools every agent gets

Three tools come attached to every run, all bound to that run's tenant: search the knowledge base, fetch a document as a window of characters, and list what is indexed. Search returns whole passages grouped by document, each located inside it — passage n of so many, and the character offset to hand straight back to the fetch tool.

The governing principle is that a tool answer is JSON the model can act on. A dense search always returns its nearest neighbours, so every non-empty answer carries a note telling the model to judge relevance for itself rather than assume it. An organisation with nothing indexed gets a description of what does exist and a hint, not an empty list. An unknown document id gets the documents that do exist. A missing embeddings provider gets a named error and a hint. There is never a bare "nothing found", because a dead end with no next move is where a tool loop starts guessing.

Document ingest, hybrid search and the usage ledger

Text arrives two ways and takes one path. Pasted text and text-like uploads are chunked paragraph-aware and indexed. Binary uploads — PDF, Office, OpenDocument, HTML — keep the original in object storage and enqueue a conversion job that turns them into Markdown through the same Workers AI binding, then runs the same indexing. Everything that could fail with a 503 is checked before anything is written, so a missing provider never leaves an orphan row.

Retrieval is hybrid: a dense cosine search over an approximate-nearest-neighbour index, and a lexical search over a text-search vector, each contributing a pool of candidates that are fused by reciprocal rank fusion. Every hit reports both of its ranks and its position inside the document, so a person or an agent can jump to the passage rather than to the file.

Every model call writes one usage row: the feature, the provider, the model, the token counts, and a cost. The cost is computed at write time from a single price table and frozen onto the row, so correcting a rate later cannot rewrite history. A model the table does not know is recorded as unpriced rather than guessed at, and the summary counts those calls separately so a partial total says that it is one.

Tracing: a span tree for every run, in Postgres and in any OTLP backend

Every chat turn, agent run and AI job is recorded as a tree of spans. The root is the agent or the chat turn; beneath it sit each model call with its tokens, latency and finish reason, each tool call with its arguments and result, and — under the tool that asked for them — the retrieval and the embeddings request. Nothing is instrumented by hand: tools are traced by the one function that runs them, and retrieval and embeddings nest under whatever span is active.

An agent run executes as several separate Workflow steps, so its trace id is derived from the run id rather than carried between them. Every step computes the same id and lands in one trace, and each attempt is a child of a root that is written once the run has settled.

Spans always land in a tenant-scoped table, pruned nightly after fourteen days, so rocketflare traces show works with no account anywhere. The same spans are exported as OpenTelemetry using the GenAI semantic conventions, over plain fetch with no SDK, to Langfuse, Arize Phoenix or any OTLP endpoint. A preset fills in authentication, so existing Langfuse keys keep working with no new secret, and Phoenix gets the protobuf encoding it requires.

Evals: cases, judges and a baseline, run against the real code

pnpm eval runs suites of cases against the chat route and the agent runtime themselves, in-process, rather than against a copy of the prompt in a notebook. A chat case drives the real route; an agent case runs the same enqueue, claim, execute and finish steps the workflow does, inline. Each case gets an organisation of its own with its documents ingested first, so retrieval can only find what the case put there. The suites live in their own workspace package on vitest-evals, and never run in the gate: a model having a bad afternoon must not block a merge.

A case is one line of JSON that carries its own expectations — the tools it should call and in what order, words the answer must contain, a rubric in plain language, a reference answer — and a judge only scores the cases that declare something for it. The deterministic judges (trajectory, contains, schema, budget) are free and never flaky. The model judges score a rubric, check every claim against the passages retrieval actually returned, and compare with a reference. The judge resolves through the same resolver as everything else, so it can be pinned per organisation or overridden on the command line, and its calls are priced in the usage ledger.

Every run is a JSON file stamped with the commit, the models and a hash of each prompt. A good run becomes a committed baseline, and pnpm eval --compare lists every case and judge that dropped and exits 1, which is what an optional CI workflow runs on a labelled pull request. A local report UI walks the transcripts and each judge’s reasoning. --provider and --model put the same suite on another model, so comparing two is a run on each and a diff of score, cost and latency.

Real traffic feeds it. Thumbs on a chat reply or an agent’s output are stored per organisation and written into that answer’s trace. An admin lists the thumbs-down from the CLI, and rocketflare evals promote drafts a case from one — the question, the earlier turns, the passages retrieved, the tools called and the answer somebody disliked — after warning that it is about to copy tenant data into the repository.