The AI ladder: retrieval, adapters, humans, and who holds the keys

0. Vocabulary

Four readers use these words — a business analyst, a developer, a compliance reviewer and a designer — and several of them are used loosely elsewhere. This document uses them this way.

1. The challenge

What goes wrong, described by what it looks like from outside:

2. The solution: climb only when the data says so

The rule that does not bend: anything that changes stays at rung 2. A price trained into weights is wrong the day it changes and cannot be cited when a customer disputes it.

When is the data ready for rung 3? Not by volume — by measurement:

1. An evaluation set first. Real questions with accepted answers, in each language served. Without one, "the model got worse" and "the model got better" are both guesses.
2. The evaluation shows retrieval is right and the writing is wrong. That is the only signal a LoRA adapter can fix.
3. Hundreds to low thousands of clean question → accepted-answer pairs for one behaviour. Consistency beats size. (A rule of thumb, not a measured threshold.)

3. CRAG as it is actually built

The corrective gate is not new here; each surface already refuses weak answers before it writes one.

Language is a route, not a weight. A question in Korean, Japanese or Chinese is matched against content in its own language first. Only when nothing clears the bar does it fall back to English, and the answer then says so in the asker's language. Scores are never blended across languages: a strong English match must not outrank a weaker Korean one that answers in the shopper's language.

What it costs: a strict gate says "I don't know" more often. That is the correct trade in commerce, and it is also the input the next section depends on.

4. Human-in-the-loop escalation

A refused answer is not an ending. It becomes an escalation with a tier, a clock and a record.

Design rules that make escalation useful, not just polite:

Why this feeds the ladder: every escalation is a question the knowledge base could not answer, paired with the answer a person gave — in both languages. That is the curated, personal-data-free dataset rung 3 needs. The humans are not a fallback for the AI; they are where its next training set comes from.

5. ADK vs Kubernetes/Helm vs the Cloudflare helmet

Three layers that are often confused because each "runs" something.

The equivalence that holds: an endpoint with permission checks in its function stack is an ADK tool with a guard — not an ADK agent. The agent is the layer that chooses which tools to call. Because the permission lives in the endpoint, the agent is swappable: ADK, an edge model, or a database vendor's own agent can sit on top, and none of them can do more than the endpoint allows.

Names are part of the contract. A swappable agent only helps if the tools it calls are named and shaped the way agents already expect. Google publishes its API design rules as API Improvement Proposals: resources addressed by hierarchical names (accounts/{account}/products/{product}, AIP-122), a small set of standard methods, and standard paging and errors. The merchant catalog tools here copy the names and shapes of Google's Merchant API, so an agent built against Google's own tools reads this catalog without remapping. Following the naming rules does not grant anything: the endpoint still decides.

Helm and the helmet share a shape, not a job. Both are one template, different values per tenant. Helm places a model server in Seoul or Iowa; the helmet decides, on the request, that a Korean visitor must grant an itemised cross-border consent before anything is stored — and that decision itself depends on a value (where this deployment keeps personal data), because a Seoul deployment must not ask it.

6. Mandates: the only way an agent acts

The helmet wraps every agent action in the same check. The agent brings a mandate, never a key.

Why a higher-order function and not a check inside each tool: a check written into each tool is a check someone can forget in the next tool. A function that wraps every tool call — takes the action and returns the guarded action — is written once, and a new tool is guarded by construction. It is the same reason consent loads first on every page rather than per feature.

What an agent never holds: a payment card (payment finishes on the processor's hosted page), a standing API key, or the shopper's login beyond the single step it was handed for.

7. SOC 2: what AI transport puts in scope

What a SOC 2 review is, precisely. It is a 360° procedural assessment: not of a product, a server or a database, but of how an organisation operates a system — who can reach data, how changes are approved and tested, how vendors are chosen and watched, how incidents are detected and answered, how data is kept and destroyed. An independent CPA firm tests those methods against the Trust Services Criteria and writes a report. The report describes a scope (which system, which criteria) and a period; it can carry exceptions where a control did not work as described.

Where the fines actually come from. A SOC 2 report does not fine anyone. Penalties come from the laws and contracts the controls exist to satisfy — US state privacy law (for example California's CCPA/CPRA, assessed per violation), the GDPR in Europe, Korea's PIPA — and from customer contracts that require the controls. A report with exceptions is usually where that scrutiny starts: it is the document a customer, an auditor or a regulator reads first.

Why AI transport changes the scope. The moment an organisation's DevOps sends data to a model — a hosted LLM, an embedding service, an agent runtime, possibly in another country — new methods have to exist, and a reviewer will ask for evidence of each:

In this estate, the honest status: most of the evidence above exists as running code and records — field-level encryption before storage, a consent log per grant, mandates checked on every action, retention and erasure jobs, a release gate. Two gaps are stated rather than hidden: agent-runtime sessions and memory are not yet in the erasure sweep, and the edge model used for public questions does not pin inference to one country. Mapping controls to the criteria is preparation for a review; it is not a SOC 2 report, and nothing here claims one.

8. What this costs

9. Summary