Closing the Loop Without Slowing the Request
By Sqwish Team ยท
The appealing version of closed-loop AI is easy to describe: a request comes in, the system decides what to do, the outcome comes back, the model updates itself, and the next request is better. It is a clean story because it compresses the whole product into one loop: act, observe, learn, repeat.
Production systems are less tidy, because requests arrive under latency budgets while outcomes arrive later, incompletely, and through different paths. Cost and latency are visible almost immediately, but user satisfaction or business impact may only become clear after a workflow completes or another system records what happened. Between the model output and the eventual outcome there may be retries, fallbacks, application logic, human intervention, and logs that are mostly correct but still imperfect.
Putting all of that directly in the request path turns learning into a reliability risk. The request-time system has to remain fast and bounded even when feedback has not arrived, training jobs are running, configuration is changing, or a provider is behaving badly.
That is the practical constraint we start from at Sqwish Labs: real-time optimization means making a better decision for the request in front of you, under the constraints that exist at that moment, while preserving enough evidence to make future decisions better. It does not mean retraining the system after every new request.
LoopOne, Sqwish Labs' first product, sits in the application layer and chooses among eligible ways to handle a repeated unit of work in an AI application, under the customer's configured objectives and constraints. Today, that often means selecting between candidate model and prompt strategies. The same shape can apply to other parts of the application layer that affect outcomes, including context, retrieval, memory, tool use, or broader workflow policy. Selection is the visible part of the system, but not the hard boundary. The harder problem is preserving the evidence, contracts, and safe update path around each selection, so what the system chose, what actually happened, and what followed from it can all be used as evidence later.
We think about that system as one loop made of three components: Serve, Observe, and Learn.
The separation itself is the architecture. Serve is optimized for request-time decisioning, Observe for attribution, and Learn for safe improvement. Each component runs at a different speed, so the boundaries between them need to be explicit rather than assumed.
Serve: decide now
Serve is closest to the customer's request. It receives the request, resolves the relevant task and objective, evaluates the eligible actions, returns a decision, and records enough lineage to explain why that decision was made. Because this is the millisecond-scale component, it cannot wait for training, analytics queries, or control-plane state to be rebuilt while a user is waiting for an answer.
That constraint shapes the runtime. The state needed to make a decision has to be prepared before the request arrives: the task being optimized, the actions that are allowed, the current objective, the constraints that must hold, and the learned decision state that is safe to use. A prototype can often load and check these things on demand, but in production that makes control-plane latency and consistency part of request-time behavior.
The design choice is to separate the durable source of truth from the serving view. Configuration, outcomes, and learned versions need durable records, while the hot path needs a coherent snapshot it can use quickly. That snapshot can change while traffic is flowing, but a request should not see a half-updated mixture of old objectives, new action definitions, and incompatible learned state. Safe refresh, consistent serving views, and previous-safe-state behavior are what let the system improve without making freshness itself a source of outages.
Fallbacks follow from the same constraint. In an adaptive system, fallback behavior is part of the product contract: when the system cannot safely evaluate the configured objective, it should take a path the customer has agreed to rather than improvise a partial objective and treat the result as optimized. That might mean using a baseline, taking a configured safe action, or failing clearly, and in each case the reason should be recorded. A silent fallback is hard to debug, and it can make future training data look more intentional than it was.
Serve therefore has two jobs: make the best bounded decision it can make now, and leave behind a faithful record of the decision the rest of the system will later learn from.
Observe: preserve the shape of what happened
Observe starts with a simple question: what happened?
Most AI systems log what the model said, which is useful, but not enough for a system that learns from production consequences. Recording that "prompt X ran on model Y" tells you something about an inference call, but it loses the role that call played in the product. A support answer, a ticket classification, and an escalation decision may all look like model calls in a trace, while still being different learning problems.
LoopOne organizes inference around functions: stable units of work in an AI application. A function is the repeated task the system is trying to improve, not merely a code path, endpoint, or prompt template. Once requests are grouped by that role, the system can compare different ways of doing the same job, attach outcomes to the right part of the workflow, and understand whether a change improved the task it was meant to improve.
This is also where the system departs from benchmark optimization. A benchmark gives the same target to every customer and every request, while a production function has a customer-defined metric, a cost envelope, a latency constraint, and a downstream consequence. The question is not whether one prompt or model wins on average, but whether this way of handling this function improves the outcome the customer actually configured.
That structure matters because closed-loop learning depends on comparability. If every inference is just another prompt-model pair in a log, production traffic becomes a loose mixture of prompts, models, retries, responses, and downstream effects. The workflow shape starts to disappear, and with it the ability to tell which decision created which consequence, or whether two historical examples are similar enough to learn from together.
Observation also has to respect time. Production feedback is not synchronized with inference: some signals are immediate, while others arrive only after a user responds, a workflow completes, or a business system records the result. Training on the newest available events without knowing which outcomes are mature lets fast-arriving signals dominate simply because they arrived first.
The result is a two-clock system. The serving clock cares about making a decision now, while the learning clock cares about whether the evidence is complete enough to trust. Dashboards may want freshness, but training needs attribution and maturity. Observe exists to turn production behavior into evidence that remains meaningful after the request has finished.
Learn: update later, activate safely
Learn is where accumulated evidence becomes new decision state. It consumes mature historical data, constructs training examples, updates the system's expectations about which actions work under which conditions, and makes those updates available to serving.
It stays separate from Serve because training can be slow, fail, or produce an update that is not safe for the current serving context. None of those cases should break the request path, which should continue making bounded decisions from the previous safe state while learning catches up.
One design principle follows from this: training produces candidates, not deployments. A newly learned policy is a proposal to the serving system, and the final part of Learn is activation: before it changes production behavior, the candidate has to be checked against the task, objective, action space, and representation the runtime actually serves. The serving side is the authority on what it can safely use, because it is the part of the system that bears the consequences of a bad activation.
This boundary is easy to blur. If the trainer can directly change what serving uses, learning and deployment collapse into the same operation, which can be convenient while the system is small but becomes fragile once objectives change, actions evolve, or representations move. The safer pattern is two-stage: learning produces a versioned candidate, the activation step validates it, and only then does serving state move. If validation fails, the previous safe state remains active and the failure is observable.
The same discipline applies to objectives. Real AI products rarely optimize a single clean number; a customer may care about quality, cost, latency, safety, or some downstream business outcome, and those priorities may shift as the product matures. The learning system has to preserve those trade-offs rather than bury them inside an opaque score. Otherwise, serving can end up making a decision that is mathematically valid but misaligned with the objective the customer actually configured.
The contract between components
Serve, Observe, and Learn only work as components of the same loop if the boundaries between them are mechanical. Serving and learning have to agree on what a function means, what an action means, what objective is being optimized, and what evidence is valid for training. Observation has to preserve enough structure to reconstruct historical examples, and activation has to know whether a learned update is compatible with the state that will serve it.
That is why we treat the inputs to learning as contracts rather than loose arrays of data. A system can keep returning numbers even when the meaning of those numbers has changed because the task, prompt, model, context, or representation shifted underneath it. In a benchmark, that may appear as noise, but in production it can make the system confidently prefer an action for reasons that no longer apply.
The practical answer is to make the important boundaries explicit. In that contract, functions define stable tasks, actions define eligible ways to handle those tasks, objectives define what the customer is optimizing for, representations define the meaning of the features or context the learned state depends on, and evidence maturity defines when an observed outcome is complete enough to train from.
The same contract mindset applies to configuration. In a learning system, configuration defines the optimization surface: what is being optimized, which actions are eligible, how constraints apply, how fallback should behave, and which learned state remains compatible. A reward-weight change and a prompt-template change are both configuration changes, but they do not mean the same thing. One may change the trade-off the customer wants to make, while the other may change the behavior of an action itself. The system has to understand the difference before it can safely learn across either change.
The engineering challenge is to make each action cheap enough to choose on every request, structured enough to learn from later, and constrained enough that adaptation remains safe. Selection is the visible interface; the loop around it is the product.
What we are building
The manual tuning loop in AI products is familiar. Teams inspect traces, run evals, watch dashboards, compare costs, change prompts, switch models, adjust retrieval, add rules, and then wait to see whether the change helped. Some of those changes help, some move the problem elsewhere, and many are hard to attribute because the product was not structured to learn from what happened.
A/B testing is useful, but it is still mostly an average-case tool: it compares a small number of fixed variants across a population and asks which one wins overall. LoopOne is built for a finer unit of evidence. By keeping outcomes tied to the request, function, action, and objective, the system can learn when one strategy is better for one class of requests and another is better elsewhere, rather than forcing every product difference into a single global winner.
That changes the operating model. The value is not only in any single policy the system learns, but in the growing record of what worked: which actions helped which requests, under which customer objectives, and with which outcomes. The hard thing is not choosing between models; it is building the production system that lets customer-specific outcomes safely shape future request-time decisions, without making production traffic wait for the learning machinery behind it.
That is why the separation between Serve, Observe, and Learn matters so much. Serving has to be fast where users are waiting, observation has to be patient where evidence is delayed, learning has to be strict where meanings can drift, and activation has to be conservative where a bad update could destabilize production behavior.
Many of the hardest problems sit where research and engineering meet: delayed feedback, safe exploration, proxy rewards, and action spaces that evolve over time. But the systems foundation has to come first. Before a product can learn from production, it has to know what it did; before it can change behavior, it has to know what state is safe to serve; and before it can optimize a request, it has to do all of that without slowing the request down.
Closing the loop is not about making the model update faster. It is about building the infrastructure that lets real outcomes shape future behavior while the product remains fast, observable, and safe enough to run in the real world.