Luna: Changing a Connected System

The model returned an answer. Luna reported a failure. Both things were true.

During a model comparison for the Those Dark Arts conversational interface, a provider returned successful responses with answer text nested inside a different field. Our adapter expected the earlier format. It could not extract the answer, so a successful upstream request became a generation failure.

The same assumption existed in the evaluation harness. Repairing only the chat worker would leave the measurements describing different behaviour from the system they were supposed to test.

Repair the relationship

The initial change was small: accept the supported answer fields, prefer the established field when it is present, and reject missing or empty answers. Only the answer text should pass into the next stage. Reasoning fields, tool calls and provider metadata should not accidentally become the public reply.

The work around that rule needed more structure. One branch repaired the worker and evaluation harness. Another corrected the model catalogue against Loki's live observations. Tests followed the changed implementation; the comparison record followed the corrected catalogue. Independent verification waited for the branches to join.

Those relationships are what made the graph useful. Independent tasks could proceed together when their inputs were ready. Dependent work waited for something concrete. The graph also identified the join where the assembled change had to be examined as a whole.

The evidence had two distinct origins. Loki supplied live measurements from the provider comparison. The factory revisions checked behaviour using local doubles, fixtures and integration rehearsals. A passing fixture could establish that the extraction rule worked under a specified response shape. It could not establish the current quality, availability or latency of every live model.

The operator changes the scope

Loki then extended the commission to adapt his iterative resolution pattern for the public chat endpoint. This added responsibilities for the loop engine, its request-pipeline integration, the client and the tests.

Inside a request, a straightforward question could still use one generation. Selected harder questions could enter a bounded refinement path and then produce a final answer. That request flow is a different graph from the graph used to build it. One governs runtime behaviour; the other coordinates the people and agents changing the system.

The distinction becomes practical when an interface changes. The client depends on the pipeline's response contract. Loop tests depend on the engine and its integration. A worker finishing its file cannot settle all of those relationships by itself.

Review can change the design

A separate adaptation review challenged the first implementation. It retained the useful pattern of private refinement followed by a public answer, while changing several controls.

Sampling settings became fixed values owned by the operator. Two reflections became a hard ceiling, with an earlier exit when normalised text repeated. Raw retrieved material remained available to initial generation and final synthesis, but was no longer copied into reflection calls. Reflection was asked to refine relevance, completeness and structure using the draft and conversation.

These changes reduced a particular exposure surface. They did not make derived text trustworthy or prove that a model would always resist an instruction embedded in retrieved content. The final answer still needed validation and moderation.

The review's most consequential correction concerned failure. Once a configured refinement loop had actually begun, its initial answer was an intermediate draft. If the loop could not produce an acceptable synthesis, returning that draft as a fallback would violate the intended public boundary. The revised request returned a fixed error instead. An ordinary one-pass answer remained valid when the loop was disabled or had not engaged.

That decision changed the engine, integration and tests. The previous version had passed its checks; the review changed what the next version was required to do.

Carry the change through the whole system

The revised behaviour was checked through the worker and client, including failure paths. Model selection, response mode and the loop's time budget remained operator decisions. These September 4–5 revisions did not deploy the system, and they are not a description of today's live configuration.

What began as an answer in the wrong field exposed the central engineering problem: a connected system carries assumptions between its parts. The work graph makes those relationships explicit. Review can revise the design, the harness carries the resulting work, and integration checks whether the parts still honour the same contract.

Read The Graph Is Not the Runtime.

---

Author: Loki. Research and drafting support: Moiré, under Loki’s direction.