Seven Lessons From Fifty Years of Systems

2026-08-30 · Engineering ·

You have seen this incident report before, maybe in your own repository. Two agents pick up the same work item and both implement it — you pay twice and merge neither result cleanly. A stalled session wakes up after a newer agent already fixed the file, and writes its stale version over the fix. A handoff between sessions lives in a chat transcript, and the next session starts with none of it. None of these failures require a smarter model. They are the classic failure modes of distributed systems, and distributed systems solved them decades ago — with leases, fencing tokens, durable logs, declared state, and signed provenance.

Section 2 of the Koinessa white paper — part of a series on the verification gap in agentic development — argues that agentic engineering does not need new theory; it needs the old theory applied where agents actually work. Here are the seven lessons, what the original sources established, and how Koinessa applies each.

Lesson 1: keep the contract small — the narrow waist

The internet scaled because its middle stayed thin. Cerf and Kahn's 1974 paper established internetworking: heterogeneous networks connected by gateways speaking one common protocol (IEEE Transactions on Communications). Beck's retrospective on the hourglass model explains why the shape endured: a small, stable interface in the middle lets everything above and below it evolve independently (Communications of the ACM, 2019).

The AI ecosystem is re-discovering this shape with tool and agent protocols — but connectivity was always the easy half. The internet standardized packet delivery without solving identity, fraud, or trust, and agent protocols are on the same trajectory: delegated authority across agent chains — capability attenuation, revocation, provenance, end-to-end accountability — remains incompletely standardized.

Koinessa applies the narrow waist directly. It exposes 366 registered tools behind one search contract, and workers from six providers — local Ollama, Ollama Cloud, and frontier harnesses — discover capability through that same local contract. The waist has been exercised by an arrival: a sixth provider descriptor (Devin CLI) registered onto an existing transport kind with no new transport code, reaching the same tool contract through one configuration stanza.

Lesson 2: a transcript is not memory

Fowler's event sourcing pattern keeps the full history of events that produced current state, not just the state itself (martinfowler.com). CQRS is its companion: separate the broad read paths from the narrow, audited write paths so each can be shaped for its job (martinfowler.com). Together they answer the questions a transcript cannot: where did this come from, when was it true, has it been superseded, what evidence supports it?

Koinessa applies both. Agents read through the Brain gateway — retrieval is lexical, vector, and graph-based over the same store — while every external write crosses a single-writer boundary. The 2026-08-23 snapshot holds 17,903 Koinessa symbols, indexed the same day, with 54,201 symbols across the federation's twelve lanes. The temporal operational ledger (BRN-002, implemented 2026-08-07) records authority classes — a signed pipeline result outranks an agent's claim — plus supersession links and as-of-time queries. The as-of query turns incident review from archaeology into a lookup: what did the system believe when this change was authorized, which evidence existed then, what later superseded it? The Brain architecture post covers the substrate.

Lesson 3: partial failure is normal — plan for the agent that fails mid-write

Distributed systems assume components fail independently, and agent swarms inherit every one of those modes: a session dies after modifying files but before reporting; a tool call times out after succeeding; two workers believe they own the same resource. The classical controls each map to one mode:

The fencing scenario is the canonical agent-swarm accident. Agent A takes lease 41 on a work item and stalls. The lease expires; Agent B takes lease 42 and finishes correctly. A wakes and writes its stale result: without fencing it silently overwrites B; with fencing the resource remembers 42 as the newest valid token and rejects 41. Stale authority must never become current authority — no amount of model intelligence substitutes for that check.

On the human side, DORA's research on trunk-based development measured the same discipline: teams that keep branches short-lived, merge to trunk at least daily, and avoid code freezes achieve higher software delivery and operational performance (dora.dev). Concurrency works when batches are small and merges are frequent.

In Koinessa, session registration, cooperative file claims, and fenced work-item leases let interactive workers from six vendors — Kimi, Claude, Grok, Codex, local Ollama, and Ollama Cloud — share one board; the session registry holds 209 registered sessions, 208 of them since August 1. Attribution has a stated boundary: a provider label exists on only 8 of 209 session rows at this snapshot, so per-vendor attribution rests on commit trailers — 18 distinct worker identities since July 1 — which are not universal. The 2026-08-23 git snapshot shows 100 mainline commits from August 19 through August 23, 691 from August 5 through August 23, and a peak of 84 on August 13. Two human operators on separate LAN workstations have directed independent swarms against the same repository through the same coordination plane — operating evidence from one system, not a controlled benchmark.

Lesson 4: declare desired state and reconcile

Kubernetes replaced "run this and hope" with a controller loop: declare the desired state, observe the actual state, act to reduce the difference, repeat (kubernetes.io). OpenTelemetry supplies the instrumentation layer — traces and spans that make observed state visible (opentelemetry.io) — and Open Policy Agent carries the same idea into policy: declared, versioned rules evaluated outside the code they govern (openpolicyagent.org).

Koinessa governs agentic work the same way. A requirements document declares outcomes; work items bind obligations; verification profiles declare required checks; evidence schemas declare the proof that must exist. The controller then compares observed state against declared requirements — coverage exists or it does not; the artifact matches its schema or it does not — instead of asking the agent whether it feels finished. Reconciliation also supplies backpressure: when a downstream system slows or fails, controllers reduce or queue work rather than converting a small outage into a flood of retries. And once every operation carries a durable identity from intent to outcome, reliability becomes computable by agent, model, task class, and repository — so model selection stops being brand preference and becomes an evidence question.

Lesson 5: sign what happened

Supply-chain security reframed builds around provenance. SLSA defines graduated levels of build integrity and a provenance format describing where, when, and how an artifact was produced (slsa.dev, specification v1.2). in-toto provides a general attestation framework for supply-chain claims (in-toto.io). Sigstore and Cosign made signing and verification accessible enough to be routine (sigstore.dev).

Agentic development needs that discipline extended backward into source provenance: which requirement initiated a change, which agent wrote it, which context influenced it, which policies constrained it, which human approved it. Git records what changed; the system around it must record why, under whose authority, and with what proof. Koinessa's evidence bundle binds the source revision, the agent session and model, the work authorization, the tool versions, the checks executed with their canonical states, the policy profile that interpreted them, the approvals, and the final artifact digest — one chain from intent to deployment, covered in the evidence-gated development loop.

Two qualifiers matter here. Mapping to these frameworks does not confer a SLSA level or trustworthy provenance without verified trust relationships. And the Brain-side bundle, sufficiency, and outcome-evidence ledgers have writers and tests in place but held zero rows at the 2026-08-23 snapshot — built-but-gated, not yet exercised — while a third-party attestation rail is specified and in review, still planned. Equally important are canonical result states: a pipeline that only knows green and red will misclassify a missing scanner as a clean scan. Koinessa distinguishes PASS, FAIL, SKIPPED-NOT-APPLICABLE, INFRASTRUCTURE ERROR, TOOL MISSING, and EVIDENCE INVALID (live), plus ABSTAIN-SPEC-CONFLICT, a first-class route for an agent to declare work infeasible under its current specification, shipped with EGD-002 (live, 12/12 stories closed 2026-08-15). The measured alternative to a legitimate exit is gaming: an explicit infeasibility path produced a 54-to-9 result in a separate experiment.

Lesson 6: protect the objective

Goodhart's law, in Strathern's 1997 formulation from a study of audit in the British university system: when a measure becomes a target, it ceases to be a good measure (European Review). Optimizing systems learn to satisfy the measurement rather than the goal. In agentic engineering the canonical failure is the repair agent that deletes a failing test to make the pipeline green.

The rule that follows: the optimizer must not control the definition of success. Koinessa's local repair loop is bounded by construction — it may fix code, configuration, routing, or infrastructure, and it may not lower thresholds, remove meaningful tests, disable scanners, or convert required checks into advisory ones. Model improvement follows the same logic in a champion-challenger arrangement: alternatives run in shadow mode or on selected low-risk work, judged by independent criteria — verified success rate, cost per accepted change, repair cycles, human-intervention rate — and a human promotes a claim only after reviewing measured evidence. High-impact behavior earns authority through simulation first: a publishing agent validates actions without sending them; a deployment agent produces the plan without applying it. This is the core of risk-tiered autonomy: automation where failure is recoverable, human gates where it is not.

Lesson 7: data is not instruction

Saltzer and Schroeder's 1975 paper defined the protection principles for computer systems — least privilege, fail-safe defaults, complete mediation — with one idea underneath: protection is structural, not inferred from context (Proceedings of the IEEE).

Retrieval-augmented agents create exactly this surface: anything that reaches the context — an internal document, a tool result, a webpage, a customer message — can carry text a model may misread as authority. The systems answer is a boundary the model never gets to infer linguistically: provenance and authority must be structural fields on context, not properties guessed from prose. In Koinessa, every context-packet source carries a typed authority class — operator, verified, evidence, agent, inferred, unknown — with per-section provenance (live). EGD-002 hardened the boundary (live, closed 2026-08-15): an external-untrusted class, compile-time rejection of packets whose policy sections claim non-privileged authority, render-time fencing that marks low-authority content as data rather than instructions, and evidence-gated memory admission with contradiction checks at write time. A hallucination in a session is temporary; a hallucination admitted into organizational memory contaminates every future session that recalls it — which is why admission is a gated write, not a courtesy.

The takeaway

Duplicate work, overwritten changes, and lost handoffs are not AI problems waiting for a smarter model. They are distributed-systems problems with fifty-year-old answers: one narrow contract, memory with structure, leases and fencing, declared state with reconciliation, signed provenance, a protected objective, and authority as data rather than prose. Koinessa's contribution is not inventing these controls — it is wiring them into the plane where agents actually work, on infrastructure you operate, with evidence at every seam. The full argument, with every snapshot number and source, is in the Koinessa white paper.