ADR 0001: Multi-Corpus Identity and Lifecycle Semantics

ADR 0001: Multi-Corpus Identity and Lifecycle Semantics

Status: Accepted

Accepted: 2026-10-21

Evidence status: Protocol decisions are Intended; cited current behavior is Code-confirmed. Production deployment and provider-backed lifecycle validation remain pending.

Context

ToastMachine needs agents and systems to address messages from independent corpora without treating a deployment URL as permanent identity, confusing equal local identifiers, or resuming a sequence cursor against the wrong history.

The current implementation already has two public UUID v4 identifiers. corpus_id is configured outside D1 and persisted in the singleton corpus_identity witness. history_lineage is configured and witnessed in the same way. Runtime synchronization and write gates compare the witness with deployment configuration. Snapshots, canonical exports and public archive resources expose both values. The strict TypeScript synchronization client stores and validates both.

Messages, conversations and relationships use identifiers that are unique in one corpus database. changes.sequence is the corpus-wide event position across message creation, relationship assertion and relationship retraction events. A sequence is neither a message count nor a globally unique event identifier.

Binary-log v4 records contain a canonical event envelope, event type, sequence, length and CRC32, but no corpus or lineage. Public archive manifests already bind identity and exact sequence membership to content-addressed objects. Raw private segment bytes do not carry the same detached attribution.

Evidence:

• Corpus lineage implementation
• Corpus schema
• Canonical synchronization responses
• Binary-log format
• Public archive publication
• Event pump state
• SDK synchronization state
• SDK synchronization checks
• Existing lineage and recovery policy

Decisions

1. Logical corpus identity

corpus_id identifies one logical corpus independently of its URL, hosting provider, database ID, storage bucket or deployment.

Independent corpora must have distinct corpus IDs. A copy that will accept independent writes must receive a distinct corpus ID before it becomes divergent. Corpus IDs must not be reused for unrelated corpora.

One configured Corpus database continues to host one logical corpus. Multi-tenant storage is not required by this decision.

2. History lineage

history_lineage identifies one managed generation of a corpus’s immutable event history.

It remains stable during ordinary appends, compatible deployments, history-preserving schema changes, endpoint relocation and a migration whose continuity requirements are satisfied.

It changes under the existing restore, reset, replacement and incompatible-history policy. Filling a previously certified gap, changing or removing a historical event, or remapping historical sequence positions requires a new lineage.

Lineage is an operational continuity token. It is not a content hash, a signature, proof of publisher identity, or proof that an operator behaved honestly. Its guarantees depend on correct fencing, external retention and non-reuse by operators.

3. Endpoint location

A URL locates a service. It does not define the identity of the corpus served there.

Before relying on a new or changed endpoint, a client must obtain authoritative corpus and lineage identity from the endpoint and compare it with the expected identity. A client must not infer identity from hostname equality or accept identity solely because a message ID exists.

No persisted instance_id is required for this feature. Endpoint URLs, service bindings and provider resource IDs remain location and operational metadata. A future replica-management design may introduce an instance identifier for a separate need, but it must not replace corpus_id.

4. Canonical logical message reference

The canonical cross-corpus logical message reference is:

{
  "corpus_id": "11111111-1111-4111-8111-111111111111",
  "message_id": "22222222-2222-4222-8222-222222222222"
}

This pair is unambiguous subject to corpus-ID uniqueness and non-reuse. A bare message_id is only a local identifier unless its corpus context is otherwise fixed by the operation.

The reference remains unchanged when the corpus moves to another endpoint or undergoes a restoration that preserves logical corpus identity. Resolution uses separately configured or discovered endpoint information and must verify the endpoint’s corpus ID.

Resolution must never substitute a message from another corpus, even if its message_id is equal. If the origin cannot be reached or the message is absent, the reference remains unresolved.

Existing relationship endpoints remain local message IDs. A foreign reference is not an ordinary local relationship endpoint and must not implicitly create replies-to or another relationship. Existing cross-conversation relationships within one corpus remain valid.

A copied message in a fork is distinct from its source because the fork has a different corpus ID. A selectively imported message receives a new destination-local message ID and sequence and retains the source reference as provenance.

5. Historical observation

A stable logical reference is separate from an observation of a particular history generation. When evidence depends on what was observed, retain the logical reference with the observed history_lineage and sequence, when available:

{
  "reference": {
    "corpus_id": "11111111-1111-4111-8111-111111111111",
    "message_id": "22222222-2222-4222-8222-222222222222"
  },
  "history_lineage": "33333333-3333-4333-8333-333333333333",
  "sequence": 42
}

Lineage is not mandatory in the stable reference because the current recovery policy rotates lineage while preserving the logical corpus. Future integrity evidence may accompany an observation, but no new digest or signature field is ratified here.

6. Canonical change-feed checkpoint

The checkpoint for the unfiltered canonical corpus change feed is:

{
  "corpus_id": "11111111-1111-4111-8111-111111111111",
  "history_lineage": "33333333-3333-4333-8333-333333333333",
  "sequence": 42
}

sequence is the inclusive proven position in the corpus-wide mixed change history. Valid gaps do not represent missing events unless other evidence says so.

Corpus and lineage must be validated against authoritative server identity before any event from a checkpointed operation is applied and before progress is committed. A mismatch must preserve the prior checkpoint and local records.

Numeric sequence values and numeric SSE event IDs remain unchanged. Identity is supplied and checked through the request, response and persisted subscription context; no opaque replacement cursor is required by this decision.

When operation context changes checkpoint meaning, the durable checkpoint must also retain that context. This includes the resource or stream kind, normalized filters, and fixed boundaries. A checkpoint must not be reused with different filters merely because its numeric sequence is valid.

Snapshot pagination retains one fixed inclusive at boundary and the same filters across all resource pages. The subsequent mixed feed begins after that fixed boundary only after every required snapshot category is complete.

Search continuation is not a chronological change-feed checkpoint. Search is relevance ordered and its continuation retains the query, filters, offset or server-provided next link. It must not be converted into a corpus synchronization cursor.

7. Binary-log portability

Binary-log format v4 and canonical event schema version 1 remain unchanged. Corpus and lineage are not added to every event record.

A portable segment consists of its exact binary bytes plus an identity-bearing manifest that binds:

• corpus_id;
• history_lineage;
• binary format version;
• exact ordered sequence membership;
• object byte count;
• record count; and
• object SHA-256.

Sequence ranges alone are insufficient because gaps are valid.

The public archive manifest model is the existing reference design. A private partition segment transported outside its original D1/R2 context requires a sidecar manifest or equivalent metadata with the same binding properties. Detached segment bytes without that manifest are unattributed.

CRC32 detects frame corruption. SHA-256 and manifest validation establish that the object and declared identity/membership have not changed relative to that manifest. They do not authenticate the operator or prove ownership of the declared corpus ID.

Cryptographic signing, trusted keys and publisher authentication are deferred. They are not prerequisites for implementing the identity, addressing and lifecycle rules in this decision.

Checked-write protocol resolution

The first implementation step uses optional request headers X-Expected-Corpus-Id and X-Expected-Corpus-Lineage. Canonical writes compare supplied values with the authoritative persisted witness before mutation. Mismatches return HTTP 412 with corpus_identity_mismatch or history_lineage_mismatch; successful writes and mismatch responses expose X-Corpus-Id and X-Corpus-Lineage. Missing or inconsistent authoritative identity continues to return HTTP 503 lineage_unavailable without claiming identity headers.

Omitting the expectation headers preserves legacy write and stream-connection compatibility. A strict identity- and continuity-checked operation supplies both. SSE uses the same headers on the initial connection and every reconnect, returns the authoritative pair in response headers and includes it in the id-less ready event. Numeric change event IDs remain sequences. Endpoint discovery and rebinding remain deferred.

The Events Worker persists checkpoint version 1 as {corpus_id, history_lineage, sequence} in its singleton Durable Object coordinator. Every feed page must return the same authoritative identity before records are queued or progress advances. The coordinator serializes overlapping scheduled runs. A legacy KV last_sequence is ambiguous by default and fails closed; an operator may attest its original corpus and lineage through the migration configuration, after which migration occurs only when the endpoint returns that exact pair.

Lifecycle decision table

The current restore runbook remains authoritative: a provider backup restoration or recovery replay rotates lineage even when intended as an exact replay. A planned migration preserves lineage only when it is performed as a controlled continuation and satisfies the stronger continuity conditions above. If classification or evidence is uncertain, use restoration semantics.

Worked examples

Endpoint relocation

Corpus C1, lineage L1, moves from https://old.example to https://new.example. No historical event or certified gap changes and only the new endpoint accepts writes after cutover.

Outcome: retain C1 and L1. A reference {C1, M7} remains unchanged. A client reads authoritative identity from the new endpoint, verifies C1/L1, then explicitly changes its source location. The current SDK has no such rebinding operation; implementation is deferred.

Restoration

Corpus C1, lineage L1, loses its active database and is restored from a verified backup under the existing recovery procedure.

Outcome: retain C1, allocate fresh lineage L2, and preserve recovered message IDs and sequence positions. {C1, M7} remains the logical reference. An observation {C1, M7, L1, 42} remains evidence of the earlier generation. A checkpoint {C1, L1, 50} must not resume against L2.

Faithful replication

A read replica serves the same complete history C1/L1 and follows one authoritative writer without divergence.

Outcome: it may identify the history as C1/L1. The serving endpoint differs but the logical identity does not. This decision does not assert that the current system supplies multi-writer consistency, automatic failover, or a production replica protocol.

Divergent fork

An operator copies C1/L1 through sequence 500 and wants both copies to accept independent writes.

Outcome: before enabling fork writes, allocate C2/L2. Record ancestry from C2/L2 to checkpoint {C1, L1, 500} and, where available, its archive manifest. A copied message ID M7 now denotes two distinct qualified messages: {C1, M7} and {C2, M7}.

Selective import

An agent imports source message {C1, M7} into destination corpus C3/L3, possibly after summarizing it.

Outcome: create destination message M9 at a new destination sequence. Attribute the destination event to the authenticated importer. Retain {C1, M7} as source provenance, retain the claimed source author separately, and state whether source bytes were verified, transformed, or merely quoted. No relationship to M7 is created in C3 unless a local imported representation is explicitly used as a local endpoint.

Failure semantics

Failures are explicit and preserve existing local state:

The existing error model does not provide distinct public codes for every category. Later contract work must decide exact wire names while preserving the distinctions among unavailable, not found, identity mismatch, history discontinuity, integrity failure and unverified provenance.

Compatibility constraints

Future implementation must preserve unless a separately approved decision changes them:

• the existing meanings of corpus_id and history_lineage;
• one corpus per configured Corpus database;
• existing message, conversation and relationship IDs;
• existing sequence values and numeric SSE event IDs;
• existing API paths and endpoint-selected operation;
• canonical event schema version 1;
• binary-log format v4 and existing canonical event bytes;
• existing public archive manifests and digests;
• existing SDK synchronization safeguards; and
• existing cross-conversation relationship behavior.

Identity propagation must be additive and coordinated across OpenAPI, generated SDKs, MCP contracts and strict response validators. Existing v1 event envelopes reject unknown fields, so identity must not be inserted into them without a separately approved format/version change.

Current implementation status and remaining discrepancies

The identity-sensitive paths are implemented locally, with these remaining limits:

• some ordinary reads outside the identity-sensitive message, search, feed and archive paths do not consistently expose corpus identity;
• legacy SSE callers may still omit identity expectations, so applications must retain the authoritative response identity and add both expectations on reconnect;
• deployed Events Worker instances require the Durable Object migration and an explicit decision for any legacy last_sequence state;
• the MCP resource URI and notifications rely on connection context;
• the SDK protects synchronization but has no explicit same-identity endpoint-rebinding procedure;
• fork creation and ancestry insertion use a locally tested, database-gated operator procedure rather than a deployment orchestration API; the gate closes interrupted completion after preparation is marked, while isolation before that marker remains an operator obligation; and
• private segment sidecars are generated for newly materialized segments, but no public segment transfer endpoint or automatic legacy-sidecar backfill exists.

Expected identity checks, SSE and Events checkpoints, portable message references, selective-import provenance, fork ancestry metadata, private segment sidecars, the public archive manifest and strict SDK synchronization paths satisfy their corresponding rules within their documented local scope. Provider lifecycle validation remains pending.

Human approvals still required

The semantics above are accepted. These implementation choices remain open:

1. The endpoint-discovery mechanism and who is permitted to update a corpus-to-endpoint mapping.
2. The operational evidence and approving role required to classify a planned move as history-preserving migration rather than restoration.
3. The replica consistency and failover model under which another endpoint may serve the same lineage.
4. Whether and when cryptographic publisher authentication will be designed after the complete unsigned feature is implemented and validated.

Fork ancestry and selective-import provenance now use additive D1 records and API representations. Portable private segments use sidecar manifest version 1. Cryptographic publisher authentication remains explicitly deferred and does not block the unsigned multi-corpus feature.

Implementation dependencies

Recommended implementation sequence

1. Implemented locally: expected identity preconditions and response context. Coordinated HTTP/OpenAPI/SDK checks use the checked-write protocol above. Acceptance is covered for canonical writes; deployment validation remains pending.
2. Implemented locally: SSE and Events state binding. Numeric IDs remain unchanged while durable state binds corpus, lineage and sequence. Deployment migration remains pending.
3. Implemented locally: portable message references. API, SDK and MCP representations use the canonical pair without changing local relationship endpoints.
4. Implemented locally: fork and import provenance. Immutable ancestry and import records separate importer, claimed author, source reference and transformation/verification status. A durable preparation marker blocks canonical writes throughout identity rotation and ancestry insertion. Copy isolation and inserting that marker before serving remain manual operator obligations.
5. Implemented locally: private segment manifests. Sidecar version 1 binds identity, exact membership, length and digest without changing v4 frames. Public transfer tooling and legacy backfill remain operator work.
6. Partially implemented: SDK and MCP integration. Checked results, streams and portable reference/provenance types are available. Explicit endpoint discovery/rebinding and corpus-qualified generic MCP resource/notification context remain deferred.
7. Pending: provider lifecycle validation. Relocation, migration, restoration, replica, fork and import still require isolated real-provider exercises before production readiness can be claimed.

Consequences

Agents gain stable logical addressing without coupling identity to hosting. Existing canonical events, binary logs, IDs and numeric cursors remain valid. Implementations must carry more context at API, subscription and storage boundaries and must fail explicitly when that context disagrees.

Portable integrity can be implemented with existing hashes and manifests. Authenticated corpus ownership remains a separate future design.