Portable Binary Segments

Portable Binary Segments

Status: Sidecar generation and offline verification implemented for newly materialized private segments

Last reviewed: 2026-10-21

Binary-log v4 frames contain canonical event v1 bytes and numeric sequences. They deliberately contain no corpus or lineage identity. A raw .log object can be checked for frame structure and CRC32, but cannot by itself be attributed to a corpus.

The Corpus materializer now writes a canonical JSON sidecar at <private-object-key>.manifest.json after verifying the segment bytes. Portable segment manifest v1 contains:

• manifest_version: 1 and binary_log_format_version: 4;
• authoritative corpus_id and history_lineage;
• numeric segment_id and partition_id;
• the exact ordered sequences array, plus first and last sequence;
• the private object identifier, exact byte count, record count and SHA-256 of the exact .log bytes.

Materialization commits D1 certification only after the segment, read-back bytes, sidecar and sidecar verification all succeed. A newly created sidecar from a failed or uncertain attempt is deleted best-effort; the segment row, membership offsets and public manifest boundary remain unchanged. The primary immutable object may remain as an unreferenced partial artifact and is safely reused only after its digest verifies. A failed cleanup leaves a conflicting sidecar that fails closed and requires operator removal before retry. Local persistence tests cover failed sidecar storage, unchanged certification state, cleanup and a later successful retry.

Sequence membership is an explicit array because segments may contain gaps. Verification rejects unsupported versions, unknown fields, invalid identity, duplicate or non-increasing positions, boundary contradictions, missing bytes or manifest, byte/count/digest mismatch, decoded membership mismatch, invalid frames, invalid UTF-8, event-schema failure and CRC32 failure. A caller with an independently trusted expected identity can require exact corpus and lineage equality. Editing an R2 key does not change the manifest’s object identifier or identity.

createPortableSegmentManifest and verifyPortableSegment in the Corpus Worker are deployment-independent byte/JSON helpers. The repository has no public private-segment export or import endpoint. Operators transferring a private segment must copy both exact objects and invoke the verifier in their transfer tooling before accepting it. Older private segments without a sidecar remain valid inside their original D1/R2 recovery context but are not portable artifacts until an operator rematerializes and verifies an appropriate sidecar.

Public archive manifest v1 remains distinct because it certifies a hash-addressed, predecessor-linked public prefix rather than one private partition segment. It already binds corpus, lineage, binary format, exact sequence membership and object digest/counts. Both formats use canonical JSON, SHA-256, explicit membership and the same v4 decoder; historical public manifest bytes and IDs are unchanged.

Complete backup manifest v1 binds corpus/lineage at the backup root and verifies each private segment against D1 membership, exact offsets and committed events. That database witness supplies the identity and membership context for legacy segments without sidecars. The existing D1/FTS export and real restore limitations remain in the backup documentation; sidecars do not turn a partial export into a consistent backup.

These checks prove internal consistency and declared origin only. They do not prove who created a manifest, who controls a corpus ID, whether the history is truthful, or whether a publisher fabricated a consistent object and manifest. Signatures, trusted keys and publisher authentication remain deferred.