Xuanwo / Sketches

Lance · Blob v2 · Design proposal

Blob Reuse Index (BRI)

Reorganize the rows. Keep the bytes.

Compaction should be able to move rows without copying their unchanged blob payloads. BRI gives new DataFiles a durable reference to existing sidecars.

Packed — a byte range in a shared sidecar Dedicated — a whole sidecar Inline and External semantics unchanged

01 / The coupling

New file. New blob address.

A sidecar is a separate file holding blob bytes. A row stores a descriptor with a file-local ID; the containing DataFile’s stem supplies the directory.

Compaction writes surviving rows into new DataFiles. Under default addressing, those rows need new sidecars—even when the payloads are unchanged.

The row layout and payload placement are coupled by the address.

A new DataFile gives the payload a new address Compaction moves a row from F17 to F42; default sidecar addressing requires copying its unchanged payload from F17’s directory into F42’s directory. WITHOUT BRI COMPACT COPY F17.lance local #3 F42.lance local #1 F17 / blob #3 Payload F42 / blob #1 Same payload
One surviving value, with illustrative file and blob IDs. Other compaction inputs are omitted.

02 / The storage contract

New reference. Same bytes.

Add BRI to the DataFile entry in the table manifest. The physical .lance file still stores Blob v2 descriptors in its encoded column data; the reused sidecar still holds the payload.

The example below shows one surviving Packed value after compaction: its local blob_id becomes 1, while its original position and size are preserved.

01 / Physical DataFileLayout unchanged

F42.lance

<base>/data/F42.lance

Blob column → packed-struct descriptor values

FieldTypeValue
kindu81 · Packed
positionu6465536
sizeu6465536
blob_idu321
blob_uriutf8""

One descriptor per Blob value. These are decoded fields, not fixed byte offsets. BRI adds no field to the descriptor or the file footer.

02 / Table manifestMetadata extended

DataFile metadata

<table>/_versions/*.manifest

Manifest
└─ fragments[]
   └─ files[]  : DataFile
      ├─ path: "F42.lance"
      ├─ base_id: absent
      └─ blob_reuse_indexNEW · DATAFILE FIELD 8
sources[]
  base_id:      absent
  blob_dir:     "F17"
  local_ids:    [1]
  physical_ids: [3]

sources groups mappings by base and directory. Its absent base_id inherits F42’s base; both are absent here, so the primary root applies.

local_ids[i] → physical_ids[i]: the sequences are positionally paired. Stored as RowIdSequence and EncodedU64Array, they are shown expanded here.

03 / Existing sidecar · payload unchanged

F42: blob_id 1 → BRI → (F42’s base, F17, physical ID 3)

Open <base>/data/F17/<encoded(3)>.blob, then read [65536, 131072). Here, encoded(3) stands for the existing blob-ID filename encoding.

Other bytes · 0–64 KiB Selected range · 64–128 KiB
One Packed value with illustrative IDs and byte counts; unrelated manifest fields are omitted. Compaction writes F42’s descriptor and its manifest mapping without reading or rewriting this sidecar. A Dedicated descriptor uses the same mapping and reads the whole object.

No BRI entry? Resolve the ID under the containing DataFile’s own directory. New and repacked sidecars use this ordinary path.

03 / The lifetime

Keep the reference. Keep the sidecar.

Cleanup follows every retained manifest, including historical versions. Each DataFile keeps its own sidecar directory live; BRI additionally protects the exact objects it references.

A retained reference keeps the sidecar alive After the old version expires and cleanup can remove F17.lance, a BRI reference from retained F42 metadata still protects the original sidecar. AFTER OLD-VERSION CLEANUP KEEP F17.lance Removed Retained manifest F42 · BRI #1 → F17 / #3 F17 / blob #3 Retained by the reference
F17.lance can be removed once no retained version needs it and cleanup’s safety rules permit deletion. Its reused sidecar stays live. Cleanup never deletes objects owned by another storage base.
Immutable mapping
Changing BRI requires a new physical DataFile.
Direct targets
Recompaction resolves the physical object, never a chain of old manifests.
Safe rollback
An uncommitted output owns its new files, never its reused sources.

The integration contract

Every consumer must honor the reference.

BoundaryRequired behavior
Manifest validationReject ambiguous IDs, invalid source bases, and misaligned mapping sequences when opening the manifest.
Readers & writersBRI requires both feature flags; unsupported clients reject it. Files without BRI retain default addressing.
ClonesShallow clones preserve mappings and effective bases. Deep clones copy the dependencies and remap them into independent storage.
Full rewriteRemoving the last BRI clears its flags in that manifest. Older retained versions with BRI still require aware clients.

Row layout and blob placement can evolve independently—connected by an explicit, durable reference.

Implementation contract, not a measured performance claim. Encoding details and policy defaults are in the linked source snapshot.