Indexed file V5 encodings

New ingestion and compaction files use V5. Readers retain V1–V4 support. Upgrade every collector, executor, compactor and blob-service reader before resuming writes. Old binaries cannot read V5; binary rollback requires converting or restoring V5 files first. Existing files are migrated through normal compaction. No instruction or public API schema changes are required.

V5 reuses the V4 schema, row-group headers and checksummed statistics, with LOGSHIP_INDEXED_FILE_V5 and footer version 5. Existing serializer identifiers remain unchanged; adaptive columns use serializer 5. Each column is a row-group compression unit; this format adds no compression pages.

Column envelope

All integers are little-endian. Offsets inside sections are relative to that section. The envelope is 47 bytes:

Metadata immediately follows the envelope, followed by values. Section bounds, counts, encoding identifiers and decoded lengths are validated before use. Scalar values retain their existing primitive byte representation. Null is distinct from an empty or zero value.

• All-null: no sections.
• Constant: one metadata entry (int32 byteLength, value bytes); optional values validity bitmap, least significant bit first. An absent bitmap means every row is valid.
• Plain: variable-width columns normally store gzip (uint32 offset, uint16 length) metadata and a separate gzip values stream. Offset uint32.MaxValue with length zero means null; a valid offset with length zero means an empty value. When decoded values reach 4 GiB, flag 4 selects (int64 offset, int32 length) entries, with length -1 representing null. Fixed-width columns instead store a gzip validity bitmap and dense gzip values, with zero-filled slots for nulls; their offsets are implicit.
• Dictionary: metadata entries (int32 length, bytes), in ID order, optionally gzip-compressed. ID zero means null; non-null IDs start at one. Values begin with an int64 offset per 1,024-row block. Each block stores either mode 0, bit width, and least-significant-bit-first packed IDs; or mode 1 and (int32 positiveRunLength, uint32 ID) runs. Use the smaller representation for each block.
• RLE: raw values records (int32 positiveRunLength, int32 byteLength, bytes), with length -1 representing null. Metadata holds (int32 runStartRow, int64 runOffset) for the run containing each 1,024-row boundary. Long runs are not duplicated at checkpoint boundaries.

IDs and runs remain seekable; gzip never wraps their checkpoint streams. Readers materialize selected values into output-owned buffers, so output survives cursor disposal and dictionary cache eviction.

Adaptive selection and resource limits

Input is consumed once into a replay store. Each scratch stream holds at most 8 MiB in memory, spills privately to delete-on-close storage, and is limited to 16 GiB. Disk exhaustion fails the write; partial files must not be published.

All-null and constant columns are selected directly. Dictionary analysis stops after 4,096 entries or 1 MiB of serialized dictionary metadata. Compaction and the legacy row writer compare a complete gzip-plain trial against only the best estimated dictionary or RLE candidate. A candidate must save at least 64 bytes and 5% of the stored plain representation, including metadata and checkpoints. Equal estimates favor RLE. High-cardinality columns fall back to plain.

HTTP ingestion uses a throughput policy: choose one encoding from bounded analysis, requiring an estimated 20% saving for dictionary/RLE, and avoid a second compression trial. Fixed-width columns stop dictionary work when an initial 128 dictionary lookups contain at least 120 distinct values; plain gzip handles such IDs/timestamps/GUIDs without building a full table. Variable columns retain full bounded dictionary analysis. Plain high-entropy variable values may remain uncompressed (flag 2 clear) while their metadata stays compressed. Existing V5 readers support these flags; this changes writer policy, not the format. Compaction can make the more expensive space-saving choice later. No new compression dependency is required.

Scan and compaction dictionary caches are capped at 8 MiB. Scan cache allocations are charged to the stage workspace; entries are immutable and output never references cached storage. Compaction decodes input in bounded batches, targeting 16 MiB across active column enumerators, plus dictionary and encoder scratch storage.

Metrics expose encoding selections, stored bytes, dictionary-limit fallbacks and encode/decode duration without per-row labels. Use the AdaptiveColumnBenchmark benchmark to compare encode/decode CPU and allocations across categorical, run-heavy, unique, nullable and constant inputs. File-size, sparse/dense query, compaction, compatibility and memory-progress checks are required before deployment.

Validation measurements

A local Release benchmark used 32,768 rows per column and compared adaptive V5 with the previous BasicNonNullable2+gzip serializer. These are synthetic warm-read measurements, not production query latency.

Decode times are BenchmarkDotNet means from seven calibrated iterations after three warmups. High-cardinality fallback is within 1% of the previous reader in this run. Encoding can consume additional CPU and temporary I/O because analysis and a fallback trial precede selection; the benchmark measures that tradeoff separately. There is no claim of universally faster writes or unchanged performance on every dataset.

Run calibrated benchmarks using:

dotnet run --project test/Logship/Logship.Service.Benchmarks/Logship.Service.Benchmarks.csproj --configuration Release -- --filter "*AdaptiveColumnBenchmark*"

For a quick file-size and allocation comparison, use the same command with --adaptive-summary after --. Validate representative production data and cold-cache/concurrent workloads before deployment.

Dictionary predicate access

V5 dictionaries, constants and all-null encodings expose an optional exact predicate-index capability. The executor evaluates its existing typed predicates against dictionary values once per row group, including the null entry, and retains a byte match mask indexed by dictionary ID. An unsupported capability falls back to scanning; it never denotes an empty result.

Dictionary rejection runs after min/max pruning and before batch memory estimation. If no value can match, neither row IDs nor later predicate dictionaries or projected columns are opened. Otherwise batches intersect their row-selection masks with the retained dictionary results. Dictionary leases are released after evaluation; retained masks and temporary typed evaluation buffers are charged to query working memory and disposed with the row-group filters. Decoded output continues to own its values.

This capability changes no persisted bytes or format versions. V1–V4 and plain columns retain scan behavior, and RLE retains run-based filtering. Future approximate indexes must verify candidates with the original predicate; their candidates must not be exposed as an exact filter. Future persisted index sections require an explicit format revision with versioned, length-delimited descriptors, rather than appending fields to the V5 envelope.

Counters expose dictionary entries evaluated (including the null entry) and row groups rejected by dictionary predicates. Benchmark with DictionaryPredicateBenchmark, or run the benchmark executable with --dictionary-summary for a quick warm-memory comparison. These measurements isolate predicate filtering and do not predict sparse projected-column decompression costs.

A local Release diagnostic over 131,072 rows and 256 distinct city strings measured 5.851 ms for a matching dictionary filter versus 21.839 ms for forced value scanning; absent matches measured 0.075 ms versus 11.693 ms. Corresponding managed allocations were approximately 0.75/13.72 MB and 0.056/13.72 MB. These are medians of seven warm-memory iterations after three warmups, including filter creation. They compare indexed filtering with a forced scan, not the previous per-batch dictionary implementation, and exclude projected-column fetches and HTTP execution.