Blob storage durability
Configure the blob storage role, durability replicas, and recovery behavior.
Blob storage durability
Blob storage is an independent backend service role. Enable backend.services.blobStorageService on blob nodes; search executors can run on different nodes. The former searchExecutorService.blobStorage configuration is accepted as a legacy alias and moved to the independent role at startup. Configure one location only.
The master coordinates durability and stores its journal at <master.dataPath>/durability.sqlite. Blob nodes retain immutable replica content at <blobStorageService.dataPath>.replicas. Working blob partitions remain under blobStorageService.dataPath. Keep these directories separate from indexed-file data and use local filesystems with durable flush and SQLite WAL support.
Runtime settings
The normal console-host appsettings.json enables the blob role alongside the master and search executor, with working partitions under <dataRoot>/blobs. Setup seeds Logship.Blob.Durability.ReplicaCount to 1 when no value is saved. These defaults allow single-node HTTP testing without an extra configuration file. One copy provides restart recovery on that node; configure additional blob nodes and increase the runtime replica count for redundancy.
backend.services.setupService.systemSettingDefaults inserts missing settings atomically and preserves persisted runtime changes, including an explicit zero. The existing systemSettings provisioning section continues overwriting its configured values on startup. Deployments using other configurations retain the policy defaults below until configured.
Durability settings use the existing system settings API and administration settings screen. Master, blob nodes, and executor publication refresh them every five seconds. No durability configuration section is required on the master or executor.
All names below have the prefix Logship.Blob.Durability..
| Setting | Default | Purpose |
|---|---|---|
| ReplicaCount | 0 | Distinct blob-service nodes required per operation, including the owner. Zero disables new operations. Set explicitly to 1 for a single-node installation or a larger count for replicated storage. |
| MaximumJournalBytes | 1073741824 | Master SQLite database capacity. |
| MaximumReplicaBytes | 107374182400 | Retained replica payload and envelope budget per blob node. |
| MaximumOperations | 1000000 | Retained operation count per journal or replica store. |
| ReserveBytes | 1073741824 | Free disk reserve protected before new writes. |
| MaximumMutationBytes | 1048576 | Mutation metadata size limit. |
| MaximumContentDescriptors | 64 | Content ranges allowed in one operation. |
| MaximumOperationBytes | 2147483648 | Combined mutation and content size limit. |
| MaximumConcurrentOperations | 4 | Active durability operations per service. |
| MaximumQueuedRequests | 64 | Additional queued operations per service. |
| MetadataCacheKiB | 4096 | SQLite metadata cache size. |
| BusyTimeoutSeconds | 5 | SQLite lock timeout. |
| OperationTimeout | 00:05:00 | Deadline for each admitted operation. |
| MaximumIndexedFileRows | 1000000 | Maximum rows in a protected indexed-file publication. |
| MaximumRepairBytesPerPass | 67108864 | Content bytes examined per replica scrub/repair pass. One larger operation can exceed this budget to make progress. |
| MaximumRepairOperationsPerPass | 1024 | Operations examined per scrub/repair pass. |
An operation retains its chosen replica count and placement across retries. Lowering settings does not weaken an existing preparation or delete retained data. Lowering concurrency drains existing work; subsequent admissions use the new limit. Tightened input limits apply to new preparations, while older operation envelopes remain readable for recovery.
The blob service's existing local partition settings still control working storage, upload admission, ingestion, cleaning, and expiry. They are separate from the shared durability policy.
Acknowledgement and recovery
The master first records the operation identity, canonical SHA-256 digest, mutation, content descriptors, and replica placement. The owner streams verified content ranges directly to the selected blob nodes. Each replica reserves capacity, verifies exact lengths and checksums, flushes its files, and records completion in SQLite. The master verifies all required copies before committing a receipt with a monotonically increasing replay boundary.
Lost responses are retried using the same operation identity. A conflicting digest is rejected. An unresolved preparation is reported as pending rather than absent. The owner can resume copying from its retained source ranges. Master recovery also reconciles completed replica copies after restart. Missing replicas never reduce the required count automatically.
Internal RPCs provide paginated committed-operation replay and verified content reads. Automatic ownership reassignment and replica promotion are not implemented. Restore the original master journal after journal loss: existing replica stores reject a replacement journal identity. Back up the journal using a consistent SQLite backup, including its WAL when applicable; copying only a live database file is insufficient.
An existing owner can restore a missing working partition from the original journal and surviving replicas. Before replay, the master aborts unacknowledged preparations and fixes the receipt and retirement boundaries. Restoration builds a separate directory, omits terminal content, verifies checksums, and checks the recovered partition before publishing it. Stop the node and quarantine a damaged working partition before requesting this recovery by restarting it. Recovery does not assign a new partition owner.
Blob nodes scrub acknowledged replica placements and repair missing or corrupt payloads from verified peers. The master schedules replacements on ready blob nodes when a placement disappears and independently verifies replacement content. Each operation retains its original required-copy count. Repair runs every five seconds with bounded passes; monitor repair failures when no healthy source survives.
Replica stores require exclusive ownership of their directory. Startup rejects payload files without their SQLite reservations. If the replica index is lost or corrupt, restore the original index or stop the node and quarantine the entire replica directory before rebuilding its assigned copies from peers. Deleting only replicas.sqlite leaves unaccounted data and is not a supported rebuild procedure.
Protected indexed files are sent to an executor through its existing RPC service and verified before searchable publication. Executors receive indexed records and blob references, not blob payload storage responsibilities.
Retention and availability
Working blobs are protected by ingestion and indexed-file references. Abandoned uploads expire, and the working segment cleaner can reclaim blobs after references are released.
Monitor blob.durability.operations, blob.durability.pending, blob.durability.retained_bytes, and blob.durability.capacity_bytes with store=journal or store=replica. Capacity measurements are accounted reservations and include incomplete receptions. Master commit latency and failed replica verification are recorded separately.
Replica reclamation requires a master-committed terminal proof for the entire segment or logical file. Releasing one reference does not authorize payload deletion. Nodes persist retirement evidence and reclamation intent before removing bytes, then release their capacity reservation exactly once. A pinned reader, interrupted deletion, or restart leaves the intent for another maintenance pass. Aborted preparations receive separate operation tombstones. Retired identities cannot acquire new placements or be received again.
Fixed replay cuts support restoration but do not compact the journal. Mutation history, receipts, placement history, replica envelopes, and retirement evidence remain within explicit operation and byte limits. Capacity exhaustion rejects writes; provision and monitor metadata capacity until compact recovery snapshots and pruning are implemented.
HTTP testing
The API host exposes the following authenticated endpoints. It routes to the account's current blob owner, so the API host does not need a local blob-service role. Use a user JWT or personal access token. Uploads require current account inflow/admin permission and an inflow scope for the destination schema. Sessions belong to the creating user.
Use an account with a nonempty UUID. The legacy all-zero default account is not a valid blob-storage identity; create a separate account for blob testing.
| Method | Path | Request |
|---|---|---|
| PUT | /accounts/{accountId}/blobs/uploads/{uploadId} | JSON { "schema": "logs", "filename": "sample.bin", "length": 3 }. Generate a UUID for uploadId; it also becomes the blob ID. Length is optional. Retry with the same UUID and body. |
| GET | /accounts/{accountId}/blobs/uploads/{uploadId} | Returns committed length, maximumChunkBytes, state, and deadlines. |
| PATCH | /accounts/{accountId}/blobs/uploads/{uploadId}?offset=0 | Raw bytes, Content-Length, Idempotency-Key containing an operation UUID, and X-Blob-SHA256 containing the chunk's 64-character hexadecimal digest. |
| POST | /accounts/{accountId}/blobs/uploads/{uploadId}/complete | Idempotency-Key and X-Blob-SHA256 containing the whole-object digest. |
| DELETE | /accounts/{accountId}/blobs/uploads/{uploadId} | Idempotency-Key. Cancels an unattached upload; referenced blobs follow table retention. |
| GET / HEAD | /accounts/{accountId}/blobs/{blobId} | Download with an optional single byte Range, If-None-Match, or If-Range. |
Reuse the same operation UUID, offset, bytes, and checksum when retrying a chunk or completion after a lost response. Resume from the status endpoint's committed length. Reusing an identity with different content is rejected. Chunk bodies are streamed with bounded buffers; chunks above maximumChunkBytes return HTTP 413. Missing chunk Content-Length returns 411. Expected storage rejections return safe typed HTTP errors rather than remote exception stacks.
Downloads require a live reference in a readable schema and a query scope that permits that schema. Authorization runs before ETag/HEAD/range evaluation. Since current reference metadata identifies schemas rather than individual referencing columns, a schema with any explicitly unreadable column is conservatively excluded. Responses use attachment disposition, application/octet-stream, nosniff, and private no-store caching.
Completing an upload does not attach it to a table. Declare a BlobReference column and ingest the uploaded UUID through the normal JSON inflow API with a per-record ingestionId; see table declarations and blob references. Accepted blob-bearing records retain durable protections while publication runs asynchronously. Read their state at GET /inflow/{accountId}/{schema}/records/{ingestionId}.
Blob-aware compaction, compact journal recovery snapshots/pruning, and UI upload controls remain pending. Public binary readiness stays disabled until the lifecycle release gates pass.