ng-eventually/docs/incidents/2026-07-14-write-loss-on-disconnect.md

Write loss on socket death (SerializationError)

Post-mortem — 2026-07-14 · Status: OPEN (not addressed).

An entity written just before a period of inactivity can be silently lost: it is absent on reconnection. (Whether the write never durably reached the broker, or reached it and is not read back on a cold reconnection, is not settled — see Epistemic caveat below. The wording here deliberately states only the observed symptom.) The account / identity survives (no fork). Observed in real conditions (Festipod, Firefox) during a pause after login/creation.

Symptom

1. The user logs in, the app creates an entity (a Festipod event).
2. A period of inactivity follows (idle, tab in the background…).
3. The broker socket dies spontaneously with SOCKET IS CLOSED Some(Left(SerializationError)).
4. On reconnection, the created entity has disappeared; the app reads back its own scope empty.

Evidence (VERIFIED — live Firefox logs, verbatim)

… REPLAY TOPIC NOT FOUND <topic> IN OVERLAY <overlay>
… NEED REPLAY true
… SENDING EVENTS FROM OUTBOX RETURNED: Err(TopicNotFound)
[user1][polyfill] resolveAccount(user1) → 1 record          ← the account SURVIVES (no fork)
[user1][polyfill] readScopeIndex(…) → 0 entities            ← but the scope is EMPTY
… set reçu: 0 objets Event (public)
… SOCKET IS CLOSED Some(Left(SerializationError)) [51, 3, 223, …]

Interpretation (plausible mechanism, not settled): the write was pushed into the local outbox, but the socket died before it was durably flushed into the broker topic; on reconnection, the outbox replay fails (Err(TopicNotFound)) because the topic was never created on the broker side → the event is abandoned. The account, for its part, had already been durably resolved (resolveAccount → 1 record): it is neither lost nor forked.

Epistemic caveat. The evidence establishes the symptom (loss + Err(TopicNotFound) + the scope read returning 0 — logged above as readScopeIndex, since renamed readUserStore). The exact mechanism is not settled between (i) loss at write time (the write never durably reaches the broker) and (ii) cold-rehydration failure (the write is on the broker but a fresh session does not reopen its own scope). The Err(TopicNotFound) on the outbox replay leans toward (i) in this Firefox case. See the @data repro below, which exhibits a neighboring symptom but does not settle (i) vs (ii).

Causal chain (TRACED — reading of the NextGraph core, to be re-verified)

• The SerializationError closes the socket. The core emits the disconnection in two places, and they are not the same file: engine/net/src/broker.rs:1074 sends LocalBrokerMessage::Disconnected, which the SDK turns into disconnections_sender.send(...) at sdk/rust/src/local_broker.rs:648. Navigate by symbol — the line numbers are volatile, and the earlier note in this file put the send in broker.rs, which it never was.
• This disconnection is pushed to subscribers via disconnections_subscribe(cb) (PUSH stream).
• NextGraph reconnection is an unimplemented // TODO (≈ broker.rs:1051-1076): nothing re-establishes the socket nor re-flushes the outbox.
• user_connect returns a snapshot { server_id, server_ip, error, since } at call time — not a stream, unusable for detecting a later drop.
• No write-durability confirmation API: a caller cannot await the guarantee that a write has reached the broker.

What the SDK exposes but does not consume

disconnections_subscribe does fire on this failure — but neither the polyfill (@ng-eventually/sdk) nor the consumer app subscribes to it. The signal exists, nobody listens to it; on the app side, no mechanism retries or warns the user.

Scope & not reproduced

• Observed on Firefox only to date. A manual test on another browser did not trigger the SerializationError nor its consequences.
• @data reproduction (Chromium, real broker) — 2026-07-14, decisive. The existing @data reconnection test (reconnexion-meme-identite) was a false green: it read A's repos back from the persistent profile's local IndexedDB, never from the broker. A genuinely cold reader (non-persistent freshBrowser context, the same wallet/account A, no local state — seeded from the wallet captured before the event) reads 0 events from A (BARRIER timed-out (8000ms), CONNECTION ESTABLISHED). A different signature from the Firefox case (no socket death; the OUTBOX empty is the reader's, trivially empty) and it does not settle (i) vs (ii) — an empty barrier is compatible with both. Established on the other hand: @data has never verified the broker durability of A's own reads, and cold rehydration from the broker fails. Repro: src/modules/event/features/reconnexion-froide-sans-local.feature (Festipod).
• To settle (i) vs (ii): independently verify that A's write reaches the broker — e.g. a warm reader / a second identity reads the event's public doc (the two-identity isolation scenario). If it sees it → the write is durable → the cold reader's 0 is a (ii) (rehydration). Otherwise → (i).

Fix leads (not arbitrated)

1. Core — fix the SerializationError and implement the reconnection TODO (re-establish the socket + re-flush the outbox).
2. SDK / polyfill — consume disconnections_subscribe → reconnection + outbox re-flush as a mitigation, independently of the core.
3. Durability API — expose a confirmation that a write has reached the broker, so that the caller can await it.

Links

• docs/nextgraph-current-state.md — current state of the core (disconnection / reconnection to be cross-referenced here).
• Product impact + consumer-side caveat: Festipod concept data-layer → caveat_write-durability-across-disconnect.