ng-eventually/.tasks/doing/T01.j-a-failure-must-not-read-as-an-absence.md

id, title, parent, depends_on, created, claimed_by, claimed_at, blocked

id	title	parent	depends_on	created	claimed_by	claimed_at	blocked
T01.j	A read that cannot reach its document must fail, not answer empty	T01	T01.i	2026-09-21	T01.j-worker	2026-09-21T14:45:49+02:00	false

Description

A read whose document is unreachable currently answers an EMPTY SUCCESS. It must answer a failure. This is the "failure disguised as an absence" family, which this repo has closed roughly eight times and still carries two open bug_ leaves for. Study it properly before changing anything: the facts below are established, the DESIGN is not.

The established facts - do NOT re-derive them, and do NOT read nextgraph-rs for them

Upstream (VERIFIED). An anchored SPARQL query on a repo the verifier does not know REJECTS. The deciding line is engine/verifier/src/request_processor.rs:269: let repo = self.repos.get(repo_id).ok_or(NgError::RepoNotFound)?; inside resolve_target_for_sparql. It crosses into JS as a REJECTED PROMISE carrying the string "RepoNotFound" (sdk/js/lib-wasm/src/lib.rs:403 nodejs, :606 browser). The observable partition upstream is "in self.repos" vs "not in self.repos": "does not exist" and "exists but this session does not know it" are INDISTINGUISHABLE, while "known but unsynced" collides with "genuinely empty". There is NO read-capability check anywhere on that path - resolve_target_for_sparql reads only nuri.target.

Our chain, three layers (VERIFIED by reading this working copy).

• fetchReadCap (packages/polyfill/src/emulated-verifier/public-store.ts:190) returns a boolean collapsing three unrelated situations: the emulation is not in force (:196), the network could not be asked (:223), the document exposes no cap (:225). Its own comment at :220 states the collapse out loud. downloadReadCap, directly below, DELIBERATELY distinguishes resolving from rejecting and documents why - so the distinction is produced and then destroyed one line later. And NO caller reads the boolean: ensureRepoOpen:208, docs.sparqlQuery:155, branch-registers.ts:316, fetchReadCaps:277 all call it as a bare statement.
• The read then proceeds on a document that was never opened.
• useShapeQuery answers isSuccess holding zero objects.

What the target says the answer should be. useShapeQuery's return type is modelled on TanStack Query's useQuery, which distinguishes purely by whether the query function REJECTS: a rejection gives isError with the error; a resolution gives isSuccess with whatever data, empty included. That maps exactly onto upstream's RepoNotFound. So an unreachable document must surface as isError, and isSuccess with zero objects must mean "it is there and it holds nothing".

The design is yours to establish, and the enumeration comes FIRST

Do NOT start from a preferred shape. .project/rules/nextgraph/no-divergence-from-nextgraph.md and polyfill-must-match-the-target both apply, and so does the discipline of listing the cases before concluding - this repo has a recorded incident of deciding from one convenient case and having to be dragged back row by row.

List the callers and their states before choosing anything. For each of the four call sites above: what is it doing, what does it already hold, and what should it do when the cap cannot be obtained? Include the INERT case - before the first cap exists the emulation deliberately does not enforce, and reading works anyway; whatever you build must stay inert there or it will break every early path.

The constraint that shapes the answer. fetchReadCap has NO upstream counterpart - upstream never consults a capability on the read path. So there is no SDK signature to mirror, and the cascade goes to the ENGINE'S MODEL: upstream's observable is that a read REJECTS or it does not. That is an argument about where the distinction should become visible, not a licence to invent a rich vocabulary of outcomes that the target does not have. Weigh it; do not treat this paragraph as the decision.

Re-examine the reverted barrier on its own merits. useShapeQuery opens nothing named in scope.graphs - the barrier was removed by T01.d. A fix was written during T01.f and REVERTED, because it had been justified as unblocking readUnion's removal, which was FALSE (an index is read by anchor, not by shape). The revert killed a bad JUSTIFICATION, not a bad change. The defect is real and unowned: an application that names an unopened document gets an empty success. Decide it fresh, on the enumeration.

Check the two open bug leaves (bug_a-lost-inbox-watch-is-never-retried, bug_an-empty-inbox-list-can-mean-a-failure). They are the same family. State whether this work closes either. Do NOT claim it does without a test that proves it - a bug leaf deleted on a plausible argument is worse than one left open.

Blast radius - this is a behaviour change applications see

Reads that silently answered empty will start failing. That is the POINT, and it is also how this breaks callers that were quietly relying on the silence. So: the published contract at .project/concepts/app-contract/polyfill-surface/contract_polyfill-surface.md must state the new behaviour, with a version bump per its own ## Change policy (a call that narrows is a MAJOR). Follow the existing pre-release scheme, do not invent one. Expect internal callers and tests to surface real breakage - each one is a case to understand, never a reason to soften the change.

Record the durable fact in .project/concepts/app-contract/ as a knowledge_ leaf: what the target answers for an unreachable, an unsynced and an empty document, and which of those a caller can tell apart. That fact is worth keeping whatever the fix turns out to be. Do NOT open a bug_ leaf for something this task repairs.

One incidental repair, since you are in this territory anyway

docs/internal-contract.md's open-repo.ts section is stale: wrong line numbers, it names a non-existent ensurePhysicalRepoOpen export, and since T01.i it also names the deleted ensureReposOpen. Correct what that section says about the functions you touch. Do NOT undertake a general sweep of that document - the drift is broad and pre-existing, and fixing all of it is not this task.

Verification

The e2e must run. Unit tests cannot judge this: the whole subject is what happens when the network does not answer. Run the polyfill e2e suite and report its counts verbatim. If the broker is unreachable, report the suite as UNRUN - never substitute a typecheck for the measurement.

Hard gate: bun test -> 0 fail; bunx tsc --noEmit -p packages/polyfill/tsconfig.json -> exit 0; bunx tsc --noEmit -p packages/polyfill/e2e/tsconfig.json -> exit 0; python3 ~/.claude/skills/concept/lint.py -> 0 ERROR. Anti-cheat: no any, no @ts-ignore, no as unknown as, no deleted or skipped test, no weakened assertion. Specifically forbidden here: catching the new failure somewhere upstream to keep an old test green - if a test breaks, it is telling you something.

An adversarial review is part of this task, not optional. When the work is done, it goes to a fresh reviewer mandated to REFUTE it, on at least these angles: a path where the failure is still swallowed; a path where the change now fails something that legitimately has nothing (an empty document must still be a SUCCESS); and the inert case. A finding counts only with a concrete failure scenario. Re-run the adversary on whatever you fix in response.

Progress

The enumeration, written before any code

Every row was read in this working copy. fetchReadCap reaches the network only when caps.isEnforcing() AND caps.capFor(doc) === undefined - held or inert, it answers without asking, so most rows below are already unaffected by whatever the network does.

Site 1 - ensureRepoOpen(nuri, holderKey?), open-repo.ts:209. Doing: bringing a repo into the session before an anchored read (the cold-start heal), asking the public store first so a bare reference to a public document suffices. Holds: the nuri. Asks only when holderKey === undefined. Today, when the cap cannot be obtained: fetchReadCap answers false exactly as if the document exposed none, mustNotAttempt is then true, and the function returns SILENTLY. Its eight live callers (readDocumentSubjects, readUserStore, restoreOwnCaps, readInboxCapPairs, readLinks, inbox.readSynced, inbox.readSyncedForDocument, inbox.processInbox) all read straight afterwards, so the failure does surface - as assertMayReach's "the connected user does not hold this document's cap", which is the WRONG reason and calls for the wrong action (give up, rather than retry). Should: propagate. INERT: returns before any round-trip, mustNotAttempt false, the open proceeds - unchanged. Note the six callers reading their own store/inbox hold that cap already, so they never ask at all.

Site 2 - docs.sparqlQuery, docs.ts:155 (published). Doing: the anchored read itself; the ask is what makes a bare reference to a public document readable. Today: false, then assertMayReach throws the same wrong reason. A failure, but mislabelled. Should: reject, with the true reason. INERT: no ask, mayReach true, the read proceeds - unchanged.

Site 3 - documentInboxAddress(doc), branch-registers.ts:317. Doing: reading a document's Header branch for its inbox address - the question a third party arrives with holding nothing but the reference. Today: false -> mustNotAttempt -> returns undefined, which means "this document has no inbox". inbox.postToDocument renders that as "its owner never opened one"; inbox.readForDocument / readSyncedForDocument render it as zero deposits. This is the pure failure-as-absence, on a published call, in both directions. The same function carries a SECOND one: the try/catch around the address query returns undefined for a broker that did not answer - the comment beside it justifies the catch by a refusal that mustNotAttempt has already excluded one line above, so what it actually swallows is breakdowns. Both repaired. Should: propagate; undefined keeps meaning "no inbox", which is an answer. INERT: unchanged.

Site 4 - fetchReadCaps(docs), public-store.ts:277. Doing: the batch form, Promise.all over the singular with a tolerance of its own. Holds: nothing. Zero callers - grep-verified across src, test and e2e, and it is not exported from src/index.ts. Should: deleted, on T01.i's own precedent for ensureReposOpen. Leaving it would leave an unowned batch policy over a call that now rejects.

Site 5, the one that is not a fetchReadCap call site - useShapeQuery (published). It asks nothing and opens nothing since T01.d. Doing: one shape in one SDK Scope, read through RdfOrmSubscription. Holds: scope.graphs, which NAMES documents. Today, for a named document whose cap is not held: the ORM answers, filterReadable drops every item, and the snapshot is isSuccess with data: [] - the empty success this task is named after. For a named document whose cap IS held but which this session has never synced: the ORM answers nothing and the snapshot is again isSuccess with data: [], for a document that is not empty. Should: before the query opens, each NAMED document in graphs is asked of the public store, refused out loud if its cap is not held, and crossed over the sync barrier; any of those failing publishes isError.

The cases that must NOT change, and why each is a success rather than a failure:

• graphs: "did:ng:i" or "" (the SDK's normalizeScope maps "" to did:ng:i): the whole local dataset. The application named no document, so there is nothing to guard and the capability filter is the right and only answer - isSuccess with data: [] means "you hold nothing of this shape". Untouched.
• graphs: [] or undefined: no objects, by the SDK's own definition. isSuccess with data: []. Untouched.
• a graphs entry that is not a did:ng: reference: it names no document this library can hold a cap for, so there is no cap question; left to the engine.
• a named document, held and synced, holding nothing of the shape: the barrier passes, the ORM answers empty, isSuccess with data: []. This is the line that must not move.
• a held document exposing no read cap: fetchReadCap answers false and MEMOISES it. An absence was read; it is an answer.
• a held document with no inbox: documentInboxAddress answers undefined, depositsForDocument answers [].
• the INERT case throughout: before the first cap exists, isEnforcing() is false, fetchReadCap returns without a round-trip, mayReach answers true for everything, and nothing new refuses. Every row above is unchanged there.

Partial scopes. graphs: [docA, docB] with only docB held refuses the WHOLE query rather than answering docB's objects. A short answer is the same lie as an empty one, and this repo has already ruled twice against skip-and-carry-on: readUnion's per-document tolerance was removed as an invention in 1.0.0-dev.8, and an ORM write refuses the whole batch when one document is not writable.

The design, and what was rejected

Chosen: fetchReadCap stops converting a failed download into false. downloadReadCap right below it already distinguishes resolving from rejecting and documents why; the collapse was one catch destroying that distinction one line later. Removing the catch leaves boolean honestly meaning "a cap was obtained", and gives all three live call sites the one thing each of them wanted. Plus the useShapeQuery barrier above, which is where the published empty success actually comes from.

Rejected, with the deciding reason for each:

• A three-valued result (held / absent / unknown), each caller deciding. Rejected on the enumeration: all three live callers want the same thing - fail - so the vocabulary would buy nothing, and fetchReadCap has no upstream counterpart, so a richer outcome set than the engine's ("a read rejects or it does not") would be invented vocabulary an application learns and unlearns.
• Fixing only fetchReadCap and leaving useShapeQuery alone. Rejected: useShapeQuery never calls fetchReadCap, so this would leave the published reactive read answering isSuccess with data: [] for a document it cannot read - the exact headline defect.
• Making the capability filter refuse instead of drop. Rejected: under did:ng:i dropping is exactly right (the broker delivers only what your keys open), and by the time the filter runs the engine has already answered. The question belongs at the ASK, not at the result.
• Refusing only the unreachable graphs and reading the rest. Rejected: see partial scopes above.
• Adding no open barrier to useShapeQuery, only the refusal. Rejected: an unsynced named document then still reads isSuccess with data: [], which is the same lie for a document that is not empty. Every other read path in this library crosses that barrier; this one stopped because T01.d removed the scope TRANSLATION that produced the document set, and the caller now names documents directly, so the barrier has a subject again.

Result

Status: done. 1.0.0-dev.9, MAJOR. Three commits: 75313f7 (the behaviour change), b0eec7b (the contract, the knowledge leaf and the narrow internal-contract repair), 534d8f9 (two guard tests).

What each call site does now when the cap cannot be obtained

• ensureRepoOpen (open-repo.ts:209) - propagates. The open has not happened, and the read that follows must not proceed as though it had. It used to decline silently and let the anchored read blame the reader for holding no cap.
• docs.sparqlQuery (docs.ts:155) - rejects with the true reason instead of assertMayReach's "you do not hold this document's cap", which is a different event calling for a different action.
• documentInboxAddress (branch-registers.ts:317) - propagates, and so does its own address read, whose catch was swallowing breakdowns into undefined. undefined now means only "no inbox".
• fetchReadCaps (public-store.ts:277) - DELETED. Zero callers, grep-verified, not in src/index.ts; an unowned per-document tolerance over a call that now rejects.
• useShapeQuery - crosses the barrier on every document scope.graphs NAMES before opening the query: public-store fetch, refusal out loud, sync barrier. Any of them failing publishes isError.

The design, and what was rejected

fetchReadCap stops converting a rejected download into false. downloadReadCap already distinguishes the two and documents why; one catch destroyed it a line later. boolean now honestly means "a cap was obtained". Rejected: a three-valued result (all three live callers want the same thing - fail - and the engine has no such vocabulary); fixing only fetchReadCap (useShapeQuery never calls it, so the headline defect would have survived); making the capability filter refuse instead of drop (under did:ng:i dropping is exactly right, and by then the engine has already answered - the question belongs at the ASK); partial answers for a multi-document scope (a short list is the same lie as an empty one); the refusal without the open (an unsynced named document would still read isSuccess with data: [], for a document that is not empty).

How "genuinely empty" stayed a success

Every row of the enumeration's "must not change" list is covered by a test that passes both with and without the change: use-shape-query.test.ts (m) a named, reachable document holding nothing of the shape is isSuccess with data: []; (h) graphs: []; (p) graphs: ""; (j) the whole dataset filtered down to the holder's own; (o) the INERT case; document-inbox-surfaces-failure.test.ts third test, a document nobody opened an inbox on answering [] on both reads and refusing the deposit with "has no inbox"; public-store.test.ts, a document exposing no cap still answering false and still asked only once.

The four tests that DO discriminate were verified non-vacuous by temporarily removing the change: (l) and (n) on the barrier, and both inbox failure tests. (q) caught a real leak when the generation check was removed.

What broke, and what it was really asserting

Four tests in public-store.test.ts, all of them expect(await fetchReadCap(unreachable)).toBe(false). What they were asserting was the MEMO's arithmetic - "a failure is not remembered, an absence is" - which is why every asksAbout count in that file is unchanged. The value they asserted alongside it was the collapse itself. They now assert a rejection, which is strictly stronger, plus one new test that the rejection names the document and keeps the broker's own words on cause.

Nothing else broke. The reason is worth recording: the three live fetchReadCap callers that read a STORE or an INBOX already hold that cap (fileOwnStructure / holdOwnCap file it at connection), so fetchReadCap answers from the registry and never reaches the network. The blast radius is much narrower than four call sites suggest - it is a third party naming a document it does not hold.

Gate

• bun test -> 431 pass / 0 fail, Ran 431 tests across 40 files (baseline 421 pass / 0 fail, measured before any change; +10 new tests, no test deleted or skipped)
• bunx tsc --noEmit -p packages/polyfill/tsconfig.json -> exit 0
• bunx tsc --noEmit -p packages/polyfill/e2e/tsconfig.json -> exit 0
• python3 ~/.claude/skills/concept/lint.py -> 0 ERROR (warnings pre-existing; the new leaf carries none)
• polyfill e2e, run at HEAD: Polyfill e2e summary: 43 passed, 0 failed, 43 total - batch 3.3 min, slowest cold sync 27s. Same count as T01.h's baseline. NOTE: the broker login is flaky - two of five runs died with [e2e broker login] gave up after 45.0s - the wallet-list screen never changed, at a different step each time, before any library code runs. A clean 43/43 was obtained twice, once at 75313f7 and once at HEAD.

The contract

1.0.0-dev.9, MAJOR, per the leaf's own ## Change policy ("an existing call narrows: ... it reports a state you did not have to handle before"). packages/polyfill/package.json bumped. Two NON-guarantees corrected because the change makes them false: "Neither shape read fetches a public store's key for you" (now useShape only, plus did:ng:i), and "a shape query the engine refuses never finishes loading" (the commonest way in is closed for a NAMED scope). Two out-of-leaf documents carried the same now-false claim and were corrected in one sentence each: docs/api-contract.md:262 and docs/readcap-and-nuri-model.md:367. emulated-verifier/read-filter.ts's header said both shape reads ask nothing - corrected, with the note that refusing at the ASK and filtering the RESULT remain two different questions.

A rights movement, stated as the grid requires. useShapeQuery now fetches a public store's cap for a document named in graphs, which files that cap in the connected holder's ring. That is a GAIN on that call. It grants nothing new overall: it is exactly the reachability docs.sparqlQuery anchored on the same document already gave, and exposeReadCap is called only for scope === "public" (VERIFIED in T01.f). No document becomes readable that was not already readable by naming it. Said in ## Non-guarantees.

knowledge_what-a-failed-read-answers-upstream records the durable fact: two observable outcomes only, the two collisions ("does not exist" vs "not known to me"; "unsynced" vs "genuinely empty"), no capability check on the read path, and the three rules that follow - including that the unsynced/empty collision is closed by the sync barrier and never by an error.

docs/internal-contract.md section 6 only: line numbers, the ensurePhysicalRepoOpen export this module has never had (it is shared-wallet/physical.ts:134), the deleted ensureReposOpen, the missing openRepoUnguarded, and ensureRepoOpen's second parameter. A dated note says the rest of that document's drift was left alone deliberately.

The two bug leaves: NEITHER is closed

• bug_a-lost-inbox-watch-is-never-retried - not closed. It asks for a recovery signal that does not depend on the identity performing a capability-bearing act. Nothing here adds one. This work arguably makes it more visible (a failed open reports a truer reason), which is not the same as retrying.
• bug_an-empty-inbox-list-can-mean-a-failure - not closed. Its mechanism is resolveAccount yielding nothing RATHER THAN raising, inside enumerateMyInboxes, which then reports a list that is complete and empty. That path was not touched: readInboxCapPairs already rethrows what it catches, and record?.docPrivate returning undefined still answers [] with no shortfall. No test proves otherwise, so the leaf stays.

Surprises

• VERIFIED: fetchReadCaps was dead. Zero callers anywhere in src, test or e2e, and absent from src/index.ts. Its last caller went with readUnion in 1.0.0-dev.8.
• VERIFIED: useShapeQuery's empty success does not come from fetchReadCap at all. That call site does not exist on this path - T01.d removed it with the scope translation. It comes from the capability filter emptying a named document's objects, and from a named-but-unsynced document answering nothing. Tests (l) and (n) fail without the barrier; the task's three-layer chain describes the ANCHORED path, which is a different (and already loud) one.
• VERIFIED: the blast radius is narrow. Six of the eight ensureRepoOpen callers hold the cap they are about to use, so they never ask the network. That is why only four tests broke.
• VERIFIED: g === "" in namedDocuments is unreachable - isNuri("") is false. Kept as documentation of the SDK's own alias, said so in the comment and the commit. Test (p) is therefore a behaviour guard, not a discriminating test, and is labelled as one.
• INFERRED: the barrier's cost. A named scope now spends one public-store fetch and one sync barrier per document per session, serially. Memoised, so a re-subscribe is a map hit; the worst case is a cold scope naming many documents, each bounded by the 8s fallback. Stated in the contract; no measurement was taken.

Follow-ups, reported and not fixed

• packages/polyfill/src/shared-wallet/account-registry.ts:78 imports documentInboxAddress and never uses it - a dead import. tsconfig.base.json sets no noUnusedLocals, so nothing flags it.
• Every e2e run logs, at connection: connect failed: docs.sparqlQuery: refused - the connected user does not hold this document's cap ... at readStoreCaps <- restoreOwnCaps <- restoreAndDrain. That is assertMayReach's own pre-existing message (this work would have produced a different one), so restoreOwnCaps is being refused one of the connected user's OWN store documents at connect, and the failure is logged rather than raised.
• The e2e broker login fails intermittently at 45s on the wallet-list screen, before any library code runs - 2 of 5 runs in this session.
• docs/internal-contract.md:7 still names surface/watch-shape.ts, surface/read-model.ts and a storeRegistry namespace listing three removed symbols. Out of this task's boundary.
• knowledge_what-a-failed-read-answers-upstream cites 213338f6, the checkout T01.h found to be three ORM releases behind. The facts it cites are engine-side rather than ORM-side, but T01.k owns reconciling what a citation's version means.