ng-eventually/.tasks/doing/T01-align-the-published-surface.md

id, title, parent, depends_on, created, claimed_by, claimed_at, blocked

id	title	parent	depends_on	created	claimed_by	claimed_at	blocked
T01	Align the published surface on the NextGraph SDK	null		2026-09-16	null	null	false

Description

The polyfill's published surface has drifted from the SDK's: invented names, invented parameters, and symbols with no upstream counterpart. This task brings it back and installs what stops it recurring.

The principle to apply — .project/rules/nextgraph/surface-mirrors-the-sdk.md, and its semantic sibling no-divergence-from-nextgraph.md. Publish the SDK's names, parameters and return values; a difference must be duly justified beside the code, and the best justification is that it is NextGraph's own direction as announced by its developers.

The facts this rests on — .project/concepts/app-contract/knowledge_what-a-read-scope-is-upstream.md. In particular: a graphs entry is one document's graph; a store's graph holds only its containment triples, so "search my private store" returns empty silently; did:ng:i means the whole local dataset with no permission narrowing; scoping a read by store is not expressible at all.

Four pieces of work:

1. watchShape becomes useShapeQuery, takes the SDK's Scope ({ graphs?, subjects? }) instead of the three literals "public" | "protected" | "private", and reads through the SDK's shape query instead of enumerating documents. The ShapeQuery return shape (TanStack-like) and the capability filter both stay — that difference is the justified kind.
2. Wrap the ORM family the polyfill does not expose: getObjects and insertObject at identical signatures with the rights filter; arbitrate OrmSubscription, DiscreteOrmSubscription, ngSession; establish upstream's shape for deletion before specifying it. Today an application must import them straight from @ng-org/orm, and then receives every identity's data from the shared wallet with nothing signalling it — a rights hole, not a missing convenience.
3. Remove readUnion entirely once nothing internal calls it. Three single-document reads become an anchored SPARQL query. ng-helpers' contract cites it and must be revised.
4. The annotation mechanism: a mandatory reference on every function, which must resolve; plus a form for "target announced by the developers, not yet written", which the current format cannot express — so an author holding the strongest justification is pushed to miscite or to understate.

Boundary that must not be lost: the store axis survives for creation (docCreate takes a destination upstream) and dies for reading.

Success criteria: every published symbol either mirrors an SDK symbol in name, parameters and return, or carries a justified annotation; no symbol without an upstream counterpart survives unjustified; gates green (bun test, both typechecks, lint.py 0 ERROR, e2e); and an adversarial review over the whole published surface finds nothing unaddressed.

Progress

Result

Subtasks

• T01.a A provenance annotation for a target announced but not yet written

• T01.b What the ORM offers for writing and deleting, and what the three remaining symbols are (reconnaissance)

• T01.c watchShape becomes useShapeQuery and takes the SDK's Scope (deps: T01.a)

• T01.d useShapeQuery reads through the SDK's shape query (deps: T01.c)

• T01.e Publish the ORM family with the rights filter (deps: T01.b, T01.d)

• T01.f Remove readUnion, and the documents that cite it (deps: T01.e)

• T01.g Adversarial review of the whole published surface (deps: all)

• T01.i Clear two mechanical leftovers from the removal (deps: T01.h)

• T01.j A read that cannot reach its document must fail, not answer empty (deps: T01.i)

• T01.l pnpm everywhere, and two dependency ranges that are looser than they look (deps: T01.j)

• T01.m There must be ONE answer to whether a document is open in this session (deps: T01.j)

• T01.k A provenance citation must say which version it describes (deps: T01.l)

Learnings digest

• T01.a — the taxonomy gains announced-not-written: its ref points at the .md in THIS repo where the statement is on record, never a source file, plus announced-by=<who>@<date>. Upstream citations now resolve-check only when the sibling checkout is present; announced records always. Two latent defects fell out: the test's closed set was missing reserved-name, and the rule said "five kinds" over six rows. All 92 existing refs already resolved.

• T01.b — deletion is answered at level 3 and fully wired: DeepSignalSet.delete -> a remove patch -> graph_orm_update -> SparqlBuilder::remove_object, which removes the SHAPE's predicates, not the subject; no document deletion exists at any level. getObjects returns a DETACHED clone; insertObject throws without @graph. Proposals: publish OrmSubscription as-is (the read filter applies to signalObject), publish NEITHER DiscreteOrmSubscription (JSON-CRDT, no @graph, nothing for a per-document filter to read) NOR ngSession (hands out the SDK handle and the session). MEASURED: useShape is still broken live, and the bug leaf's mechanism was FALSE - the deployment is AHEAD of our pinned @ng-org/*, its orm_start_graph takes config BEFORE the callback, so the five-argument client lands its callback in the wrong slot. T01.d's blocker STANDS. Also VERIFIED: ORM writes bypass sparql_update entirely, so the emulated write guard never sees an ORM add or delete.

• T01.c - useShapeQuery takes the SDK's Scope, re-exported as a passthrough. Our three literals became StoreType, internal only: VERIFIED they are upstream's own store_type, so only the NAME was invented. A store is now named by its REFERENCE (privateStore() / protectedStore() / publicStore()), and createEntityDoc(store?) takes one, private when omitted, as doc_create does. listMyEntityDocs, resolveScopeGraph and resolveWriteGraph unpublished, still internal - T01.d removes the read path's last caller. VERIFIED: public and protected share one store id here, so the reference must be the index document. 1.0.0-dev.5.

• Package refresh (2026-09-21) - @ng-org/* moved to web alpha.14 / orm alpha.21 / shex-orm alpha.9 / alien-deepsignals alpha.13. The live ORM read now STARTS against the deployed broker: 6 passed, 0 failed, 6 declared, ready in 16ms, delete reached the document. bug_useshape-live-read-needs-a-newer-broker DELETED - the deployment was never behind, our packages were. T01.d's blocker LIFTS. Also surfaced: OrmSubscription is now RdfOrmSubscription (T01.e's proposal names the old one); Scope.graphs is REQUIRED and its [] means the EMPTY set while subjects: [] means NO filtering - the two axes read [] in opposite directions, which T01.d must not get wrong; six e2e files imported ng-e2e-helpers without the polyfill declaring it, fixed as workspace:*.

• T01.d - useShapeQuery reads through RdfOrmSubscription.getOrCreate, injected as configure({ ormSubscription }); absent, the read publishes isError naming it. Scope resolution, the ensureReposOpen barrier, readUnion and the @type filter are gone from that path; graphs/subjects/did:ng:i reach the engine verbatim. The observable, snapshot, state and capability filter stay ours. isError KEPT, wired to the one real failure (the query refusing to open). A start the ENGINE refuses is swallowed upstream, so it reads isPending for ever - recorded in code and contract, no timeout invented. userStoreDoc and liveStartWithConfig deleted. VERIFIED: listMyEntityDocs is NOT dead, its own suite and an e2e journey call it. The change silently broke examples/notebook/, repaired out of boundary and deliberately. 1.0.0-dev.6, major, 42/42 e2e.

• T01.e - getObjects and insertObject published WRAPPED at upstream's names, parameters and returns. Read is capability-filtered per @graph; the write takes assertMayReach + assertMayWrite BEFORE the call, and graph_orm_update now carries the same ownership guard, so the ORM rights hole is closed. NOT published, each with a checkable reason: RdfOrmSubscription (pooled with no identity in the key), DiscreteOrmSubscription, ngSession, getRaw, watch/effect, removeObject (in alpha.21 but absent from the pinned nextgraph-rs, so no citation resolves). 1.0.0-dev.7, MAJOR. ROOT CAUSE FOUND for the consumer's orm_start_graph is not a function: makeNg()'s passthrough did .bind(ng), and the SDK's ng is an async proxy whose get trap answers EVERY key including bind with a path-carrying callable - so the published ng returned a PROMISE for all 88 members. Now Reflect.apply, pinned by test/ng-proxy.test.ts. The e2e only proves it since initNg routes the ORM through the wrapper - before, polyfill-entry.ts handed initNg the raw SDK object, which is why the suite stayed green while the documented path was dead.

• T01.f - readUnion and UnionSubject unpublished, read-model.ts deleted, replaced by an internal readDocumentSubjects(doc) doing ONE anchored query. The blocker recorded on this task was MISDIAGNOSED: it was stated as "no shape read serves a public index", but an index is read BY ANCHOR, not by shape, and docs.sparqlQuery already calls fetchReadCap(anchor) - so the public read already worked and useShapeQuery never needed touching. The barrier written for it was reverted. VERIFIED UPSTREAM (engine/verifier/src/request_processor.rs:269, let repo = self.repos.get(repo_id).ok_or(NgError::RepoNotFound)?): an anchored query on a repo the verifier does not know REJECTS, crossing to JS as a rejected Promise carrying "RepoNotFound" (sdk/js/lib-wasm/src/lib.rs:403,606). The observable partition is "in self.repos" vs "not in" - "does not exist" and "not known to me" are INDISTINGUISHABLE, while "known but unsynced" collides with "genuinely empty". No read-capability check exists on that path at all, which confirms fetchReadCap is emulation machinery with no upstream counterpart. Consequence: readUnion's onUnreadable was an INVENTION, not a heritage, so shedding it is an alignment gain; the tolerance is RELOCATED to the callers, with a visible catch, and no tolerant batch helper replaced it. Two behaviour changes: an unreachable document is now refused out loud, and an application composing its own raw SPARQL sees the urn:ng-eventually: scaffolding that readUnion filtered - no rights move (VERIFIED: exposeReadCap is called only for scope === "public", account-registry.ts:1205), but a MODEL LEAK the application must unlearn at migration. 1.0.0-dev.8, major. The ng-helpers citation of readUnion lives in THEIR usage declaration, never in this repo's leaves - the parent's claim named the wrong document. Follow-ups left open: useShapeQuery opens nothing named in scope.graphs, ensureReposOpen is now dead, examples/notebook/tsconfig.json fails typecheck (pre-existing).

• T01.h - NO version moved: all four @ng-org/* were already at their latest published version (web alpha.14, orm alpha.21, shex-orm alpha.9, alien-deepsignals alpha.13). The refresh this task was written for had already happened inside T01.d. What the install DID surface: the tree was BUN-installed, not pnpm-installed - every @ng-org symlink pointed into node_modules/.bun/, which still held web alpha.13, shex-orm alpha.8 and alien-deepsignals alpha.11 BESIDE the current ones. Resolved versions were right, so the single-instance hazard was latent rather than active; reconciled by CI=true pnpm install --no-frozen-lockfile, lockfile unmoved. Gates: 424 pass / 0 fail, both typechecks exit 0, lint 0 ERROR, orm-live-read 5 passed / 0 failed, e2e 43 passed / 0 failed. THE HEADLINE, VERIFIED INDEPENDENTLY: the pinned nextgraph-rs checkout (213338f6) declares orm 0.1.2-alpha.18 while this repo compiles against 0.1.2-alpha.21 - THREE releases behind, so every ref= citation in the provenance mechanism describes a version no longer used. Second-order consequence to watch: a claim about alpha.21 REFUTED by reading the pinned tree is refuted against alpha.18, which is not the same statement. Three doctrine leaves corrected (knowledge_what-the-orm-writes-and-deletes-upstream, knowledge_what-a-read-scope-is-upstream, knowledge_what-each-suite-judges), each carrying facts that had become false. Also found: web/shex-orm/alien-deepsignals publish a bare 0.1.2 that semver ranks ABOVE the alphas but dates from 2025 - exact pins exclude it, but ng-e2e-helpers' peer range >=0.1.2-alpha.13 would admit it. Follow-ups: pnpm-workspace.yaml lists only packages/* while root workspaces also lists examples/*, so examples/notebook is outside pnpm entirely; the dead .bun store is still on disk and a future bun install would re-point symlinks back into it; playwright 1.61.1 -> 1.63.0.

• T01.i - ensureReposOpen deleted (dead, grep-verified); its 3 tests went with it and that is legitimate, they asserted the BATCH's own properties (parallelism, deduplication, empty input) which no longer have a subject - the 4th, exercising ensureRepoOpen singular, was kept and relocated. 424 -> 421 pass, the gap fully accounted. THE LEAD ON THE EXAMPLE WAS WRONG and the agent disproved it: the pnpm-workspace.yaml omission was real but NOT the cause of the TS2345; the actual cause was a NoteShape literal missing its satisfies ShapeType<BaseType> clause, so valType widened from "iri"/"string" to string. Proven by fixing only that, with the workspace untouched. The workspace gap was fixed anyway because item 3 needed it, and the duplication check came back clean - examples/notebook now resolves @ng-org/* to the same store paths as packages/polyfill. Bun store deleted (88M), and the RECURRENCE TRIGGER found: bun.lock was tracked in git while .gitignore only covered the older bun.lockb; now untracked, both ignored, and packageManager: pnpm@10.26.0 declared. Left alone, reported: docs/internal-contract.md's open-repo section was already stale (wrong line numbers, names a non-existent ensurePhysicalRepoOpen) and now also names the deleted ensureReposOpen; an orphaned @ng-org/web@0.1.2-alpha.13 sits in the pnpm store with zero referencing symlinks.

• T01.j - SPLIT after six independent adversarial rounds. DELIVERED, none of it faulted across those six reviews: the four call sites propagate instead of collapsing (docs.sparqlQuery, ensureRepoOpen, documentInboxAddress, and fetchReadCap, which stops turning a rejected download into false), fetchReadCaps deleted (dead), the knowledge_what-a-failed-read-answers-upstream leaf, the empty-is-success guards, e2e named-scope coverage run live by TWO independent reviewers, and the REACHABILITY half of the barrier - a holder of nothing gets isError with dataLen: 0, a public document opens from a bare reference. NOT delivered, and now written into the contract as a non-guarantee pointing at T01.m: the ARRIVAL half holds only within one session, and fails across configure() and when a broker rejects after the open timeout. 1.0.0-dev.9, 449 pass / 0 fail, e2e 46 passed / 46. THE ROOT CAUSE, which is T01.m's subject: "is this document open in this session" has THREE answers that can disagree - held and syncState in open-repo.ts, and fanOuts in subscribe.ts, the one nobody counted. Round 6 replaced one of two disagreeing records; there were three. TWO PROCESS LESSONS worth more than the code. First, a round counter is the wrong stopping rule: rounds 2-5 each fixed the reported corner and opened another, and what finally decided the split was a review producing the COMMON CAUSE rather than one more corner - stop when a review explains, not when a counter trips. Second, the implementer asserted three times that a change had simplified or shrunk something and all three were contradicted by measurement (534->563, 664->727, and a numstat of 34 insertions / 8 deletions for "a condition was removed"); two were relayed to the owner before being measured. The rule now held, and written into T01.m: a structural claim carries its wc -l or git show --numstat in the same sentence, or it is not written. Also found along the way, each fixed at its source: subscribe.ts already offered onSetupFailed for a refused subscribe and nothing had wired it; resetOpenedRepos abandoned in-flight opens without settling them, so a caller waited the full fallback and proceeded with sync state "unknown", which index-deposit.noRowsMeansAbsent accepts as a definitive absence - a latent failure-as-absence bug found by instrumenting a one-in-four flaky test instead of re-running it.