ng-eventually/.tasks/doing/T01-align-the-published-surface.md

id, title, parent, depends_on, created, claimed_by, claimed_at, blocked

id	title	parent	depends_on	created	claimed_by	claimed_at	blocked
T01	Align the published surface on the NextGraph SDK	null		2026-09-16	null	null	false

Description

The polyfill's published surface has drifted from the SDK's: invented names, invented parameters, and symbols with no upstream counterpart. This task brings it back and installs what stops it recurring.

The principle to apply — .project/rules/nextgraph/surface-mirrors-the-sdk.md, and its semantic sibling no-divergence-from-nextgraph.md. Publish the SDK's names, parameters and return values; a difference must be duly justified beside the code, and the best justification is that it is NextGraph's own direction as announced by its developers.

The facts this rests on — .project/concepts/app-contract/knowledge_what-a-read-scope-is-upstream.md. In particular: a graphs entry is one document's graph; a store's graph holds only its containment triples, so "search my private store" returns empty silently; did:ng:i means the whole local dataset with no permission narrowing; scoping a read by store is not expressible at all.

Four pieces of work:

1. watchShape becomes useShapeQuery, takes the SDK's Scope ({ graphs?, subjects? }) instead of the three literals "public" | "protected" | "private", and reads through the SDK's shape query instead of enumerating documents. The ShapeQuery return shape (TanStack-like) and the capability filter both stay — that difference is the justified kind.
2. Wrap the ORM family the polyfill does not expose: getObjects and insertObject at identical signatures with the rights filter; arbitrate OrmSubscription, DiscreteOrmSubscription, ngSession; establish upstream's shape for deletion before specifying it. Today an application must import them straight from @ng-org/orm, and then receives every identity's data from the shared wallet with nothing signalling it — a rights hole, not a missing convenience.
3. Remove readUnion entirely once nothing internal calls it. Three single-document reads become an anchored SPARQL query. ng-helpers' contract cites it and must be revised.
4. The annotation mechanism: a mandatory reference on every function, which must resolve; plus a form for "target announced by the developers, not yet written", which the current format cannot express — so an author holding the strongest justification is pushed to miscite or to understate.

Boundary that must not be lost: the store axis survives for creation (docCreate takes a destination upstream) and dies for reading.

Success criteria: every published symbol either mirrors an SDK symbol in name, parameters and return, or carries a justified annotation; no symbol without an upstream counterpart survives unjustified; gates green (bun test, both typechecks, lint.py 0 ERROR, e2e); and an adversarial review over the whole published surface finds nothing unaddressed.

Progress

Result

Subtasks

• T01.a A provenance annotation for a target announced but not yet written

• T01.b What the ORM offers for writing and deleting, and what the three remaining symbols are (reconnaissance)

• T01.c watchShape becomes useShapeQuery and takes the SDK's Scope (deps: T01.a)

• T01.d useShapeQuery reads through the SDK's shape query (deps: T01.c)

• T01.e Publish the ORM family with the rights filter (deps: T01.b, T01.d)

• T01.f Remove readUnion, and the documents that cite it (deps: T01.e)

• T01.g Adversarial review of the whole published surface (deps: all)

• T01.i Clear two mechanical leftovers from the removal (deps: T01.h)

• T01.j A read that cannot reach its document must fail, not answer empty (deps: T01.i)

• T01.l pnpm everywhere, and two dependency ranges that are looser than they look (deps: T01.j)

• T01.m There must be ONE answer to whether a document is open in this session (deps: T01.j)

• T01.n The polyfill's peer ranges exclude exactly the versions it needs (deps: T01.m)

• T01.k A provenance citation must say which version it describes (deps: T01.n)

Learnings digest

• T01.a — the taxonomy gains announced-not-written: its ref points at the .md in THIS repo where the statement is on record, never a source file, plus announced-by=<who>@<date>. Upstream citations now resolve-check only when the sibling checkout is present; announced records always. Two latent defects fell out: the test's closed set was missing reserved-name, and the rule said "five kinds" over six rows. All 92 existing refs already resolved.

• T01.b — deletion is answered at level 3 and fully wired: DeepSignalSet.delete -> a remove patch -> graph_orm_update -> SparqlBuilder::remove_object, which removes the SHAPE's predicates, not the subject; no document deletion exists at any level. getObjects returns a DETACHED clone; insertObject throws without @graph. Proposals: publish OrmSubscription as-is (the read filter applies to signalObject), publish NEITHER DiscreteOrmSubscription (JSON-CRDT, no @graph, nothing for a per-document filter to read) NOR ngSession (hands out the SDK handle and the session). MEASURED: useShape is still broken live, and the bug leaf's mechanism was FALSE - the deployment is AHEAD of our pinned @ng-org/*, its orm_start_graph takes config BEFORE the callback, so the five-argument client lands its callback in the wrong slot. T01.d's blocker STANDS. Also VERIFIED: ORM writes bypass sparql_update entirely, so the emulated write guard never sees an ORM add or delete.

• T01.c - useShapeQuery takes the SDK's Scope, re-exported as a passthrough. Our three literals became StoreType, internal only: VERIFIED they are upstream's own store_type, so only the NAME was invented. A store is now named by its REFERENCE (privateStore() / protectedStore() / publicStore()), and createEntityDoc(store?) takes one, private when omitted, as doc_create does. listMyEntityDocs, resolveScopeGraph and resolveWriteGraph unpublished, still internal - T01.d removes the read path's last caller. VERIFIED: public and protected share one store id here, so the reference must be the index document. 1.0.0-dev.5.

• Package refresh (2026-09-21) - @ng-org/* moved to web alpha.14 / orm alpha.21 / shex-orm alpha.9 / alien-deepsignals alpha.13. The live ORM read now STARTS against the deployed broker: 6 passed, 0 failed, 6 declared, ready in 16ms, delete reached the document. bug_useshape-live-read-needs-a-newer-broker DELETED - the deployment was never behind, our packages were. T01.d's blocker LIFTS. Also surfaced: OrmSubscription is now RdfOrmSubscription (T01.e's proposal names the old one); Scope.graphs is REQUIRED and its [] means the EMPTY set while subjects: [] means NO filtering - the two axes read [] in opposite directions, which T01.d must not get wrong; six e2e files imported ng-e2e-helpers without the polyfill declaring it, fixed as workspace:*.

• T01.d - useShapeQuery reads through RdfOrmSubscription.getOrCreate, injected as configure({ ormSubscription }); absent, the read publishes isError naming it. Scope resolution, the ensureReposOpen barrier, readUnion and the @type filter are gone from that path; graphs/subjects/did:ng:i reach the engine verbatim. The observable, snapshot, state and capability filter stay ours. isError KEPT, wired to the one real failure (the query refusing to open). A start the ENGINE refuses is swallowed upstream, so it reads isPending for ever - recorded in code and contract, no timeout invented. userStoreDoc and liveStartWithConfig deleted. VERIFIED: listMyEntityDocs is NOT dead, its own suite and an e2e journey call it. The change silently broke examples/notebook/, repaired out of boundary and deliberately. 1.0.0-dev.6, major, 42/42 e2e.

• T01.e - getObjects and insertObject published WRAPPED at upstream's names, parameters and returns. Read is capability-filtered per @graph; the write takes assertMayReach + assertMayWrite BEFORE the call, and graph_orm_update now carries the same ownership guard, so the ORM rights hole is closed. NOT published, each with a checkable reason: RdfOrmSubscription (pooled with no identity in the key), DiscreteOrmSubscription, ngSession, getRaw, watch/effect, removeObject (in alpha.21 but absent from the pinned nextgraph-rs, so no citation resolves). 1.0.0-dev.7, MAJOR. ROOT CAUSE FOUND for the consumer's orm_start_graph is not a function: makeNg()'s passthrough did .bind(ng), and the SDK's ng is an async proxy whose get trap answers EVERY key including bind with a path-carrying callable - so the published ng returned a PROMISE for all 88 members. Now Reflect.apply, pinned by test/ng-proxy.test.ts. The e2e only proves it since initNg routes the ORM through the wrapper - before, polyfill-entry.ts handed initNg the raw SDK object, which is why the suite stayed green while the documented path was dead.

• T01.f - readUnion and UnionSubject unpublished, read-model.ts deleted, replaced by an internal readDocumentSubjects(doc) doing ONE anchored query. The blocker recorded on this task was MISDIAGNOSED: it was stated as "no shape read serves a public index", but an index is read BY ANCHOR, not by shape, and docs.sparqlQuery already calls fetchReadCap(anchor) - so the public read already worked and useShapeQuery never needed touching. The barrier written for it was reverted. VERIFIED UPSTREAM (engine/verifier/src/request_processor.rs:269, let repo = self.repos.get(repo_id).ok_or(NgError::RepoNotFound)?): an anchored query on a repo the verifier does not know REJECTS, crossing to JS as a rejected Promise carrying "RepoNotFound" (sdk/js/lib-wasm/src/lib.rs:403,606). The observable partition is "in self.repos" vs "not in" - "does not exist" and "not known to me" are INDISTINGUISHABLE, while "known but unsynced" collides with "genuinely empty". No read-capability check exists on that path at all, which confirms fetchReadCap is emulation machinery with no upstream counterpart. Consequence: readUnion's onUnreadable was an INVENTION, not a heritage, so shedding it is an alignment gain; the tolerance is RELOCATED to the callers, with a visible catch, and no tolerant batch helper replaced it. Two behaviour changes: an unreachable document is now refused out loud, and an application composing its own raw SPARQL sees the urn:ng-eventually: scaffolding that readUnion filtered - no rights move (VERIFIED: exposeReadCap is called only for scope === "public", account-registry.ts:1205), but a MODEL LEAK the application must unlearn at migration. 1.0.0-dev.8, major. The ng-helpers citation of readUnion lives in THEIR usage declaration, never in this repo's leaves - the parent's claim named the wrong document. Follow-ups left open: useShapeQuery opens nothing named in scope.graphs, ensureReposOpen is now dead, examples/notebook/tsconfig.json fails typecheck (pre-existing).

• T01.h - NO version moved: all four @ng-org/* were already at their latest published version (web alpha.14, orm alpha.21, shex-orm alpha.9, alien-deepsignals alpha.13). The refresh this task was written for had already happened inside T01.d. What the install DID surface: the tree was BUN-installed, not pnpm-installed - every @ng-org symlink pointed into node_modules/.bun/, which still held web alpha.13, shex-orm alpha.8 and alien-deepsignals alpha.11 BESIDE the current ones. Resolved versions were right, so the single-instance hazard was latent rather than active; reconciled by CI=true pnpm install --no-frozen-lockfile, lockfile unmoved. Gates: 424 pass / 0 fail, both typechecks exit 0, lint 0 ERROR, orm-live-read 5 passed / 0 failed, e2e 43 passed / 0 failed. THE HEADLINE, VERIFIED INDEPENDENTLY: the pinned nextgraph-rs checkout (213338f6) declares orm 0.1.2-alpha.18 while this repo compiles against 0.1.2-alpha.21 - THREE releases behind, so every ref= citation in the provenance mechanism describes a version no longer used. Second-order consequence to watch: a claim about alpha.21 REFUTED by reading the pinned tree is refuted against alpha.18, which is not the same statement. Three doctrine leaves corrected (knowledge_what-the-orm-writes-and-deletes-upstream, knowledge_what-a-read-scope-is-upstream, knowledge_what-each-suite-judges), each carrying facts that had become false. Also found: web/shex-orm/alien-deepsignals publish a bare 0.1.2 that semver ranks ABOVE the alphas but dates from 2025 - exact pins exclude it, but ng-e2e-helpers' peer range >=0.1.2-alpha.13 would admit it. Follow-ups: pnpm-workspace.yaml lists only packages/* while root workspaces also lists examples/*, so examples/notebook is outside pnpm entirely; the dead .bun store is still on disk and a future bun install would re-point symlinks back into it; playwright 1.61.1 -> 1.63.0.

• T01.i - ensureReposOpen deleted (dead, grep-verified); its 3 tests went with it and that is legitimate, they asserted the BATCH's own properties (parallelism, deduplication, empty input) which no longer have a subject - the 4th, exercising ensureRepoOpen singular, was kept and relocated. 424 -> 421 pass, the gap fully accounted. THE LEAD ON THE EXAMPLE WAS WRONG and the agent disproved it: the pnpm-workspace.yaml omission was real but NOT the cause of the TS2345; the actual cause was a NoteShape literal missing its satisfies ShapeType<BaseType> clause, so valType widened from "iri"/"string" to string. Proven by fixing only that, with the workspace untouched. The workspace gap was fixed anyway because item 3 needed it, and the duplication check came back clean - examples/notebook now resolves @ng-org/* to the same store paths as packages/polyfill. Bun store deleted (88M), and the RECURRENCE TRIGGER found: bun.lock was tracked in git while .gitignore only covered the older bun.lockb; now untracked, both ignored, and packageManager: pnpm@10.26.0 declared. Left alone, reported: docs/internal-contract.md's open-repo section was already stale (wrong line numbers, names a non-existent ensurePhysicalRepoOpen) and now also names the deleted ensureReposOpen; an orphaned @ng-org/web@0.1.2-alpha.13 sits in the pnpm store with zero referencing symlinks.

• T01.j - SPLIT after six independent adversarial rounds. DELIVERED, none of it faulted across those six reviews: the four call sites propagate instead of collapsing (docs.sparqlQuery, ensureRepoOpen, documentInboxAddress, and fetchReadCap, which stops turning a rejected download into false), fetchReadCaps deleted (dead), the knowledge_what-a-failed-read-answers-upstream leaf, the empty-is-success guards, e2e named-scope coverage run live by TWO independent reviewers, and the REACHABILITY half of the barrier - a holder of nothing gets isError with dataLen: 0, a public document opens from a bare reference. NOT delivered, and now written into the contract as a non-guarantee pointing at T01.m: the ARRIVAL half holds only within one session, and fails across configure() and when a broker rejects after the open timeout. 1.0.0-dev.9, 449 pass / 0 fail, e2e 46 passed / 46. THE ROOT CAUSE, which is T01.m's subject: "is this document open in this session" has THREE answers that can disagree - held and syncState in open-repo.ts, and fanOuts in subscribe.ts, the one nobody counted. Round 6 replaced one of two disagreeing records; there were three. TWO PROCESS LESSONS worth more than the code. First, a round counter is the wrong stopping rule: rounds 2-5 each fixed the reported corner and opened another, and what finally decided the split was a review producing the COMMON CAUSE rather than one more corner - stop when a review explains, not when a counter trips. Second, the implementer asserted three times that a change had simplified or shrunk something and all three were contradicted by measurement (534->563, 664->727, and a numstat of 34 insertions / 8 deletions for "a condition was removed"); two were relayed to the owner before being measured. The rule now held, and written into T01.m: a structural claim carries its wc -l or git show --numstat in the same sentence, or it is not written. Also found along the way, each fixed at its source: subscribe.ts already offered onSetupFailed for a refused subscribe and nothing had wired it; resetOpenedRepos abandoned in-flight opens without settling them, so a caller waited the full fallback and proceeded with sync state "unknown", which index-deposit.noRowsMeansAbsent accepts as a definitive absence - a latent failure-as-absence bug found by instrumenting a one-in-four flaky test instead of re-running it.

• T01.l - ng-e2e-helpers' peer range tightened to >=0.1.2-alpha.13 <0.1.2, MEASURED BY RESOLUTION: before, 4 versions admitted with maxSatisfying = the bare 0.1.2 of 2025-10-19; after, 3, and pnpm resolves 0.1.2-alpha.15. The premise it was handed was half wrong and it said so: pnpm never actually RESOLVED to 0.1.2 because it prefers the latest dist-tag, which upstream happens to point at alpha.15 - so the trap was one dist-tag move from firing, and only resolution testing shows that either way. playwright 1.61.1 -> 1.63.0 (raises engines.node to >=20, drops fsevents), e2e 46 passed / 46 on the first run. Both typecheck scripts moved from bunx tsc to pnpm exec tsc, VERIFIED with ~/.bun/bin off PATH and command -v bun empty. MEASURED about the optional peer: a consumer of the packed polyfill receives NO @ng-org package under pnpm, so the old bun install semantics survived the tool change - but THIS checkout does get the ORM, through packages/polyfill/package.json:33's devDependency, so the runner's "install it in packages/polyfill" was a false lead and both texts now say the optional peer governs what a CONSUMER receives. pnpm link PERFORMED and it works: the polyfill resolves to the checkout and stays at ONE copy, so the single-instance hazard is untripped - but @ng-org/web and @ng-org/orm end up at TWO copies when the app also imports the SDK, pnpm warning by name, and file: gives one copy at the cost of repacking, which forfeits live editing. Left alone, reported: the polyfill's own four "*" peers admit ONLY the bare 0.1.2 of 16 published versions and EXCLUDE every alpha, because * matches no pre-release - inert while the polyfill is unpublished, certain to bite at first publish (now T01.n); examples/notebook has no typecheck script; bun build at orm-live-read.ts:128 and bun run in mint-wallet are still bun-bound.

• T01.m - THE COUNT WAS ELEVEN, not the three the split had named. Deleted from open-repo.ts: held, syncState, inFlight, settleInFlight, registryEpoch - the last two IMPLIED openness through a short-circuit and nobody had ever counted them, including six adversarial rounds of T01.j. Surviving as the one answer, in surface/subscribe.ts (NOT emulated-verifier/, which is what the task file wrongly said): fanOuts, establishing, realUnsub, lastState, attempt - and lastState turned out to have been a second barrier record all along, the correct one. THE INVARIANT, written at open-repo.ts:100-128 and on DocChannel: "is this document open in this session" is answered in ONE place, docChannel(nuri), and open-repo records only the bootstrap open it ASKED FOR. Teardown is now safe BY CONSTRUCTION with no cross-module notification added, because nothing outside records openness any more; resetOpenedRepos disowns rather than forgets, since forgetting yields the lax "unknown". CONTRACT: both non-guarantees DELETED and the arrival guarantee unqualified, 1.0.0-dev.10, guarantee and test in the same commit - and an INDEPENDENT review attacked that first and hardest and could not break it. 462 pass / 0 fail six times, e2e 46 passed / 46. My own spec was wrong twice and the agent said so instead of contorting to satisfy it: the module path, and failure B's acceptance test ("2 subscribes in-window"), which is UNACHIEVABLE because the fan-out correctly dedupes. Four adversarial passes found SIX real defects in the agent's own work, including two regressions and a test of its own that asserted the defect. THE LESSON WORTH KEEPING, from the last one: it had the right fix, could not make a test discriminate, and REVERTED it - correctly - but recorded "could not construct an ordering" as an OPEN QUESTION rather than a clean bill. That wording is what let a later pass finish it. The window is exactly ONE microtask hop, swept 0-6: at 0 the clear has not happened so delivery is legitimate, at 1 a forgotten barrier is delivered, at 2+ lastState is already null so nothing is queued. Its earlier fake put the rejection three hops behind the replay, so the replay always won - what it had wrong was the TEST, not the remedy. Also verified by the independent review, and worth knowing because the guarantee is right for a reason nobody had written down: "unknown" is unreachable where a barrier was expected NOT for the stated reason, but because fetchReadCap -> downloadReadCap -> ensurePhysicalRepoOpen populates the open record BEFORE mustNotAttempt is evaluated, so even the refused path leaves a trace. Follow-ups left open: a never-settling doc_subscribe is unrecoverable; public-store.attempted skips an open after configure; forgetEverything clears four maps never.

• T01.n - the four @ng-org peer ranges fixed, VERIFIED BY RESOLUTION against the registry with the real node-semver engine rather than by reading. WORSE THAN THE TASK STATED: @ng-org/orm's "*" admitted ZERO of its 22 published versions - that library publishes no bare 0.1.2 at all, so the range was UNSATISFIABLE, not merely wrong. The other three admitted exactly 1, the stale bare 0.1.2 of 2025, and excluded every alpha. After: web >=0.1.2-alpha.14 <0.1.2 (2 admitted, maxSatisfying alpha.15), orm >=0.1.2-alpha.21 <0.1.2 (2), shex-orm >=0.1.2-alpha.9 <0.1.2 (2), alien-deepsignals >=0.1.2-alpha.13 <0.1.2 (2). The floor was taken PER PACKAGE from its own devDependencies pin rather than copying the sibling package's alpha.13 that the task offered as an example - that floor predates its own devDep bump and is a stale artifact, not a rule. peerDependenciesMeta: optional confirmed correct for all four and VERIFIED rather than assumed: every @ng-org/* reference in packages/polyfill/src is import type only, zero runtime imports, the application injecting the real functions through bootstrap.ts. examples/notebook gained the typecheck script it lacked. Fresh install ends with ONE .pnpm store entry per package and every consuming project's realpath matching. 462 pass / 0 fail. It also caught an error of MINE: the task file I wrote carried a baseline of 449 while the measurement says 462 - written before T01.m added its tests and never refreshed. That is the second of my own specs to carry a wrong number, after T01.m's unachievable acceptance test. Left alone, reported: ng-e2e-helpers' own floor looks stale for the same reason (INFERRED, out of scope).