Sylvain Duchesne
71 lines
24 KiB
Markdown
71 lines
24 KiB
Markdown
---
|
|
id: T01
|
|
title: Align the published surface on the NextGraph SDK
|
|
parent: null
|
|
depends_on: []
|
|
created: 2026-09-16
|
|
claimed_by: null
|
|
claimed_at: null
|
|
blocked: false
|
|
---
|
|
|
|
## Description
|
|
|
|
The polyfill's published surface has drifted from the SDK's: invented names, invented parameters, and symbols with no upstream counterpart. This task brings it back and installs what stops it recurring.
|
|
|
|
**The principle to apply** — `.project/rules/nextgraph/surface-mirrors-the-sdk.md`, and its semantic sibling `no-divergence-from-nextgraph.md`. Publish the SDK's **names**, **parameters** and **return values**; a difference must be duly justified beside the code, and the best justification is that it is NextGraph's own direction as announced by its developers.
|
|
|
|
**The facts this rests on** — `.project/concepts/app-contract/knowledge_what-a-read-scope-is-upstream.md`. In particular: a `graphs` entry is one document's graph; a store's graph holds only its containment triples, so "search my private store" returns empty silently; `did:ng:i` means the whole local dataset with no permission narrowing; scoping a read by store is not expressible at all.
|
|
|
|
**Four pieces of work:**
|
|
|
|
1. `watchShape` becomes `useShapeQuery`, takes the SDK's `Scope` (`{ graphs?, subjects? }`) instead of the three literals `"public" | "protected" | "private"`, and reads through the SDK's shape query instead of enumerating documents. The `ShapeQuery` return shape (TanStack-like) and the capability filter both stay — that difference is the justified kind.
|
|
2. Wrap the ORM family the polyfill does not expose: `getObjects` and `insertObject` at identical signatures with the rights filter; arbitrate `OrmSubscription`, `DiscreteOrmSubscription`, `ngSession`; establish upstream's shape for deletion before specifying it. Today an application must import them straight from `@ng-org/orm`, and then receives every identity's data from the shared wallet with nothing signalling it — a rights hole, not a missing convenience.
|
|
3. Remove `readUnion` entirely once nothing internal calls it. Three single-document reads become an anchored SPARQL query. `ng-helpers`' contract cites it and must be revised.
|
|
4. The annotation mechanism: a mandatory reference on every function, which must **resolve**; plus a form for "target announced by the developers, not yet written", which the current format cannot express — so an author holding the strongest justification is pushed to miscite or to understate.
|
|
|
|
**Boundary that must not be lost:** the store axis survives for **creation** (`docCreate` takes a destination upstream) and dies for **reading**.
|
|
|
|
**Success criteria:** every published symbol either mirrors an SDK symbol in name, parameters and return, or carries a justified annotation; no symbol without an upstream counterpart survives unjustified; gates green (`bun test`, both typechecks, `lint.py` 0 ERROR, e2e); and an adversarial review over the whole published surface finds nothing unaddressed.
|
|
|
|
## Progress
|
|
|
|
## Result
|
|
|
|
## Subtasks
|
|
- T01.a A provenance annotation for a target announced but not yet written
|
|
- T01.b What the ORM offers for writing and deleting, and what the three remaining symbols are (reconnaissance)
|
|
- T01.c watchShape becomes useShapeQuery and takes the SDK's Scope (deps: T01.a)
|
|
- T01.d useShapeQuery reads through the SDK's shape query (deps: T01.c)
|
|
- T01.e Publish the ORM family with the rights filter (deps: T01.b, T01.d)
|
|
- T01.f Remove readUnion, and the documents that cite it (deps: T01.e)
|
|
- T01.g Adversarial review of the whole published surface (deps: all)
|
|
|
|
- T01.i Clear two mechanical leftovers from the removal (deps: T01.h)
|
|
- T01.j A read that cannot reach its document must fail, not answer empty (deps: T01.i)
|
|
- T01.l pnpm everywhere, and two dependency ranges that are looser than they look (deps: T01.j)
|
|
- T01.m There must be ONE answer to whether a document is open in this session (deps: T01.j)
|
|
- T01.n The polyfill's peer ranges exclude exactly the versions it needs (deps: T01.m)
|
|
- T01.k A provenance citation must say which version it describes (deps: T01.n)
|
|
|
|
- T01.o The read filter is bypassed by plain properties, and no fake shows it
|
|
- T01.p Four minor faults on the published surface (deps: T01.o)
|
|
|
|
## Learnings digest
|
|
|
|
- T01.a — the taxonomy gains `announced-not-written`: its `ref` points at the `.md` in THIS repo where the statement is on record, never a source file, plus `announced-by=<who>@<date>`. Upstream citations now resolve-check only when the sibling checkout is present; announced records always. Two latent defects fell out: the test's closed set was missing `reserved-name`, and the rule said "five kinds" over six rows. All 92 existing refs already resolved.
|
|
- T01.b — deletion is answered at level 3 and fully wired: `DeepSignalSet.delete` -> a remove patch -> `graph_orm_update` -> `SparqlBuilder::remove_object`, which removes the SHAPE's predicates, not the subject; no document deletion exists at any level. `getObjects` returns a DETACHED clone; `insertObject` throws without `@graph`. Proposals: publish `OrmSubscription` as-is (the read filter applies to `signalObject`), publish NEITHER `DiscreteOrmSubscription` (JSON-CRDT, no `@graph`, nothing for a per-document filter to read) NOR `ngSession` (hands out the SDK handle and the session). MEASURED: `useShape` is still broken live, and the bug leaf's mechanism was FALSE - the deployment is AHEAD of our pinned `@ng-org/*`, its `orm_start_graph` takes `config` BEFORE the callback, so the five-argument client lands its callback in the wrong slot. T01.d's blocker STANDS. Also VERIFIED: ORM writes bypass `sparql_update` entirely, so the emulated write guard never sees an ORM add or delete.
|
|
- T01.c - `useShapeQuery` takes the SDK's `Scope`, re-exported as a passthrough. Our three literals became `StoreType`, internal only: VERIFIED they are upstream's own `store_type`, so only the NAME was invented. A store is now named by its REFERENCE (`privateStore()` / `protectedStore()` / `publicStore()`), and `createEntityDoc(store?)` takes one, private when omitted, as `doc_create` does. `listMyEntityDocs`, `resolveScopeGraph` and `resolveWriteGraph` unpublished, still internal - T01.d removes the read path's last caller. VERIFIED: public and protected share one store id here, so the reference must be the index document. `1.0.0-dev.5`.
|
|
- Package refresh (2026-09-21) - `@ng-org/*` moved to web alpha.14 / orm alpha.21 / shex-orm alpha.9 / alien-deepsignals alpha.13. The live ORM read now STARTS against the deployed broker: `6 passed, 0 failed, 6 declared`, ready in 16ms, delete reached the document. `bug_useshape-live-read-needs-a-newer-broker` DELETED - the deployment was never behind, our packages were. T01.d's blocker LIFTS. Also surfaced: `OrmSubscription` is now `RdfOrmSubscription` (T01.e's proposal names the old one); `Scope.graphs` is REQUIRED and its `[]` means the EMPTY set while `subjects: []` means NO filtering - the two axes read `[]` in opposite directions, which T01.d must not get wrong; six e2e files imported `ng-e2e-helpers` without the polyfill declaring it, fixed as `workspace:*`.
|
|
- T01.d - `useShapeQuery` reads through `RdfOrmSubscription.getOrCreate`, injected as `configure({ ormSubscription })`; absent, the read publishes `isError` naming it. Scope resolution, the `ensureReposOpen` barrier, `readUnion` and the `@type` filter are gone from that path; `graphs`/`subjects`/`did:ng:i` reach the engine verbatim. The observable, snapshot, state and capability filter stay ours. `isError` KEPT, wired to the one real failure (the query refusing to open). A start the ENGINE refuses is swallowed upstream, so it reads `isPending` for ever - recorded in code and contract, no timeout invented. `userStoreDoc` and `liveStartWithConfig` deleted. VERIFIED: `listMyEntityDocs` is NOT dead, its own suite and an e2e journey call it. The change silently broke `examples/notebook/`, repaired out of boundary and deliberately. `1.0.0-dev.6`, major, 42/42 e2e.
|
|
- T01.e - `getObjects` and `insertObject` published WRAPPED at upstream's names, parameters and returns. Read is capability-filtered per `@graph`; the write takes `assertMayReach` + `assertMayWrite` BEFORE the call, and `graph_orm_update` now carries the same ownership guard, so the ORM rights hole is closed. NOT published, each with a checkable reason: `RdfOrmSubscription` (pooled with no identity in the key), `DiscreteOrmSubscription`, `ngSession`, `getRaw`, `watch`/`effect`, `removeObject` (in alpha.21 but absent from the pinned `nextgraph-rs`, so no citation resolves). `1.0.0-dev.7`, MAJOR. ROOT CAUSE FOUND for the consumer's `orm_start_graph is not a function`: `makeNg()`'s passthrough did `.bind(ng)`, and the SDK's `ng` is an async proxy whose `get` trap answers EVERY key including `bind` with a path-carrying callable - so the published `ng` returned a PROMISE for all 88 members. Now `Reflect.apply`, pinned by `test/ng-proxy.test.ts`. The e2e only proves it since `initNg` routes the ORM through the wrapper - before, `polyfill-entry.ts` handed `initNg` the raw SDK object, which is why the suite stayed green while the documented path was dead.
|
|
|
|
- T01.f - `readUnion` and `UnionSubject` unpublished, `read-model.ts` deleted, replaced by an internal `readDocumentSubjects(doc)` doing ONE anchored query. The blocker recorded on this task was MISDIAGNOSED: it was stated as "no shape read serves a public index", but an index is read BY ANCHOR, not by shape, and `docs.sparqlQuery` already calls `fetchReadCap(anchor)` - so the public read already worked and `useShapeQuery` never needed touching. The barrier written for it was reverted. VERIFIED UPSTREAM (`engine/verifier/src/request_processor.rs:269`, `let repo = self.repos.get(repo_id).ok_or(NgError::RepoNotFound)?`): an anchored query on a repo the verifier does not know REJECTS, crossing to JS as a rejected Promise carrying `"RepoNotFound"` (`sdk/js/lib-wasm/src/lib.rs:403,606`). The observable partition is "in `self.repos`" vs "not in" - "does not exist" and "not known to me" are INDISTINGUISHABLE, while "known but unsynced" collides with "genuinely empty". No read-capability check exists on that path at all, which confirms `fetchReadCap` is emulation machinery with no upstream counterpart. Consequence: `readUnion`'s `onUnreadable` was an INVENTION, not a heritage, so shedding it is an alignment gain; the tolerance is RELOCATED to the callers, with a visible `catch`, and no tolerant batch helper replaced it. Two behaviour changes: an unreachable document is now refused out loud, and an application composing its own raw SPARQL sees the `urn:ng-eventually:` scaffolding that `readUnion` filtered - no rights move (VERIFIED: `exposeReadCap` is called only for `scope === "public"`, `account-registry.ts:1205`), but a MODEL LEAK the application must unlearn at migration. `1.0.0-dev.8`, major. The `ng-helpers` citation of `readUnion` lives in THEIR usage declaration, never in this repo's leaves - the parent's claim named the wrong document. Follow-ups left open: `useShapeQuery` opens nothing named in `scope.graphs`, `ensureReposOpen` is now dead, `examples/notebook/tsconfig.json` fails typecheck (pre-existing).
|
|
- T01.h - NO version moved: all four `@ng-org/*` were already at their latest published version (web alpha.14, orm alpha.21, shex-orm alpha.9, alien-deepsignals alpha.13). The refresh this task was written for had already happened inside T01.d. What the install DID surface: the tree was BUN-installed, not pnpm-installed - every `@ng-org` symlink pointed into `node_modules/.bun/`, which still held web alpha.13, shex-orm alpha.8 and alien-deepsignals alpha.11 BESIDE the current ones. Resolved versions were right, so the single-instance hazard was latent rather than active; reconciled by `CI=true pnpm install --no-frozen-lockfile`, lockfile unmoved. Gates: `424 pass / 0 fail`, both typechecks exit 0, lint 0 ERROR, `orm-live-read` `5 passed / 0 failed`, e2e `43 passed / 0 failed`. THE HEADLINE, VERIFIED INDEPENDENTLY: the pinned `nextgraph-rs` checkout (`213338f6`) declares orm `0.1.2-alpha.18` while this repo compiles against `0.1.2-alpha.21` - THREE releases behind, so every `ref=` citation in the provenance mechanism describes a version no longer used. Second-order consequence to watch: a claim about alpha.21 REFUTED by reading the pinned tree is refuted against alpha.18, which is not the same statement. Three doctrine leaves corrected (`knowledge_what-the-orm-writes-and-deletes-upstream`, `knowledge_what-a-read-scope-is-upstream`, `knowledge_what-each-suite-judges`), each carrying facts that had become false. Also found: web/shex-orm/alien-deepsignals publish a bare `0.1.2` that semver ranks ABOVE the alphas but dates from 2025 - exact pins exclude it, but `ng-e2e-helpers`' peer range `>=0.1.2-alpha.13` would admit it. Follow-ups: `pnpm-workspace.yaml` lists only `packages/*` while root `workspaces` also lists `examples/*`, so `examples/notebook` is outside pnpm entirely; the dead `.bun` store is still on disk and a future `bun install` would re-point symlinks back into it; `playwright` 1.61.1 -> 1.63.0.
|
|
- T01.i - `ensureReposOpen` deleted (dead, grep-verified); its 3 tests went with it and that is legitimate, they asserted the BATCH's own properties (parallelism, deduplication, empty input) which no longer have a subject - the 4th, exercising `ensureRepoOpen` singular, was kept and relocated. `424 -> 421 pass`, the gap fully accounted. THE LEAD ON THE EXAMPLE WAS WRONG and the agent disproved it: the `pnpm-workspace.yaml` omission was real but NOT the cause of the `TS2345`; the actual cause was a `NoteShape` literal missing its `satisfies ShapeType<BaseType>` clause, so `valType` widened from `"iri"`/`"string"` to `string`. Proven by fixing only that, with the workspace untouched. The workspace gap was fixed anyway because item 3 needed it, and the duplication check came back clean - `examples/notebook` now resolves `@ng-org/*` to the same store paths as `packages/polyfill`. Bun store deleted (88M), and the RECURRENCE TRIGGER found: `bun.lock` was tracked in git while `.gitignore` only covered the older `bun.lockb`; now untracked, both ignored, and `packageManager: pnpm@10.26.0` declared. Left alone, reported: `docs/internal-contract.md`'s open-repo section was already stale (wrong line numbers, names a non-existent `ensurePhysicalRepoOpen`) and now also names the deleted `ensureReposOpen`; an orphaned `@ng-org/web@0.1.2-alpha.13` sits in the pnpm store with zero referencing symlinks.
|
|
- T01.j - SPLIT after six independent adversarial rounds. DELIVERED, none of it faulted across those six reviews: the four call sites propagate instead of collapsing (`docs.sparqlQuery`, `ensureRepoOpen`, `documentInboxAddress`, and `fetchReadCap`, which stops turning a rejected download into `false`), `fetchReadCaps` deleted (dead), the `knowledge_what-a-failed-read-answers-upstream` leaf, the empty-is-success guards, e2e named-scope coverage run live by TWO independent reviewers, and the REACHABILITY half of the barrier - a holder of nothing gets `isError` with `dataLen: 0`, a public document opens from a bare reference. NOT delivered, and now written into the contract as a non-guarantee pointing at T01.m: the ARRIVAL half holds only within one session, and fails across `configure()` and when a broker rejects after the open timeout. `1.0.0-dev.9`, `449 pass / 0 fail`, e2e `46 passed / 46`. THE ROOT CAUSE, which is T01.m's subject: "is this document open in this session" has THREE answers that can disagree - `held` and `syncState` in `open-repo.ts`, and `fanOuts` in `subscribe.ts`, the one nobody counted. Round 6 replaced one of two disagreeing records; there were three. TWO PROCESS LESSONS worth more than the code. First, a round counter is the wrong stopping rule: rounds 2-5 each fixed the reported corner and opened another, and what finally decided the split was a review producing the COMMON CAUSE rather than one more corner - stop when a review explains, not when a counter trips. Second, the implementer asserted three times that a change had simplified or shrunk something and all three were contradicted by measurement (534->563, 664->727, and a numstat of 34 insertions / 8 deletions for "a condition was removed"); two were relayed to the owner before being measured. The rule now held, and written into T01.m: a structural claim carries its `wc -l` or `git show --numstat` in the same sentence, or it is not written. Also found along the way, each fixed at its source: `subscribe.ts` already offered `onSetupFailed` for a refused subscribe and nothing had wired it; `resetOpenedRepos` abandoned in-flight opens without settling them, so a caller waited the full fallback and proceeded with sync state `"unknown"`, which `index-deposit.noRowsMeansAbsent` accepts as a definitive absence - a latent failure-as-absence bug found by instrumenting a one-in-four flaky test instead of re-running it.
|
|
- T01.l - `ng-e2e-helpers`' peer range tightened to `>=0.1.2-alpha.13 <0.1.2`, MEASURED BY RESOLUTION: before, 4 versions admitted with `maxSatisfying` = the bare `0.1.2` of 2025-10-19; after, 3, and pnpm resolves `0.1.2-alpha.15`. The premise it was handed was half wrong and it said so: pnpm never actually RESOLVED to `0.1.2` because it prefers the `latest` dist-tag, which upstream happens to point at alpha.15 - so the trap was one dist-tag move from firing, and only resolution testing shows that either way. `playwright` 1.61.1 -> 1.63.0 (raises `engines.node` to `>=20`, drops `fsevents`), e2e `46 passed / 46` on the first run. Both `typecheck` scripts moved from `bunx tsc` to `pnpm exec tsc`, VERIFIED with `~/.bun/bin` off PATH and `command -v bun` empty. MEASURED about the optional peer: a consumer of the packed polyfill receives NO `@ng-org` package under pnpm, so the old `bun install` semantics survived the tool change - but THIS checkout does get the ORM, through `packages/polyfill/package.json:33`'s devDependency, so the runner's "install it in packages/polyfill" was a false lead and both texts now say the optional peer governs what a CONSUMER receives. `pnpm link` PERFORMED and it works: the polyfill resolves to the checkout and stays at ONE copy, so the single-instance hazard is untripped - but `@ng-org/web` and `@ng-org/orm` end up at TWO copies when the app also imports the SDK, pnpm warning by name, and `file:` gives one copy at the cost of repacking, which forfeits live editing. Left alone, reported: the polyfill's own four `"*"` peers admit ONLY the bare `0.1.2` of 16 published versions and EXCLUDE every alpha, because `*` matches no pre-release - inert while the polyfill is unpublished, certain to bite at first publish (now T01.n); `examples/notebook` has no `typecheck` script; `bun build` at `orm-live-read.ts:128` and `bun run` in `mint-wallet` are still bun-bound.
|
|
- T01.m - THE COUNT WAS ELEVEN, not the three the split had named. Deleted from `open-repo.ts`: `held`, `syncState`, `inFlight`, `settleInFlight`, `registryEpoch` - the last two IMPLIED openness through a short-circuit and nobody had ever counted them, including six adversarial rounds of T01.j. Surviving as the one answer, in `surface/subscribe.ts` (NOT `emulated-verifier/`, which is what the task file wrongly said): `fanOuts`, `establishing`, `realUnsub`, `lastState`, `attempt` - and `lastState` turned out to have been a second barrier record all along, the correct one. THE INVARIANT, written at `open-repo.ts:100-128` and on `DocChannel`: "is this document open in this session" is answered in ONE place, `docChannel(nuri)`, and `open-repo` records only the bootstrap open it ASKED FOR. Teardown is now safe BY CONSTRUCTION with no cross-module notification added, because nothing outside records openness any more; `resetOpenedRepos` disowns rather than forgets, since forgetting yields the lax `"unknown"`. CONTRACT: both non-guarantees DELETED and the arrival guarantee unqualified, `1.0.0-dev.10`, guarantee and test in the same commit - and an INDEPENDENT review attacked that first and hardest and could not break it. `462 pass / 0 fail` six times, e2e `46 passed / 46`. My own spec was wrong twice and the agent said so instead of contorting to satisfy it: the module path, and failure B's acceptance test ("2 subscribes in-window"), which is UNACHIEVABLE because the fan-out correctly dedupes. Four adversarial passes found SIX real defects in the agent's own work, including two regressions and a test of its own that asserted the defect. THE LESSON WORTH KEEPING, from the last one: it had the right fix, could not make a test discriminate, and REVERTED it - correctly - but recorded "could not construct an ordering" as an OPEN QUESTION rather than a clean bill. That wording is what let a later pass finish it. The window is exactly ONE microtask hop, swept 0-6: at 0 the clear has not happened so delivery is legitimate, at 1 a forgotten barrier is delivered, at 2+ `lastState` is already null so nothing is queued. Its earlier fake put the rejection three hops behind the replay, so the replay always won - what it had wrong was the TEST, not the remedy. Also verified by the independent review, and worth knowing because the guarantee is right for a reason nobody had written down: `"unknown"` is unreachable where a barrier was expected NOT for the stated reason, but because `fetchReadCap` -> `downloadReadCap` -> `ensurePhysicalRepoOpen` populates the open record BEFORE `mustNotAttempt` is evaluated, so even the refused path leaves a trace. Follow-ups left open: a never-settling `doc_subscribe` is unrecoverable; `public-store.attempted` skips an open after `configure`; `forgetEverything` clears four maps never.
|
|
- T01.n - the four `@ng-org` peer ranges fixed, VERIFIED BY RESOLUTION against the registry with the real node-semver engine rather than by reading. WORSE THAN THE TASK STATED: `@ng-org/orm`'s `"*"` admitted ZERO of its 22 published versions - that library publishes no bare `0.1.2` at all, so the range was UNSATISFIABLE, not merely wrong. The other three admitted exactly 1, the stale bare `0.1.2` of 2025, and excluded every alpha. After: web `>=0.1.2-alpha.14 <0.1.2` (2 admitted, maxSatisfying alpha.15), orm `>=0.1.2-alpha.21 <0.1.2` (2), shex-orm `>=0.1.2-alpha.9 <0.1.2` (2), alien-deepsignals `>=0.1.2-alpha.13 <0.1.2` (2). The floor was taken PER PACKAGE from its own devDependencies pin rather than copying the sibling package's `alpha.13` that the task offered as an example - that floor predates its own devDep bump and is a stale artifact, not a rule. `peerDependenciesMeta: optional` confirmed correct for all four and VERIFIED rather than assumed: every `@ng-org/*` reference in `packages/polyfill/src` is `import type` only, zero runtime imports, the application injecting the real functions through `bootstrap.ts`. `examples/notebook` gained the `typecheck` script it lacked. Fresh install ends with ONE `.pnpm` store entry per package and every consuming project's realpath matching. `462 pass / 0 fail`. It also caught an error of MINE: the task file I wrote carried a baseline of 449 while the measurement says 462 - written before T01.m added its tests and never refreshed. That is the second of my own specs to carry a wrong number, after T01.m's unachievable acceptance test. Left alone, reported: `ng-e2e-helpers`' own floor looks stale for the same reason (INFERRED, out of scope).
|
|
- T01.g - the closing review REFUTED the parent's success criterion, which is what it was for. 8 findings over 53 published symbols (not the 25 the orchestrator's naive grep counted - it measured export STATEMENTS), 41 clean. THE BLOCKING ONE IS A RIGHTS LEAK, and it is this batch's whole failure mode applied to access control: `read-filter.ts` returned non-function members RAW on the written ground that "a plain property carries no items" - true of the CITED tree (orm alpha.18, a bare `DeepSignalSet`), FALSE of the INSTALLED alpha.21, whose `useShape` returns a `UseShapeResult` carrying every identity's items on the plain properties `data`, `promise` and `subscription`. A stranger read the owner's data through all three and nothing threw. It became T01.o. Three majors: the contract instructed an injection that does not resolve (`@ng-org/orm`'s `.` maps to `core.d.ts`, which has no `useShape`); four present-tense claims that the SDK returns a bare `DeepSignalSet` are false of alpha.21, TWO of them load-bearing provenance grounds - the `readUnion` failure mode again, invisible to `provenance.test.ts` by construction; and the only live gate for the published `useShape` injects a HAND-WRITTEN substitute, so no fake in the repository returned the shipped shape, which is exactly why every gate was green. Four minors became T01.p. THE LESSON FOR THE ORCHESTRATOR, not the code: ten adversarial passes and every re-run gate of this batch were blind to this, because the only artifact exercising that path was a fake that had never been checked against what ships. A green suite is worth what its fakes are faithful to, and nobody had asked.
|