Reconnexion/ng-eventually
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(client): read via per-doc ANCHORED queries; document virtual vs physical wallet

Browse Source 
The anchorless union query (`GRAPH ?g`) scanned EVERY named graph in the local
store (the whole shared physical wallet) → O(wallet size), slow/timeouts on a
bloated wallet. Rewrite `readUnion` to run ONE ANCHORED `sparql_query` per by-need
doc (in parallel, per-doc tolerant): an anchored query is restricted to that
repo's graph, so it is O(1) per doc, INDEPENDENT of physical-wallet size. Keep the
ReadCap defense-in-depth gate.

docs/simulation.md: new "Physical wallet vs virtual wallet" section — the physical
shared wallet is a substrate that accumulates and must NEVER be enumerated/scanned;
each user's VIRTUAL wallet (the account's scope index in the shim) is the bounded
thing you enumerate ("list my documents"), then read those docs per-doc anchored.
read-model.md / nextgraph-current-state.md updated to the per-doc anchored rule.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Sylvain Duchesne
2026-07-05 22:50:14 +02:00
parent 6a3501e700
commit ac2b026955
5 changed files with 210 additions and 140 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/client/src/read-model.ts
+92 -85
View File
@@ -1,40 +1,48 @@
/**
* read-model — the LISTING primitive of the polyfill: open/sync a set of
* documents, then run ONE anchorless union `sparql_query` over the LOCAL UNION
* and return the triples grouped per subject. This is the mechanism documented
* in docs/read-model.md, verified against the real broker (T03.k probe): a
* `GRAPH ?g { ?s ?p ?o }` body with NO anchor spans every graph currently opened
* in the session store.
* read-model — the LISTING primitive of the polyfill: read a BOUNDED, by-need set
* of documents, each with its OWN anchored `sparql_query`, and return the triples
* grouped per subject. This is the mechanism documented in docs/read-model.md.
*
* ── WHY per-doc ANCHORED, never an anchorless union-scan ───────────────────
* An ANCHORED `sparql_query(sid, "SELECT ?s ?p ?o WHERE { ?s ?p ?o }", base, doc)`
* is restricted to the anchor repo's graph: `resolve_target_for_sparql(Repo)` →
* `Some(repo_graph_name)`, which becomes the query's DEFAULT graph. A body with NO
* `GRAPH` wrapper reads only that default graph → ONLY that doc's triples, O(1) per
* doc, INDEPENDENT of how many other graphs the local store holds.
*
* An ANCHORLESS query (`anchor` undefined → `UserSite` → `set_default_graph_as_union`)
* spans EVERY named graph currently in the session store. On a SHARED / bloated
* wallet that accumulates across runs, that is O(wallet size) → the observed ~90s
* timeouts. So the read path must NEVER union-scan all graphs: it reads exactly the
* bounded by-need set, one anchored query per doc.
*
* NB (verified, docs/read-model.md § probe step 4): an explicit `GRAPH ?g { … }`
* body iterates the NAMED graphs regardless of the default graph, so an anchor does
* NOT bound such a body. The per-doc read therefore uses a DEFAULT-GRAPH body (no
* `GRAPH` wrapper) so the anchor's one-repo restriction actually applies.
*
* ── WHY not the reactive ORM fan-out ──────────────────────────────────────
* `useShape({ graphs: […manyDocs] })` drives `orm_start_graph` over a fan-out of
* per-entity graphs; a freshly-created / not-yet-synced doc in that fan-out makes
* `RepoNotFound` abort the whole subscription → the readyPromise never resolves →
* the ~75s hang (docs/nextgraph-current-state.md § The ORM fan-out hang). Listing
* MUST instead be a one-shot union `sparql_query`. There is no reactive union
* query, so reactivity is assembled by RE-QUERYING on a change signal.
*
* ── The two steps ─────────────────────────────────────────────────────────
* 1. OPEN/SYNC the docs. A repo is only in the union once opened into the
* session's oxigraph store. Opening is done PER-DOC via an anchored probe
* query (`resolveTargetForSparql(anchor)` opens that one repo) — per-doc, so a
* single unreachable doc fails in isolation instead of aborting a fan-out.
* 2. UNION QUERY. One anchorless `SELECT ?g ?s ?p ?o WHERE { GRAPH ?g { ?s ?p ?o } }`
* over the local union; group rows by subject.
* MUST instead be a set of one-shot anchored `sparql_query`s. There is no reactive
* union query, so reactivity is assembled by RE-QUERYING on a change signal.
*
* ── GENERIC by construction ───────────────────────────────────────────────
* Zero application domain here: the consumer passes the doc NURIs to open (from
* Zero application domain here: the consumer passes the doc NURIs to read (from
* the discovery index for public events, or its own scope docs for my-entities)
* and interprets the returned per-subject property bags. All NextGraph I/O routes
* through the T01.a `docs` primitives (the REAL injected `ng`), so this module
* imports no `@ng-org` package.
*
* At migration the union query is unchanged (native SPARQL union over the wallet);
* only the "open/sync" step swaps to opening real per-user store repos by cap.
* At the real multi-store migration the per-doc anchored read is unchanged (native
* SPARQL, anchored to one repo); only bringing a repo into the session (open by cap)
* changes — the anchored query already resolves a same-session repo directly.
*/
import { docCreate, sparqlUpdate, sparqlQuery } from "./docs";
import { getStoreRegistryDeps, getCaps, getCurrentUser } from "./polyfill";
import { getCaps, getCurrentUser, getStoreRegistryDeps } from "./polyfill";
import { assertNuri } from "./sparql";
import type { Nuri } from "./types";
@@ -44,11 +52,11 @@ import type { Nuri } from "./types";
void docCreate;
void sparqlUpdate;
/** One subject read from the union, with its properties (predicate → values). */
/** One subject read from a doc, with its properties (predicate → values). */
export interface UnionSubject {
/** The subject IRI (`?s`). */
/** The subject IRI (`?s`) — in the polyfill, the doc's own NURI. */
subject: string;
/** The graph the subject was read from (`?g` — the repo graph name). */
/** The graph (doc NURI) the subject was read from. */
graph: string;
/** predicate IRI → the list of object values (literals or IRIs) for it. */
props: Record<string, string[]>;
@@ -72,93 +80,92 @@ async function sessionId(): Promise<string> {
}
/**
* Touch ONE doc with a cheap anchored `ASK`, tolerant per-doc.
* Read ONE doc with an ANCHORED default-graph query, tolerant per-doc.
*
* VERIFIED (T03.k, see docs/nextgraph-current-state.md § *A repo is only queryable
* once OPENED/synced*): the current JS SDK exposes **no primitive that SYNCS an
* unknown repo**. An anchored `sparql_query` resolves via `resolve_target_for_sparql`
* → `self.repos.get(id).ok_or(RepoNotFound)` — it never PULLS an absent repo (and
* neither do `doc_subscribe` nor `orm_start_graph`; the real loader
* `load_repo_from_read_cap` is `pub(crate)`, unexposed). So this `ASK` cannot sync:
* on a repo already in `self.repos` (every doc `doc_create`d this session — which,
* in the mono-wallet polyfill, is ALL of them) it is a no-op that confirms presence;
* on a genuinely-absent repo it throws `RepoNotFound` HERE, in isolation, and the
* doc is skipped — never aborting the listing of the others (the ORM fan-out's
* failure mode). Returns whether the touch succeeded.
* The anchor (`doc` NURI) restricts the query to that repo's graph as the DEFAULT
* graph (`resolve_target_for_sparql(Repo)` → `Some(repo_graph_name)`); a body with
* NO `GRAPH` wrapper reads exactly that default graph → ONLY this doc's triples.
* This is O(1) in the doc's own size and INDEPENDENT of the rest of the (possibly
* bloated / shared) session store — it never iterates other graphs.
*
* All same-session repos (every doc `doc_create`d this session — in the mono-wallet
* polyfill, ALL of them) are already in `self.repos`, so the anchored query resolves
* the repo directly with no separate open. A genuinely-absent repo throws
* `RepoNotFound` HERE, in isolation, and the doc is skipped — never aborting the
* others (unlike the ORM fan-out). Returns the doc's rows, or `[]` on failure.
*
* At the real multi-store migration this becomes a real sync: opening a per-user
* store repo by cap is a native broker fetch (`verifier.rs:1423` `OpenRepo` TODO).
*/
async function openDoc(sid: string, doc: Nuri): Promise<boolean> {
async function readDoc(
sid: string,
doc: Nuri,
): Promise<Array<Record<string, { value: string } | undefined>>> {
try {
const nuri = assertNuri(doc);
await sparqlQuery(sid, "ASK { ?s ?p ?o }", undefined, nuri);
return true;
// Anchored to `nuri` → default graph = this repo. NO `GRAPH ?g` wrapper, so
// the anchor's one-repo restriction applies (an explicit `GRAPH ?g` body would
// iterate all named graphs regardless of the anchor — see docs § probe step 4).
const res = await sparqlQuery(
sid,
"SELECT ?s ?p ?o WHERE { ?s ?p ?o }",
undefined,
nuri,
);
return bindings(res);
} catch (error) {
console.error("[read-model] open failed for", doc, error);
return false;
console.error("[read-model] read failed for", doc, error);
return [];
}
}
/**
* OPEN/SYNC a set of docs, then read the LOCAL UNION with ONE anchorless
* `sparql_query`, grouped per subject. `docs` are the NURIs to bring into the
* union (the consumer resolves them by need — index for public, own scope docs
* for mine). Docs that fail to open are skipped (see {@link openDoc}).
* Read a BOUNDED, by-need set of docs — each with its OWN anchored query — and
* return the triples grouped per subject. `docs` are the NURIs to read (the
* consumer resolves them by need — index for public, own scope docs for mine).
* Docs that fail are skipped (see {@link readDoc}); a failing doc never aborts the
* batch.
*
* The union query is anchorless (`anchor` undefined → LOCAL UNION) with an
* explicit `GRAPH ?g { ?s ?p ?o }` body, so it spans every opened graph and
* attributes each row to its source graph.
* NEVER an anchorless union-scan over all graphs (which is O(wallet size) and wrong
* on a shared / bloated wallet). Each doc is read with an anchored default-graph
* query, O(1) per doc, independent of wallet size — a non-empty wallet no longer
* matters. Reads run in parallel via `Promise.all`.
*/
export async function readUnion(docs: Nuri[]): Promise<UnionSubject[]> {
const sid = await sessionId();
const unique = [...new Set(docs.filter(Boolean))];
// 1. OPEN/SYNC — per-doc, tolerant (a bad doc never aborts the whole list).
await Promise.all(unique.map((d) => openDoc(sid, d)));
if (unique.length === 0) return [];
// 2. UNION QUERY — ONE anchorless query over the local union. The union spans
// ALL opened graphs (including the shim/inbox internals), so constrain it to
// the requested subjects with a VALUES block. In the polyfill each entity's
// subject IRI IS its own document NURI (writeEntity uses the doc NURI as the
// subject), so the requested doc NURIs ARE the subjects to read — precise and
// independent of the repo_graph_name overlay suffix on ?g.
const values = unique.map((d) => `<${assertNuri(d)}>`).join(" ");
const query =
`SELECT ?g ?s ?p ?o WHERE { GRAPH ?g { ?s ?p ?o . VALUES ?s { ${values} } } }`;
let rows: Array<Record<string, { value: string } | undefined>> = [];
try {
rows = bindings(await sparqlQuery(sid, query, undefined, undefined));
} catch (error) {
console.error("[read-model] union query failed:", error);
return [];
}
// One anchored query per doc, in parallel, tolerant (a bad doc yields []).
const perDoc = await Promise.all(
unique.map(async (d) => ({ doc: assertNuri(d), rows: await readDoc(sid, d) })),
);
// Cap gate (defence-in-depth). The union spans every OPENED graph, so a subject
// whose DOCUMENT is under a read policy the current user may not satisfy must be
// dropped — isolation holds BY CONSTRUCTION (the app only resolves docs it is
// entitled to) AND BY FILTER here. Generic: the lib owns the cap registry; a doc
// under no policy (`!governsRead`) flows through unchanged. In this polyfill each
// subject IRI IS its own document NURI, so the cap key is the subject.
// Cap gate (defence-in-depth). A doc whose read policy the current user may not
// satisfy is dropped. Isolation holds BY CONSTRUCTION (the app only resolves docs
// it is entitled to) AND BY FILTER here. Generic: the lib owns the cap registry;
// a doc under no policy (`!governsRead`) flows through unchanged. In this polyfill
// each subject IRI IS its own document NURI, so the cap key is the doc NURI.
const caps = getCaps();
const user = getCurrentUser();
const denied = (doc: string): boolean => caps.governsRead(doc) && !caps.canRead(doc, user);
const bySubject = new Map<string, UnionSubject>();
for (const row of rows) {
const s = row.s?.value;
const p = row.p?.value;
const o = row.o?.value;
const g = row.g?.value ?? "";
if (!s || !p || o === undefined) continue;
if (denied(s)) continue;
let entry = bySubject.get(s);
if (!entry) {
entry = { subject: s, graph: g, props: {} };
bySubject.set(s, entry);
for (const { doc, rows } of perDoc) {
if (caps.governsRead(doc) && !caps.canRead(doc, user)) continue;
// Anchored to `doc`, so every row belongs to `doc`; the subject IS the doc NURI
// (writeEntity invariant). Pin subject/graph to the doc NURI (the anchor), which
// is stable regardless of the repo_graph_name overlay suffix the store carries.
for (const row of rows) {
const p = row.p?.value;
const o = row.o?.value;
if (!p || o === undefined) continue;
let entry = bySubject.get(doc);
if (!entry) {
entry = { subject: doc, graph: doc, props: {} };
bySubject.set(doc, entry);
}
(entry.props[p] ??= []).push(o);
}
(entry.props[p] ??= []).push(o);
}
return [...bySubject.values()];
}
packages/client/test/read-model.test.ts
+30 -31
View File
@@ -2,32 +2,28 @@ import { test, expect, mock } from "bun:test";
import { readUnion } from "../src/read-model";
import { configure, configureStoreRegistry } from "../src/polyfill";
// A fake `ng` whose sparql_query answers the ANCHORLESS union query with the
// triples of the requested subjects, and the anchored ASK (open step) with an
// empty result. Each entity subject IRI IS its own document NURI (writeEntity
// convention), so the fixture keys triples by the doc NURI.
// A fake `ng` whose sparql_query answers the ANCHORED per-doc query (SELECT ?s ?p ?o
// WHERE { ?s ?p ?o }, anchor = the doc NURI) with ONLY that doc's triples. There is
// NO anchorless union scan: each doc is read independently by its own anchor. Each
// entity subject IRI IS its own document NURI (writeEntity convention), so the
// fixture keys triples by the doc NURI and returns them for the matching anchor.
function fakeNgWith(triplesByDoc: Record<string, Array<[string, string]>>) {
return {
doc_create: mock(async () => "did:ng:o:new"),
sparql_update: mock(async () => undefined),
sparql_query: mock(async (_sid: string, query: string, _base: unknown, anchor: unknown) => {
// The open step is `ASK { ?s ?p ?o }` with an anchor → return a truthy ASK.
if (query.startsWith("ASK")) return { boolean: true };
// The union query is anchorless (anchor undefined) with a VALUES ?s block.
if (anchor !== undefined) return { results: { bindings: [] } };
const bindings: Array<Record<string, { value: string }>> = [];
for (const [doc, triples] of Object.entries(triplesByDoc)) {
// Only surface docs whose NURI is named in the VALUES block.
if (!query.includes(`<${doc}>`)) continue;
for (const [p, o] of triples) {
bindings.push({
g: { value: `${doc}:graph` },
s: { value: doc },
p: { value: p },
o: { value: o },
});
}
sparql_query: mock(async (_sid: string, _query: string, _base: unknown, anchor: unknown) => {
// Every read is ANCHORED to one doc NURI — never anchorless.
if (anchor === undefined) {
throw new Error("read-model must NEVER run an anchorless (union) query");
}
const doc = anchor as string;
const triples = triplesByDoc[doc];
if (!triples) return { results: { bindings: [] } };
const bindings = triples.map(([p, o]) => ({
s: { value: doc },
p: { value: p },
o: { value: o },
}));
return { results: { bindings } };
}),
};
@@ -46,23 +42,26 @@ function inject(triplesByDoc: Record<string, Array<[string, string]>>) {
const TYPE = "http://www.w3.org/1999/02/22-rdf-syntax-ns#type";
const FP = "http://festipod.org/";
test("readUnion opens each doc then runs ONE anchorless union query", async () => {
test("readUnion reads each doc with its OWN anchored query (never anchorless)", async () => {
const ng = inject({
"did:ng:o:a": [[TYPE, `${FP}Event`], [`${FP}title`, "A"]],
"did:ng:o:b": [[TYPE, `${FP}Event`], [`${FP}title`, "B"]],
});
const subjects = await readUnion(["did:ng:o:a", "did:ng:o:b"]);
// Two per-doc ASK opens + one anchorless union query = 3 sparql_query calls.
expect(ng.sparql_query).toHaveBeenCalledTimes(3);
const anchorless = ng.sparql_query.mock.calls.filter(
(c: unknown[]) => !String(c[1]).startsWith("ASK") && c[3] === undefined,
// One anchored query per doc = 2 sparql_query calls, each anchored (c[3] set).
expect(ng.sparql_query).toHaveBeenCalledTimes(2);
const anchored = ng.sparql_query.mock.calls.filter((c: unknown[]) => c[3] !== undefined);
expect(anchored.length).toBe(2);
// The anchors are exactly the requested doc NURIs.
expect(new Set(anchored.map((c: unknown[]) => c[3]))).toEqual(
new Set(["did:ng:o:a", "did:ng:o:b"]),
);
expect(anchorless.length).toBe(1);
expect(subjects.length).toBe(2);
const a = subjects.find((s) => s.subject === "did:ng:o:a")!;
expect(a.props[`${FP}title`]).toEqual(["A"]);
expect(a.graph).toBe("did:ng:o:a");
});
test("readUnion groups predicates per subject", async () => {
@@ -86,12 +85,12 @@ test("readUnion returns [] for an empty doc set (no query)", async () => {
expect(ng.sparql_query).toHaveBeenCalledTimes(0);
});
test("a doc that fails to open is skipped, not aborting the union", async () => {
test("a doc that fails to read is skipped, not aborting the batch", async () => {
const ng = fakeNgWith({ "did:ng:o:ok": [[TYPE, `${FP}Event`], [`${FP}title`, "ok"]] });
// Make the OPEN (ASK) throw for the bad doc only.
const orig = ng.sparql_query;
// Make the anchored read throw for the bad doc only.
ng.sparql_query = mock(async (sid: string, query: string, base: unknown, anchor: unknown) => {
if (query.startsWith("ASK") && anchor === "did:ng:o:bad") throw new Error("RepoNotFound");
if (anchor === "did:ng:o:bad") throw new Error("RepoNotFound");
return orig(sid, query, base, anchor);
}) as any;
configure({ ng: ng as any, useShape: (() => {}) as any });
@@ -101,6 +100,6 @@ test("a doc that fails to open is skipped, not aborting the union", async () =>
});
const subjects = await readUnion(["did:ng:o:ok", "did:ng:o:bad"]);
// The bad doc opened-failed but the good one still lists.
// The bad doc failed its read but the good one still lists.
expect(subjects.map((s) => s.subject)).toEqual(["did:ng:o:ok"]);
});