Data-layer BDD testing infrastructure and steps/frontend → steps/ui rename

- Rename steps/frontend/ to steps/ui/ across all modules and shared - Add data-layer test harness (mock + real broker modes) with Playwright - Add inscription data-layer steps (@data scenarios) - Add test auth setup script and browser debug script - Update docs (architecture, BDD testing, data-layer testing) - Add ADR for headless wallet creation decision Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-12 17:56:48 +01:00
parent 901fd659df
commit 6f9b3ece34
26 changed files with 1721 additions and 9869 deletions
@@ -0,0 +1,54 @@
+# Automated Headless Wallet Creation for CI
+
+**Date:** 2026-03-12 15:00
+**Status:** Accepted
+
+## Context
+
+Data-layer BDD tests (`@data` scenarios) require a NextGraph wallet in a persistent Chromium profile. Previously, the first run required manual interaction: a visible browser opened and the user had to create a wallet and close the browser. This blocked CI execution.
+
+## Options Considered
+
+### Option A: Programmatic wallet creation via NG SDK
+Call `ng.wallet_create()` directly from Node/Bun, bypassing the UI entirely.
+
+**Arguments for:**
+- Fastest execution
+- No browser needed for wallet creation
+
+**Arguments against:**
+- `@ng-org/web` is browser-only (WASM + postMessage)
+- Would need to reverse-engineer the registration API at `account.nextgraph.eu`
+- Doesn't test the real auth flow
+
+### Option B: Automate the browser UI flow headlessly
+Use Playwright to drive the same wallet creation UI a real user would use, but in headless mode.
+
+**Arguments for:**
+- Tests the real auth/login feature end-to-end
+- No API reverse-engineering needed
+- Same persistent profile used for subsequent test runs
+- CI-ready with no manual steps
+
+**Arguments against:**
+- Depends on `nextgraph.eu` and `account.nextgraph.eu` being reachable
+- UI changes in NextGraph could break the automation
+- Adds ~27s to first run
+
+## Decision
+
+Option B — automate the browser UI. The wallet creation flow (navigate to `nextgraph.eu` → "Create Wallet" → accept ToS at `account.nextgraph.eu` → fill username/password → submit) is itself a legitimate test of the app's auth feature. The dependency on external services is acceptable since the tests already depend on the broker being reachable.
+
+## Consequences
+
+**Positive:**
+- Tests are fully CI-ready (no human interaction)
+- Auth/login flow is tested as a side effect
+- Single command `bun run test:data` works from a clean state
+
+**Negative:**
+- Requires internet access (nextgraph.eu, account.nextgraph.eu)
+- Fragile to NextGraph UI changes (button text, form IDs)
+
+**Risks:**
+- `account.nextgraph.eu` rate limiting could block CI runs that frequently recreate wallets
@@ -18,7 +18,7 @@ src/modules/
 Each module can contain:
 - `screens/` — React screen components
 - `features/` — Gherkin `.feature` files (BDD specs)
- `steps/{frontend,backend,e2e}/` — Cucumber step definitions by layer
+- `steps/{ui,data,e2e}/` — Cucumber step definitions by layer

 ## Import Rules

@@ -45,7 +45,7 @@ src/modules/event/screens/EventDetailScreen.tsx
 | `hooks/` | `useShapeWithDefaults` (NextGraph) |
 | `shapes/` | SHEX definitions + ORM TypeScript bindings |
 | `utils/` | `ngSession.ts`, `ngBootstrap.ts` |
-| `steps/frontend/` | Shared BDD step definitions (navigation, screen, form) |
+| `steps/ui/` | Shared BDD step definitions (navigation, screen, form) |
 | `support/` | Cucumber `world.ts`, `hooks.ts` |
 | `types/` | `gherkin.ts` (ParsedFeature, ParsedScenario types) |
 | `lib/` | `utils.ts` (cn helper for Tailwind) |
@@ -15,12 +15,12 @@ Each module has step directories for three test layers:

 ```
 src/modules/event/steps/
-  frontend/     # UI/screen assertions (active)
-  backend/      # Data layer assertions (planned)
+  ui/           # UI/screen assertions (active)
+  data/         # Data layer assertions
  e2e/          # Full integration (planned)
 ```

-Shared steps (cross-domain) live in `src/shared/steps/frontend/`.
+Shared steps (cross-domain) live in `src/shared/steps/ui/`.

 ## Feature Files

@@ -37,7 +37,7 @@ Tagged with `@CATEGORY @priority-N` for filtering.

 ## Step Definitions

-### Shared Steps (`src/shared/steps/frontend/`)
+### Shared Steps (`src/shared/steps/ui/`)

 | File | Purpose |
 |------|---------|
@@ -45,7 +45,7 @@ Tagged with `@CATEGORY @priority-N` for filtering.
 | `form.steps.ts` | Form field validation, required fields, import/duplicate detection |
 | `screen.steps.ts` | Screen content assertions (participants, events, profiles, QR codes) |

-### How Frontend Steps Work
+### How UI Steps Work

 Steps analyze screen **source code** (not rendered DOM):
 1. `world.ts` loads screen `.tsx` file content via `loadScreenSource()`
@@ -89,13 +89,17 @@ Scripts in `scripts/` parse features and steps into TypeScript data files consum
 |--------|-------|--------|
 | `parse-features.ts` | `src/modules/*/features/*.feature` | `src/shared/data/features.ts` |
 | `parse-test-results.ts` | `reports/cucumber-report.json` | `src/shared/data/testResults.ts` |
-| `extract-step-definitions.ts` | `src/shared/steps/frontend/*.ts` | `src/shared/data/stepDefinitions.ts` |
+| `extract-step-definitions.ts` | `src/shared/steps/ui/*.ts` | `src/shared/data/stepDefinitions.ts` |

 Run all: `bun run test:cucumber`

+## Data-Layer Testing
+
+`@data` scenarios test through the real NextGraph broker. See [data-layer-testing](./data-layer-testing.md) for full architecture.
+
 ## Adding New Steps

-1. **Module-specific**: Create in `src/modules/{module}/steps/frontend/`
-2. **Cross-domain**: Add to `src/shared/steps/frontend/`
+1. **Module-specific**: Create in `src/modules/{module}/steps/ui/`
+2. **Cross-domain**: Add to `src/shared/steps/ui/`
 3. Import `FestipodWorld` type from `../../support/world` (shared) or adjust relative path
 4. Run `bun run steps:extract` to regenerate tooltip data
@@ -0,0 +1,113 @@
+# Data-Layer Testing
+
+BDD scenarios tagged `@data` test the real NextGraph data pipeline through a broker, not mocked data.
+
+## Overview
+
+`@data` scenarios run Cucumber steps against a real NextGraph broker. Playwright drives a Chromium instance that authenticates with the broker, which loads our test harness in an iframe. The harness uses real `useShape`/ORM subscriptions and exposes a `window.__testData` bridge for step definitions.
+
+## Architecture
+
+```
+Cucumber steps → Playwright (Chromium, persistent profile)
+                      ↓
+               https://nextgraph.eu/auth/#/?o=http://127.0.0.1:{port}
+                      ↓
+               Broker wallet login (automated)
+                      ↓
+               Broker loads app in iframe → http://127.0.0.1:{port}
+                      ↓
+               harness-ng.tsx (init → useShape → ORM → broker)
+                      ↓
+               window.__testData bridge
+```
+
+## Dual Mode
+
+- **Real broker** (default): `harness-ng.tsx` with NextGraph ORM through broker iframe
+- **Mock fallback**: `harness.tsx` with standalone DeepSignalSets (if NG harness build fails)
+
+## Wallet Lifecycle
+
+Fully automated — no manual interaction required. CI-ready.
+
+### First Run (wallet creation)
+1. `BeforeAll` detects no `.wallet-ready` marker in `.playwright-profile/`
+2. Launches headless Chromium with persistent profile
+3. Navigates to `https://nextgraph.eu/` → clicks "Create Wallet"
+4. Redirected to `account.nextgraph.eu` → clicks "I accept" (ToS)
+5. Redirected back → fills username/password form → submits
+6. Wallet created in localStorage → marker written
+7. This is also a real test of the app's auth/login feature
+
+### Subsequent Runs (automated login)
+1. Marker found → skip wallet creation
+2. Headless Chromium with persistent profile
+3. Automated login: click "Login" → click wallet link → fill password → submit
+4. Broker authenticates, loads app harness in iframe
+5. Harness initializes NG, creates ORM subscriptions, seeds data if needed
+6. `window.__testData.ready` → steps execute via `appFrame.evaluate()`
+
+### Wallet Credentials
+- Name: `festipod-tests`
+- Password: `festipod-tests`
+
+## Key Technical Details
+
+### Chromium Flags
+```
+--disable-features=PrivateNetworkAccessRespectPreflightResults,BlockInsecurePrivateNetworkRequests,...
+--allow-insecure-localhost
+--disable-web-security
+```
+Required because broker at `nextgraph.eu` (public) loads harness from `http://127.0.0.1:{port}` (local) in an iframe — Chromium's Private Network Access blocks this by default.
+
+### Persistent Profile (`.playwright-profile/`)
+- Stores NG wallet in localStorage (`ng_wallets` on `nextgraph.eu`, `ng_bootstrap` on `nextgraph.net`)
+- Gitignored
+- Must use full Chrome binary, not `chrome-headless-shell`
+
+### HTTP Server
+- Started in `BeforeAll` on auto-assigned port (`127.0.0.1:0`)
+- Serves harness HTML at `/` and JS bundle at `/harness.js` (separate files — inline script breaks due to special characters in bundle)
+- Shut down in `AfterAll`
+
+### ORM Subscriptions
+Harness creates subscriptions for all three shapes with scope `did:ng:i` (private store):
+- `FpEventShapeType` → events
+- `FpUserProfileShapeType` → users
+- `FpParticipationShapeType` → participations
+
+### Test Bridge (`window.__testData`)
+Exposed by the harness, consumed by steps via `appFrame.evaluate()`:
+- `events`, `users`, `participations` — live DeepSignalSets
+- `currentUserId` — IRI of the test user
+- `getEvent(id)`, `getEventByTitle(title)` — lookups
+- `joinEvent(eventId, userId)`, `leaveEvent(eventId, userId)` — mutations
+- `isParticipating(eventId, userId)`, `getEventParticipants(eventId)` — queries
+- `updateEvent(eventId, updates)` — field updates
+
+## Files
+
+| File | Purpose |
+|------|---------|
+| `src/shared/test-harness/harness-ng.tsx` | Real broker harness (useShape through broker iframe) |
+| `src/shared/test-harness/harness.tsx` | Mock harness (DeepSignalSets, no broker) |
+| `src/shared/support/hooks.ts` | Playwright lifecycle (wallet creation, login automation, iframe detection) |
+| `src/shared/support/world.ts` | World with `page`/`appFrame` fields |
+| `src/modules/event/steps/data/inscription.steps.ts` | Inscription data steps |
+| `.playwright-profile/` | Persistent Chromium profile (gitignored) |
+| `scripts/debug-browser.ts` | Manual browser debug tool — launches headed Chromium to inspect broker interactions |
+| `.playwright-profile-debug/` | Chromium profile created by debug-browser.ts (gitignored) |
+
+## Commands
+
+```bash
+bun run test:data      # Run @data scenarios (real broker if wallet exists, mock fallback)
+bun run test:cucumber  # Run all scenarios (UI + data)
+```
+
+## See Also
+
+- [BDD Testing](./bdd-testing.md) — general Cucumber setup, UI-layer steps
+- [Data Layer](./data-layer.md) — NextGraph stack, shapes, context providers