From f4c497681f9b88af5faff85a5b744f3f645f0c5c Mon Sep 17 00:00:00 2001
From: Jan Novak <kacerr.cz@gmail.com>
Date: Thu, 7 May 2026 22:37:52 +0200
Subject: [PATCH] chore: CHANGELOG and progress tracker for M5.3

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---
 CHANGELOG.md                                  |   5 +
 ...-05-03-2349-go-backend-rewrite-progress.md |   2 +-
 ...-2114-go-rewrite-m5-3-python-api-shadow.md | 113 ++++++++++++++++++
 3 files changed, 119 insertions(+), 1 deletion(-)
 create mode 100644 docs/plans/2026-05-07-2114-go-rewrite-m5-3-python-api-shadow.md

diff --git a/CHANGELOG.md b/CHANGELOG.md
index f3d2fcf..a3684f8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,10 @@
 # Changelog
 
+## 2026-05-07 22:37 CEST — feat(py): M5.3 — Python /api/* shadow endpoints
+
+- `app.py`: four new JSON routes (`/api/version`, `/api/adults`, `/api/juniors`, `/api/payments`) mirroring the Go `/api/*` handlers; `_unwrap_view_model_for_api()` helper expands pre-serialised JSON strings and renames `month_labels_json` → `month_labels`, `raw_payments_json` → `raw_payments` to match Go wire contract.
+- `tests/test_app.py`: four new smoke tests asserting top-level key sets and that unwrapped fields are objects (not strings).
+
 ## 2026-05-07 20:13 CEST — feat(go): M5.2 — HTTP handlers for /api/adults, /api/juniors, /api/payments, /api/version
 
 - `web/api/handler.go`: `Handler` struct + `ServeAdults`, `ServeJuniors`, `ServePayments`, `ServeVersion` using `membership.Sources`.
diff --git a/docs/plans/2026-05-03-2349-go-backend-rewrite-progress.md b/docs/plans/2026-05-03-2349-go-backend-rewrite-progress.md
index 883b487..b97f69f 100644
--- a/docs/plans/2026-05-03-2349-go-backend-rewrite-progress.md
+++ b/docs/plans/2026-05-03-2349-go-backend-rewrite-progress.md
@@ -99,7 +99,7 @@ Goal: byte-equal JSON between Python and Go for every route. This is the parity
 
 - [x] **M5.1** Hand-author Go structs for `/api/adults`, `/api/juniors`, `/api/payments`, `/api/version` with explicit `json:` tags matching Python keys; emit JSON Schemas via `github.com/invopop/jsonschema` to `tests/fixtures/api-schema/` — `f253e3f`
 - [x] **M5.2** Implement Go handlers for `/api/*` routes composing `services/*` results into the JSON structs — `7d48e8f`
-- [ ] **M5.3** Add Python `/api/X` shadow endpoints in [app.py](app.py): `jsonify(view_model_dict)` — no transformation
+- [x] **M5.3** Add Python `/api/X` shadow endpoints in [app.py](app.py): `jsonify(view_model_dict)` — no transformation — `40e4a9e`
 - [ ] **M5.4** Build `cmd/parity/main.go`: hits both backends' `/api/X`, normalizes allowlist (`render_time.total`, `build_meta`), prints `cmp.Diff`. Add `make parity` target
 
 **Gate:** For each route, `make parity` reports zero non-allowlisted diffs across the M3 fixture corpus.
diff --git a/docs/plans/2026-05-07-2114-go-rewrite-m5-3-python-api-shadow.md b/docs/plans/2026-05-07-2114-go-rewrite-m5-3-python-api-shadow.md
new file mode 100644
index 0000000..b7d75a7
--- /dev/null
+++ b/docs/plans/2026-05-07-2114-go-rewrite-m5-3-python-api-shadow.md
@@ -0,0 +1,113 @@
+# M5.3 — Python `/api/X` shadow endpoints
+
+Companion to:
+- [2026-05-03-2349-go-backend-rewrite.md](2026-05-03-2349-go-backend-rewrite.md) (master design)
+- [2026-05-03-2349-go-backend-rewrite-progress.md](2026-05-03-2349-go-backend-rewrite-progress.md) (M5.3 row)
+- [2026-05-07-1431-m5-json-api-parity.md](2026-05-07-1431-m5-json-api-parity.md) (Python view-model extraction prep)
+- [2026-05-07-1650-go-rewrite-m5-1-api-structs-schemas.md](2026-05-07-1650-go-rewrite-m5-1-api-structs-schemas.md) (Go wire types)
+
+## Context
+
+M5.1 (Go wire types + JSON Schemas) and M5.2 (Go HTTP handlers for `/api/adults` `/api/juniors` `/api/payments` `/api/version`) have merged. M5.3 mirrors the same four endpoints on the Python Flask side so M5.4's `cmd/parity` tool can hit both backends and diff the JSON. After M5.3, every byte the Go side emits has a Python counterpart to compare against.
+
+The Python view-model builders ([scripts/views.py](scripts/views.py)) already produce dicts very close to the wire shape — except three template-only fields (`member_data`, `month_labels_json`, `raw_payments_json`) are pre-`json.dumps`'d for inline `<script>` blocks. M5.1's plan called this out explicitly: M5.3's `/api/X` is `jsonify(unwrap_json_strings(view_model_dict))` — a 4-line shim, not real transformation logic.
+
+## Approach
+
+Add four shadow routes to [app.py](app.py) and one private unwrap helper. Builders and templates are untouched.
+
+### Decisions
+
+1. **Unwrap shim lives in `app.py`**, not `scripts/views.py`. It's 4 lines, only the API routes use it, and it's parity-only scaffolding that M8 will delete. Keeping `views.py` free of HTTP-layer concerns means cleaner deletion later.
+2. **No data-loading helper extraction.** Each shadow route duplicates ~8 lines of cache loads from its sibling HTML route. A helper would have to thread `attendance_url` / `payments_url` / `bank_account` and the adults-vs-juniors-vs-payments branching back out — net negative for code that M8 will erase wholesale.
+3. **Drop `record_step` calls in API routes.** `record_step` only feeds `inject_render_time` (a Jinja `context_processor`); JSON responses don't go through templates, so timing breakdown has no consumer.
+4. **`/api/version` is a one-liner.** `BUILD_META` already has the keys (`tag`, `commit`, `build_date`) Go emits. Just `jsonify(BUILD_META)`. Existing `/version` route stays as-is — the new endpoint sits alongside.
+5. **Tests assert key sets and unwrap, not values.** Hard-code `EXPECTED_ADULTS_KEYS` etc. as module constants in [tests/test_app.py](tests/test_app.py). Catches drift in unit tests rather than waiting for M5.4 parity diffs. Type-check the unwrapped fields (`isinstance(member_data, dict)` etc.) to prove the shim ran.
+
+### The shim
+
+```python
+def _unwrap_view_model_for_api(vm: dict) -> dict:
+    out = dict(vm)
+    out["member_data"] = _json.loads(out.pop("member_data"))
+    out["month_labels"] = _json.loads(out.pop("month_labels_json"))
+    out["raw_payments"] = _json.loads(out.pop("raw_payments_json"))
+    return out
+```
+
+Note the **rename**: `month_labels_json` → `month_labels`, `raw_payments_json` → `raw_payments` (matches Go contract per [adults.go](go/internal/web/api/adults.go) JSON tags).
+
+### Route skeleton (`/api/adults`)
+
+```python
+@app.route("/api/adults")
+def api_adults():
+    attendance_url = f"https://docs.google.com/spreadsheets/d/{ATTENDANCE_SHEET_ID}/edit"
+    payments_url = f"https://docs.google.com/spreadsheets/d/{PAYMENTS_SHEET_ID}/edit"
+    members_data = get_cached_data("attendance_regular", ATTENDANCE_SHEET_ID, get_members_with_fees)
+    if not members_data:
+        return jsonify({"error": "no data"}), 503
+    members, sorted_months = members_data
+    transactions = get_cached_data("payments_transactions", PAYMENTS_SHEET_ID, fetch_sheet_data, PAYMENTS_SHEET_ID, CREDENTIALS_PATH)
+    exceptions = get_cached_data(
+        "exceptions_dict", PAYMENTS_SHEET_ID, fetch_exceptions,
+        PAYMENTS_SHEET_ID, CREDENTIALS_PATH,
+        serialize=lambda d: [[list(k), v] for k, v in d.items()],
+        deserialize=lambda c: {tuple(k): v for k, v in c},
+    )
+    result = reconcile(members, sorted_months, transactions, exceptions)
+    vm = build_adults_view_model(
+        members, sorted_months, result, transactions,
+        datetime.now().strftime("%Y-%m"),
+        attendance_url=attendance_url, payments_url=payments_url, bank_account=BANK_ACCOUNT,
+    )
+    return jsonify(_unwrap_view_model_for_api(vm))
+```
+
+`/api/juniors` mirrors with `adapt_junior_members` + `JUNIOR_SHEET_GID` + `build_juniors_view_model`. `/api/payments` skips the unwrap (its builder has no JSON-string fields): `return jsonify(vm)`. `/api/version`: `return jsonify(BUILD_META)`.
+
+## Files to modify
+
+- [app.py](app.py)
+  - L10: add `jsonify` to `from flask import ...`.
+  - Add `_unwrap_view_model_for_api` near [BUILD_META](app.py#L67) (it already imports `json as _json`).
+  - Add four routes: `/api/version`, `/api/adults`, `/api/juniors`, `/api/payments`. Place them after [`/version`](app.py#L143) for grouping.
+- [tests/test_app.py](tests/test_app.py)
+  - Add `EXPECTED_ADULTS_KEYS`, `EXPECTED_JUNIORS_KEYS`, `EXPECTED_PAYMENTS_KEYS`, `EXPECTED_VERSION_KEYS` module constants (sourced from [adults.go](go/internal/web/api/adults.go) / [juniors.go](go/internal/web/api/juniors.go) / [payments.go](go/internal/web/api/payments.go) / [version.go](go/internal/web/api/version.go) JSON tags).
+  - Four new test functions: `test_api_adults`, `test_api_juniors`, `test_api_payments`, `test_api_version`. Reuse the existing `_bypass_cache` patcher and the `fetch_sheet_data` / `fetch_exceptions` / `get_members_with_fees` / `get_junior_members_with_fees` mocks already in the file.
+  - Each adults/juniors test asserts: `200`, `response.is_json`, `set(json.keys()) == EXPECTED_*_KEYS`, `isinstance(json["member_data"], dict)`, `isinstance(json["month_labels"], dict)`, `isinstance(json["raw_payments"], dict)`.
+- [CHANGELOG.md](CHANGELOG.md): post-merge entry per CLAUDE.md format.
+- [docs/plans/2026-05-03-2349-go-backend-rewrite-progress.md:102](docs/plans/2026-05-03-2349-go-backend-rewrite-progress.md#L102): tick M5.3 with merge SHA.
+
+## Reusable existing code
+
+- [`build_adults_view_model`](scripts/views.py#L64), [`build_juniors_view_model`](scripts/views.py#L240), [`build_payments_view_model`](scripts/views.py#L432) — call as-is, no changes.
+- [`reconcile`](scripts/match_payments.py) and [`adapt_junior_members`](scripts/views.py#L48) — same.
+- [`get_cached_data`](app.py#L36), [`fetch_sheet_data`](scripts/match_payments.py), [`fetch_exceptions`](scripts/match_payments.py), [`get_members_with_fees`](scripts/attendance.py), [`get_junior_members_with_fees`](scripts/attendance.py) — same call sites as the HTML routes.
+- [`BUILD_META`](app.py#L67) — already shaped to match Go's `VersionResponse`.
+
+## Verification
+
+1. `make test` — all existing tests still pass; four new `test_api_*` tests pass.
+2. `python -c "import app"` — catches the missing `jsonify` import.
+3. `make web-py`, then:
+   - `curl -s localhost:5001/api/version | jq .` → `{tag, commit, build_date}`.
+   - `curl -s localhost:5001/api/adults | jq 'keys'` → 14 keys, no `_json` suffix anywhere.
+   - `curl -s localhost:5001/api/adults | jq '.member_data | type, .month_labels | type, .raw_payments | type'` → all `"object"` (proves unwrap).
+   - Same checks on `/api/juniors` and `/api/payments`.
+4. Visit `/adults`, `/juniors`, `/payments` in browser — HTML still renders identically (regression check on builders).
+5. **Optional pre-M5.4 peek:** `make web-go` on :8080 + `make web-py` on :5001, then `diff <(curl -s :5001/api/adults | jq -S .) <(curl -s :8080/api/adults | jq -S .)`. Expect non-zero diff (raw transaction key shape, see below) — that is fine; M5.4 surfaces and resolves these.
+
+## Out of scope (M5.4 will surface and resolve)
+
+These are known parity friction points; **don't fix in M5.3** — the whole point of M5.4 is to enumerate them:
+
+- **`raw_payments[name][i]` row shape**: Python emits raw Google Sheets row dicts with column-header keys (e.g. `"VS"`, `"Sync ID"`); Go's `RawTransaction` uses snake_case (`vs`, `sync_id`). Keys and types will diverge.
+- **`unmatched[]`**: same divergence (same raw row dicts).
+- **`null` vs missing keys**: Python omits keys never set; Go zero-value structs may emit `null`/`""` depending on `omitempty`.
+- **`Decimal` / float precision** in `grouped_payments` amounts.
+- **Field insertion order**: `jsonify` preserves insertion order (Python 3.7+); Go marshals struct fields in declaration order. Likely fine, parity tool will tell.
+
+## Branch + MR
+
+Per CLAUDE.md: branch `feat/go-m5-3-python-api-shadow`, push with `-u`, open MR via `tea pr create --title ... --description ... --base main --head feat/go-m5-3-python-api-shadow`. Do not merge from CLI.