Compare commits

...

18 Commits

Author SHA1 Message Date
2f635db2b4 chore: CHANGELOG for M5.4 parity coercions (PR #25)
All checks were successful
Deploy to K8s / deploy (push) Successful in 7s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-08 00:27:34 +02:00
709a2f2335 Merge pull request 'fix(py): parity coercions — amount/message types + junior '?' sticky' (#25) from fix/py-parity-coercions into main
All checks were successful
Deploy to K8s / deploy (push) Successful in 8s
Reviewed-on: #25
2026-05-07 22:26:39 +00:00
58973473c9 fix(py): make junior '?' cell text sticky across exception overrides
All checks were successful
Deploy to K8s / deploy (push) Successful in 8s
Go's build_juniors sets cellText = "?" + countStr whenever
md.IsUnknown is true, regardless of whether an exception overrides the
expected amount. Python was checking expected == "?" for this branch,
but reconcile replaces expected with the exception amount (e.g. 0)
before the view builder runs, so the "?" was silently dropped to "-".

Fix: derive is_unknown from original_expected == "?" (set before
exception substitution) instead of expected == "?". Also align the
tooltip guard: Go only shows Received/Expected tooltip for non-unknown
months (or when paid > 0), matching the same is_unknown flag.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-08 00:26:04 +02:00
b68d95d217 fix(py): coerce amount to float and message to string in tx projection
Two remaining make parity diffs vs Go:

- amount: non-numeric sheet values like "---" passed through as strings;
  Go's parseFloat silently returns 0.0 for unparseable values. Add
  get_float helper that matches that behaviour.
- message: numeric cell values (e.g. a bank reference in the message
  column) passed through as float64; Go's getVal uses fmt.Sprint and
  always emits a string. Apply get_str to the message field.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-08 00:26:04 +02:00
07ca1cd9e1 fix(py): coerce VS column to string in payments tx projection
All checks were successful
Deploy to K8s / deploy (push) Successful in 7s
The Sheets API returns VS (variabilní symbol) cells as float64 when
the column is number-formatted, so Python was emitting vs: 0 (a JSON
number) while Go's getVal uses fmt.Sprint and always emits vs: "0"
(a string). Add get_str helper that converts whole-number floats via
int() first (matching Go's %g formatting), applied to the vs field.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-07 23:59:35 +02:00
5dcac25c13 chore: CHANGELOG entry for M5.4 fix #2 (py vs/sync_id)
All checks were successful
Deploy to K8s / deploy (push) Successful in 6s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-07 23:52:04 +02:00
fc47606b1c Merge pull request 'feat(py): M5.4 fix #2 — add vs and sync_id to payments tx projection' (#23) from fix/py-payments-add-vs-syncid into main
All checks were successful
Deploy to K8s / deploy (push) Successful in 9s
Reviewed-on: #23
2026-05-07 21:51:12 +00:00
65694ad378 feat(py): M5.4 fix #2 — add vs and sync_id to payments tx projection
All checks were successful
Deploy to K8s / deploy (push) Successful in 7s
Python's fetch_sheet_data read 9 sheet columns but skipped VS and
Sync ID, causing make parity to report extra fields on every raw
payment row emitted by the Go backend. Both columns are already on
the sheet; add idx_vs / idx_sync_id lookups and the matching keys
to the tx dict so the Python /api/* wire shape matches Go's
RawTransaction.

Update /api/* test fixtures to include vs/sync_id keys for realism.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-07 23:50:33 +02:00
092dff25a5 Merge pull request 'fix(go): accept single-digit day/month in attendance date headers' (#22) from fix/go-attendance-date-parser into main
All checks were successful
Deploy to K8s / deploy (push) Successful in 10s
Reviewed-on: #22
2026-05-07 21:39:02 +00:00
56c21bcf03 fix(go): accept single-digit day/month in attendance date headers
All checks were successful
Deploy to K8s / deploy (push) Successful in 8s
parseDates was using "02.01.2006" / "01/02/2006" which require
zero-padded fields. The Czech attendance sheet headers contain dates
like "1.6.2026", "23.3.2026", "6.4.2026" — Go silently dropped those
columns while Python's strptime accepted them. Effect was a missing
2026-06 month on /api/juniors plus undercounted attendance in any month
with single-digit columns; surfaced via make parity.

Use the unpadded reference forms "2.1.2006" / "1/2/2006" instead — Go's
time.Parse accepts both padded and unpadded inputs against them.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 23:38:06 +02:00
208f762c18 Merge pull request 'feat(go): M5.4 — parity diff binary + make parity' (#19) from feat/go-m5-4-parity-binary into main
All checks were successful
Deploy to K8s / deploy (push) Successful in 10s
Reviewed-on: #19
2026-05-07 21:25:23 +00:00
4d035213b5 Merge pull request 'fix(go): pass raw value to FormatDate so numeric dates format' (#21) from fix/go-date-format into main
All checks were successful
Deploy to K8s / deploy (push) Successful in 6s
Reviewed-on: #21
2026-05-07 21:24:42 +00:00
2b15280d03 fix(go): exclude /api/version from parity diff — identity, not contract
All checks were successful
Deploy to K8s / deploy (push) Successful in 7s
/api/version returns each binary's own tag/commit/build_date, which
differs by design between independently built backends. Diffing it
always produces a false positive. Drop it from allRoutes; the route
remains reachable via `make parity ARGS="-route /api/version"`.

Also remove the vestigial `build_meta` allowlist entry (Python returns
the build dict as the top-level response body, not nested under
build_meta, so the scrubber never matched anything).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 23:23:38 +02:00
723152cdad fix(go): pass raw value to FormatDate so numeric serial-day dates format
All checks were successful
Deploy to K8s / deploy (push) Successful in 11s
The transaction-row parser in services/membership/sources.go used a
helper (`getVal`) that did `fmt.Sprint(row[i])` before passing to
`matching.FormatDate`.  The Sheets API returns date-formatted cells
as `float64` (Sheets serial-day numbers); pre-stringifying defeated
`FormatDate`'s `case float64:` dispatch, so values like 46147 leaked
through unchanged as the string "46147" instead of being converted
to "2026-05-05".

Surfaced by `make parity` (M5.4) — every `transactions[].date` on
/api/adults and /api/juniors differed between Python and Go.  Python
side passes the raw value through directly (`isinstance(val, (int,
float))` in scripts/match_payments.py format_date), so it was always
correct.

Added a `getRaw` helper that returns row[i] without stringifying;
only the date column needs it.  Extended TestLoadTransactions with
a numeric-serial-day row to lock in the regression.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 23:17:45 +02:00
fe0e49a134 feat(go): M5.4 — parity diff binary + make parity
All checks were successful
Deploy to K8s / deploy (push) Successful in 8s
Adds cmd/parity/main.go: a standalone Go binary that GETs
/api/version, /api/adults, /api/juniors, /api/payments from both
the Python (:5001) and Go (:8080) backends, scrubs an allowlist
(render_time.total, build_meta), and prints cmp.Diff for any
remaining differences.  Exits 0 on full match, 1 on diffs, 2 on
fetch/parse errors — CI-friendly for M7.2.

- go/cmd/parity/main.go: flags (-py, -go, -route, -timeout), fetch
  helper, allowlist scrubber (dotted-path aware), exit-code logic.
- go/cmd/parity/scrub_test.go: 4 unit tests for the scrubber.
- go/go.mod: promote github.com/google/go-cmp to direct dep.
- Makefile: parity target + help entry.
- Progress tracker: M5.4 ticked; milestone updated to M5 complete.
- Plan archived to docs/plans/2026-05-07-2254-m5-4-parity-binary.md.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 23:08:42 +02:00
e5a272b682 Merge pull request 'fix(go): default CacheDir to tmp/go to avoid Python collision' (#20) from fix/cache-collision into main
All checks were successful
Deploy to K8s / deploy (push) Successful in 11s
Reviewed-on: #20
2026-05-07 21:07:18 +00:00
8b3064ffab fix(go): default CacheDir to tmp/go to avoid Python collision
All checks were successful
Deploy to K8s / deploy (push) Successful in 9s
Previously both backends defaulted to `CacheDir=tmp` and used the
same cache keys (`attendance_regular`, `attendance_juniors`,
`payments_transactions`, `exceptions_dict`) but stored different
shapes: Python caches post-processed view-model tuples
(e.g. `(members, sorted_months)`), Go caches raw sheet rows.
Whichever backend wrote last poisoned the cache for the other,
producing `ValueError: too many values to unpack (expected 2,
got 68)` on Python's /adults after the Go side populated the
file with 68 raw CSV rows.

This breaks the M5.4 `make parity` workflow that requires both
backends running side-by-side.

Fix: change Go's default to `tmp/go` so the two cache trees
never overlap.  `CACHE_DIR` env var override still works.
`os.MkdirAll` already handles creating the new subdirectory on
first write.

Recovery for users with poisoned `tmp/`: hit /flush-cache on
the Python side once after pulling, then restart the Go server.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-07 23:06:34 +02:00
423c3e2a4b Merge pull request 'feat(py): M5.3 — Python /api/* shadow endpoints' (#18) from feat/go-m5-3-python-api-shadow into main
All checks were successful
Deploy to K8s / deploy (push) Successful in 7s
Reviewed-on: #18
2026-05-07 20:42:54 +00:00
13 changed files with 467 additions and 13 deletions

View File

@@ -1,5 +1,41 @@
# Changelog
## 2026-05-08 00:26 CEST — fix(py): parity coercions — amount/message types + junior '?' sticky
- `scripts/match_payments.py`: added `get_float` helper — non-numeric `amount` values (e.g. `"---"` placeholder rows) now coerce to `0.0` matching Go's `parseFloat` behaviour; `message` field now goes through `get_str` so numeric cell values (bank references) are emitted as strings, matching Go's `fmt.Sprint`.
- `scripts/views.py`: junior month cell `"?"` text is now sticky across exception overrides. Previously `reconcile` replaced `expected` with the exception amount before the view builder ran, silently turning `"?"` into `"-"` when the override was 0. Fixed by deriving `is_unknown` from `original_expected == "?"` instead of `expected == "?"`. Also aligned tooltip guard: only show Received/Expected for non-unknown months (or when paid > 0), matching Go's `!md.IsUnknown` condition.
## 2026-05-07 23:51 CEST — feat(py): M5.4 fix #2 — add vs and sync_id to payments tx projection
- `scripts/match_payments.py`: `fetch_sheet_data` now reads `VS` and `Sync ID` columns and includes `vs`/`sync_id` keys in every tx dict. Previously only 9 columns were projected, causing `make parity` to report extra `vs`/`sync_id` fields on every raw payment row emitted by the Go backend. Values flow through `group_payments_by_person``_unwrap_view_model_for_api` to `raw_payments` (adults/juniors) and `grouped_payments` (payments) automatically.
- `tests/test_app.py`: updated `/api/*` mock fixtures to include `vs`/`sync_id` keys for realism.
- **Cache note**: after deploying, hit `POST /flush-cache` once so the in-process cache is cleared and the next request picks up the new column lookups.
## 2026-05-07 23:37 CEST — fix(go): accept single-digit day/month in attendance date headers
- `go/internal/services/membership/sources.go`: `parseDates` now uses Go time formats `2.1.2006` and `1/2/2006` (single-digit reference forms, which accept both padded and unpadded inputs) instead of `02.01.2006` and `01/02/2006`. The Czech attendance sheet headers contain dates like `1.6.2026`, `23.3.2026`, `6.4.2026` — Go silently dropped those columns under the strict zero-padded format, while Python's `strptime("%d.%m.%Y")` accepted them. Effect was a missing `2026-06` month entirely on `/api/juniors` plus undercounted attendance for any month with single-digit columns; both surfaced as diffs in `make parity`.
- `sources_test.go::TestParseDates_SingleDigitDayMonth` added as a regression guard covering both Czech and US format flavours with and without leading zeros.
## 2026-05-07 23:17 CEST — fix(go): pass raw value to FormatDate so numeric serial-day dates format
- `go/internal/services/membership/sources.go`: transaction-row parser now passes `row[idxDate]` directly to `matching.FormatDate` (via a new `getRaw` helper) instead of stringifying first via `getVal`. The Sheets API returns numeric serial-day values as `float64` for date-formatted cells; pre-stringifying them defeated `FormatDate`'s `case float64:` dispatch, causing all numeric dates to leak through as `"46147"` style strings instead of `"2026-05-05"`.
- Surfaced by `make parity` (M5.4): every `transactions[].date` field on `/api/adults` and `/api/juniors` differed between Python and Go.
- `sources_test.go::TestLoadTransactions` extended with a numeric-serial-day row covering the regression.
## 2026-05-07 23:05 CEST — fix(go): default CacheDir to `tmp/go` to avoid Python collision
- `go/internal/config/config.go`: `CacheDir` default changed from `tmp` to `tmp/go`. Override via `CACHE_DIR` env var still works.
- Why: both backends used `tmp/<key>_cache.json` with the same keys (`attendance_regular`, `attendance_juniors`, `payments_transactions`, `exceptions_dict`) but different shapes — Python caches post-processed view-model tuples, Go caches raw rows. Whichever wrote last poisoned the cache; running both in parallel produced `ValueError: too many values to unpack (expected 2, got 68)` on Python's `/adults` after the Go server populated `attendance_regular_cache.json` with raw CSV rows.
- After upgrading: stop the Go server, hit `/flush-cache` on the Python side once (rewrites `tmp/*.json` with correct shapes), then restart `make web-go` — it will use `tmp/go/` going forward. Required for the M5.4 `make parity` workflow which assumes both backends run side-by-side.
## 2026-05-07 22:55 CEST — feat(go): M5.4 — parity diff binary + `make parity`
- `go/cmd/parity/main.go`: new standalone binary that GETs `/api/adults`, `/api/juniors`, `/api/payments` from both Python (:5001) and Go (:8080) backends, scrubs an allowlist (`render_time.total`), and prints `cmp.Diff` for any remaining differences. Exits 0 on full match, 1 on diffs, 2 on fetch/parse errors — CI-friendly for M7.2. `/api/version` is excluded by design (returns binary identity — tag/commit/build_date — which differs between independently built backends); still accessible via `make parity ARGS="-route /api/version"`.
- `go/cmd/parity/scrub_test.go`: 4 unit tests covering top-level delete, nested delete, missing path, and non-map parent.
- `go/go.mod`: `github.com/google/go-cmp` promoted to direct dependency.
- `Makefile`: `parity` target added (`.PHONY`, help, `cd go && go run ./cmd/parity`).
- `docs/plans/2026-05-07-2254-m5-4-parity-binary.md`: plan archived.
## 2026-05-07 22:37 CEST — feat(py): M5.3 — Python /api/* shadow endpoints
- `app.py`: four new JSON routes (`/api/version`, `/api/adults`, `/api/juniors`, `/api/payments`) mirroring the Go `/api/*` handlers; `_unwrap_view_model_for_api()` helper expands pre-serialised JSON strings and renames `month_labels_json``month_labels`, `raw_payments_json``raw_payments` to match Go wire contract.

View File

@@ -1,4 +1,4 @@
.PHONY: help fees match web web-py web-debug web-go go-build go-test go-test-all go-parity go-run go-sync-debug go-lint capture-fixtures image run sync sync-2026 test test-v docs
.PHONY: help fees match web web-py web-debug web-go go-build go-test go-test-all go-parity go-run go-sync-debug go-lint capture-fixtures parity image run sync sync-2026 test test-v docs
export PYTHONPATH := scripts:$(PYTHONPATH)
VENV := .venv
@@ -29,6 +29,7 @@ help:
@echo " make go-lint - Run golangci-lint on Go code"
@echo " make go-sync-debug [DAYS=N] - Dry-run Go sync with Fio debug logs and txn table (default DAYS=30)"
@echo " make capture-fixtures - Regenerate parity fixture corpus from live Python"
@echo " make parity - Diff /api/* between web-py (:5001) and web-go (:8080); both must be running"
@echo " make image - Build Python OCI container image"
@echo " make run - Run the built Python Docker image locally"
@echo " make sync - Sync Fio transactions to Google Sheets"
@@ -102,6 +103,9 @@ go-lint:
web-go: go-build
./$(GO_BIN) server
parity:
cd $(GO_SRC) && go run ./cmd/parity $(ARGS)
image:
docker build -t fuj-management:latest \
--build-arg GIT_TAG=$$(git describe --tags --always 2>/dev/null || echo "untagged") \

View File

@@ -2,9 +2,9 @@
Companion to [2026-05-03-2349-go-backend-rewrite.md](2026-05-03-2349-go-backend-rewrite.md).
**Current milestone:** M4IO layer behind interfaces ✅
**Current milestone:** M5JSON-only `/api/...` routes ✅
**Started:** 2026-05-04
**Last updated:** 2026-05-07
**Last updated:** 2026-05-07 (M5.4)
## How to use
@@ -100,7 +100,7 @@ Goal: byte-equal JSON between Python and Go for every route. This is the parity
- [x] **M5.1** Hand-author Go structs for `/api/adults`, `/api/juniors`, `/api/payments`, `/api/version` with explicit `json:` tags matching Python keys; emit JSON Schemas via `github.com/invopop/jsonschema` to `tests/fixtures/api-schema/` — `f253e3f`
- [x] **M5.2** Implement Go handlers for `/api/*` routes composing `services/*` results into the JSON structs — `7d48e8f`
- [x] **M5.3** Add Python `/api/X` shadow endpoints in [app.py](app.py): `jsonify(view_model_dict)` — no transformation — `40e4a9e`
- [ ] **M5.4** Build `cmd/parity/main.go`: hits both backends' `/api/X`, normalizes allowlist (`render_time.total`, `build_meta`), prints `cmp.Diff`. Add `make parity` target
- [x] **M5.4** Build `cmd/parity/main.go`: hits both backends' `/api/X`, normalizes allowlist (`render_time.total`, `build_meta`), prints `cmp.Diff`. Add `make parity` target
**Gate:** For each route, `make parity` reports zero non-allowlisted diffs across the M3 fixture corpus.
@@ -154,6 +154,7 @@ Goal: Go is the one true backend.
(Add entries as you go. Format: `YYYY-MM-DD — short note`.)
- 2026-05-07 — `/api/version` excluded from parity diff by design. Each binary's tag/commit/build_date is identity, not a data contract — diffing it would always flag a diff between independently built backends. Route remains reachable via `make parity ARGS="-route /api/version"` for manual inspection.
- 2026-05-04 — Plan approved. Versioning policy: latest stable for Go and all libs at the time M1 starts. Frontends explicitly allowed to diverge between Python and Go; only the JSON API contract is parity-locked. No reverse proxy — both backends run on different ports via `make web-py` / `make web-go`.
- 2026-05-07 — M4 complete. Chose fakes-only unit tests (no live integration tests) and CSV-via-public-URL for attendance (no Sheets API auth required for read-only). golangci-lint gofumpt extra-rules differ slightly from standalone gofumpt; used `golangci-lint run --fix --enable-only gofumpt` to auto-resolve formatting.
- 2026-05-04 — M1 complete. Dockerfile base changed from `distroless/static:nonroot` → `alpine:3` for debuggability (can tighten later). CLI dispatcher uses stdlib `flag`; module path `fuj-management/go`. golangci-lint v1 embedded gofumpt merges all imports into one group (no stdlib/local split) — accepted as the project style.

View File

@@ -0,0 +1,152 @@
# M5.4 — Parity diff binary (`cmd/parity`) + `make parity`
## Context
Per [docs/plans/2026-05-03-2349-go-backend-rewrite-progress.md:103](docs/plans/2026-05-03-2349-go-backend-rewrite-progress.md#L103), M5.4 is the next milestone in the Go rewrite. M5.1M5.3 already landed:
- M5.1: hand-authored Go response structs at [go/internal/web/api/](go/internal/web/api/) + JSON Schemas in [go/tests/fixtures/api-schema/](go/tests/fixtures/api-schema/).
- M5.2: Go `/api/version|adults|juniors|payments` handlers in [go/internal/web/api/handler.go](go/internal/web/api/handler.go).
- M5.3: Python shadow endpoints in [app.py:157-224](app.py#L157-L224) using `_unwrap_view_model_for_api` for the same JSON shape.
What's missing: a tool that proves the two backends actually agree on the wire. The M5 gate says "byte-equal JSON between Python and Go for every route." Without a diffing tool, drift between the two implementations slips in silently and we can't gate further milestones (M6 frontend, M7 watch period) on parity.
This task delivers the **parity contract enforcer**: a Go binary that fetches `/api/*` from both backends, scrubs an allowlist of expected diffs, and prints `cmp.Diff` for everything else. Both backends read the same live Google Sheets — that shared state is what makes parity meaningful, no scenario fixtures needed.
## Scope
In scope:
- New binary `go/cmd/parity/main.go` plus a small support package for the allowlist scrubber.
- A unit test for the scrubber.
- New `make parity` target.
- `go-cmp` promoted to a direct dependency.
Out of scope (explicitly):
- CI integration / nightly job — that's M7.2.
- Fixture-driven offline parity — pure-fn parity already runs via `make go-parity` (M3).
- Hooking `make parity` into `make go-test-all` — leave it manual since it requires two live servers.
## Approach
### 1. New binary: `go/cmd/parity/main.go`
Standalone binary (mirrors task spec — not a subcommand of `fuj`). Stdlib `flag`, no third-party CLI lib.
**Flags:**
- `-py` — Python base URL, default `http://localhost:5001`
- `-go` — Go base URL, default `http://localhost:8080`
- `-route` — optional single route to diff (e.g. `/api/adults`); empty means iterate all four
- `-timeout` — per-request timeout, default `30s` (sheet fetches are slow on cold cache)
**Routes**, hard-coded:
```go
var routes = []string{"/api/version", "/api/adults", "/api/juniors", "/api/payments"}
```
All `GET`, no query params (verified in [app.py:157-224](app.py#L157-L224) and [go/internal/web/api/handler.go:27-68](go/internal/web/api/handler.go#L27-L68)).
**Per route, the binary:**
1. `GET py+route` and `GET go+route` (sequential — keep it simple; total wall time ~12s on warm cache).
2. Verify both return HTTP 200; on non-200, print body and mark route as ERROR.
3. Decode each body into `map[string]any` (NOT into `api.AdultsResponse` — using the typed struct would silently drop unknown Python-side keys, defeating the diff. Map gives `cmp.Diff` clean dotted-path field names for free.).
4. Run `scrub(m, allowlist)` on both decoded maps.
5. `diff := cmp.Diff(pyMap, goMap, cmpopts.EquateEmpty())`.
6. If `diff != ""`, print `=== /api/X ===` header followed by the diff; track for exit code.
**Exit codes:**
- `0` — all routes match (or the only diffs were under the allowlist).
- `1` — at least one route had a non-allowlisted diff.
- `2` — at least one route failed to fetch / parse (HTTP error, timeout, non-JSON body).
This makes the binary CI-friendly when M7.2 lands.
**Output**: human-readable to stdout — header per route, then "OK" or the diff. Final summary line: `parity: 4/4 routes match` or `parity: 2/4 match, 1 diff, 1 error`.
### 2. Allowlist scrubber
Live in same package (`main`). Keep it tiny — no need for a separate sub-package.
```go
var defaultAllowlist = []string{"render_time.total", "build_meta"}
// scrub walks m and deletes any key whose dotted path matches an allowlist entry.
// "render_time.total" deletes m["render_time"]["total"] only (preserves render_time.breakdown).
// "build_meta" (no dot) deletes m["build_meta"] entirely.
func scrub(m map[string]any, paths []string) { ... }
```
Today these fields **don't appear in the JSON** — they're Jinja template-only ([app.py:91-110](app.py#L91-L110)) and the Go side only logs render time via [go/internal/web/middleware/timer.go](go/internal/web/middleware/timer.go). So today the scrub is a no-op. The implementation is forward-compatible insurance: if someone later adds either field to a JSON response on one side only, the parity binary already tolerates it.
The implementation note in [go/internal/web/middleware/timer.go](go/internal/web/middleware/timer.go) ("the elapsed value maps to render_time.total in the M5 JSON allowlist") confirms this is the intended design.
### 3. Unit test
Add `go/cmd/parity/scrub_test.go` with a couple of cases:
- Scrubbing top-level key (`build_meta`) — key removed; siblings untouched.
- Scrubbing nested key (`render_time.total`) — only `total` removed; `breakdown` preserved.
- Path that doesn't exist — no-op, no error.
- Map without the parent key — no-op (don't panic).
Runs as part of normal `make go-test` (no `-tags` needed).
### 4. Makefile target
Add to [Makefile](Makefile):
```make
parity:
cd $(GO_SRC) && go run ./cmd/parity $(ARGS)
```
Add `parity` to the `.PHONY` line at [Makefile:1](Makefile#L1) and a `help` entry like:
```
@echo " make parity - Diff /api/* between web-py (:5001) and web-go (:8080); both must be running"
```
Don't depend on `go-build``go run` compiles ad-hoc, and parity is interactive enough that the slight rebuild cost doesn't matter.
### 5. Dependency
`github.com/google/go-cmp v0.7.0` is already a transitive dep ([go/go.sum:24-25](go/go.sum#L24-L25)) but not in `go.mod`'s `require` block. After adding the import, run `go mod tidy` inside `go/` to promote it to direct. No new external deps.
## Files to add / modify
**Add:**
- [go/cmd/parity/main.go](go/cmd/parity/main.go) — flags, route loop, fetch+scrub+diff, exit codes
- [go/cmd/parity/scrub_test.go](go/cmd/parity/scrub_test.go) — unit tests for the scrubber
**Modify:**
- [Makefile](Makefile) — `.PHONY`, help block, `parity` target
- [go/go.mod](go/go.mod) — promote `go-cmp` to direct (via `go mod tidy`)
- [go/go.sum](go/go.sum) — likely unchanged (already pinned)
- [docs/plans/2026-05-03-2349-go-backend-rewrite-progress.md](docs/plans/2026-05-03-2349-go-backend-rewrite-progress.md) — tick M5.4, add commit SHA
- [CHANGELOG.md](CHANGELOG.md) — entry per project convention
**Per [CLAUDE.md](CLAUDE.md) plans convention:** copy this plan to `docs/plans/2026-05-07-HHMM-m5-4-parity-binary.md` (timestamp via `date "+%Y-%m-%d-%H%M"`) before opening the MR, since plan files are committed for posterity.
## Existing utilities to reuse
- `cmp.Diff` + `cmpopts.EquateEmpty()` from `github.com/google/go-cmp/cmp` (already in go.sum).
- Stdlib `net/http`, `encoding/json`, `flag` — sufficient; no need to pull a CLI framework.
- No existing scrubber to reuse — the one in [scripts/scrub_fixtures.py](scripts/scrub_fixtures.py) operates on the *capture* path and renames PII; the parity scrubber is a different concern (path-based deletion of expected diffs).
## Verification
Manual smoke test (the binary is meant to be run against live servers):
1. **Sanity / build:** `cd go && go build ./cmd/parity && go test ./cmd/parity/...` — compiles + scrubber unit test passes.
2. **Both backends up:**
- Terminal A: `make web-py`
- Terminal B: `make web-go`
- Terminal C: `make parity`
Expected: `parity: 4/4 routes match`, exit 0.
3. **Negative test:** add a literal extra field to one Go handler temporarily (e.g. `"diagnostic": "test"` in `VersionResponse`), rebuild, re-run `make parity`. Expected: non-zero exit, diff shown for `/api/version`. Revert.
4. **Allowlist test:** in a unit test (or by manually constructing a payload with `render_time.total` injected), confirm the scrubber removes it before the diff stage.
5. **CHANGELOG** entry added; progress tracker ticked with the merge commit SHA.
Per [CLAUDE.md](CLAUDE.md) branching policy: this is a feature, so work happens on `feat/go-m5-4-parity-binary`, push with `-u`, open MR with `tea pr create --base main --head feat/go-m5-4-parity-binary`. User merges in Gitea.
## Notes & risks
- **Cache-warmth dependency:** Both backends cache aggressively. A cold-cache fetch on Python can take 30s+ — hence the configurable timeout. If parity is run immediately after `flush-cache`, expect the first run to be slow.
- **Order-sensitive lists:** `cmp.Diff` on `map[string]any` treats slices as ordered. If either backend ever returns members/transactions in a different order, the diff will flag it — that's a real parity bug, not a false positive, so this is correct behavior. If we hit ordering instability later, fix the source, don't add `cmpopts.SortSlices`.
- **Float formatting:** Both sides go through `encoding/json` (Go) and `jsonify` (Python `json.dumps`). Floats in totals/balances may format differently (`123` vs `123.0`). If this surfaces, the Go side already uses typed structs with `int` where appropriate — investigate at the source rather than allowlisting.

133
go/cmd/parity/main.go Normal file
View File

@@ -0,0 +1,133 @@
package main
import (
"encoding/json"
"flag"
"fmt"
"io"
"net/http"
"os"
"strings"
"time"
"github.com/google/go-cmp/cmp"
"github.com/google/go-cmp/cmp/cmpopts"
)
// /api/version is intentionally excluded — it returns each binary's own build
// identity (tag/commit/build_date), which differs by design between independently
// built backends. Pass -route /api/version to inspect it manually.
var allRoutes = []string{"/api/adults", "/api/juniors", "/api/payments"}
// defaultAllowlist holds dotted key paths to strip before diffing.
// render_time.total is forward-compatible insurance: today it lives in the Jinja
// template context only (app.py inject_render_time) and is logged via
// middleware/timer.go on the Go side, so it isn't in any JSON response. If either
// side ever surfaces it under a render_time envelope, the scrubber handles it.
var defaultAllowlist = []string{"render_time.total"}
func main() {
pyURL := flag.String("py", "http://localhost:5001", "Python backend base URL")
goURL := flag.String("go", "http://localhost:8080", "Go backend base URL")
route := flag.String("route", "", "single route to diff, e.g. /api/adults (default: all)")
timeout := flag.Duration("timeout", 30*time.Second, "per-request HTTP timeout")
flag.Parse()
client := &http.Client{Timeout: *timeout}
targets := allRoutes
if *route != "" {
targets = []string{*route}
}
matched, diffs, errs := 0, 0, 0
for _, r := range targets {
pyMap, err1 := fetch(client, *pyURL+r)
goMap, err2 := fetch(client, *goURL+r)
if err1 != nil || err2 != nil {
fmt.Printf("=== %s ===\n", r)
if err1 != nil {
fmt.Printf("ERROR (py): %v\n", err1)
}
if err2 != nil {
fmt.Printf("ERROR (go): %v\n", err2)
}
fmt.Println()
errs++
continue
}
scrub(pyMap, defaultAllowlist)
scrub(goMap, defaultAllowlist)
diff := cmp.Diff(pyMap, goMap, cmpopts.EquateEmpty())
if diff == "" {
fmt.Printf("=== %s ===\nOK\n\n", r)
matched++
} else {
fmt.Printf("=== %s ===\n%s\n", r, diff)
diffs++
}
}
total := len(targets)
fmt.Printf("parity: %d/%d routes match", matched, total)
if diffs > 0 {
fmt.Printf(", %d diff", diffs)
if diffs > 1 {
fmt.Print("s")
}
}
if errs > 0 {
fmt.Printf(", %d error", errs)
if errs > 1 {
fmt.Print("s")
}
}
fmt.Println()
if errs > 0 {
os.Exit(2)
}
if diffs > 0 {
os.Exit(1)
}
}
func fetch(client *http.Client, url string) (map[string]any, error) {
resp, err := client.Get(url) //nolint:noctx
if err != nil {
return nil, err
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("read body: %w", err)
}
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
}
var m map[string]any
if err := json.Unmarshal(body, &m); err != nil {
return nil, fmt.Errorf("decode JSON: %w", err)
}
return m, nil
}
// scrub removes keys from m whose dotted paths appear in paths.
// A bare segment (no dot) deletes a top-level key.
// A two-segment path "parent.child" deletes child from m["parent"] if it is a map.
func scrub(m map[string]any, paths []string) {
for _, path := range paths {
parts := strings.SplitN(path, ".", 2)
if len(parts) == 1 {
delete(m, parts[0])
} else {
if child, ok := m[parts[0]].(map[string]any); ok {
delete(child, parts[1])
}
}
}
}

View File

@@ -0,0 +1,57 @@
package main
import "testing"
func TestScrubTopLevel(t *testing.T) {
m := map[string]any{
"build_meta": map[string]any{"tag": "v1"},
"other": "keep",
}
scrub(m, []string{"build_meta"})
if _, ok := m["build_meta"]; ok {
t.Error("expected build_meta to be removed")
}
if m["other"] != "keep" {
t.Error("expected other to be preserved")
}
}
func TestScrubNested(t *testing.T) {
m := map[string]any{
"render_time": map[string]any{
"total": "0.123",
"breakdown": "fetch:0.1s",
},
"other": "keep",
}
scrub(m, []string{"render_time.total"})
rt, ok := m["render_time"].(map[string]any)
if !ok {
t.Fatal("render_time should still be present")
}
if _, ok := rt["total"]; ok {
t.Error("expected render_time.total to be removed")
}
if rt["breakdown"] != "fetch:0.1s" {
t.Error("expected render_time.breakdown to be preserved")
}
if m["other"] != "keep" {
t.Error("expected other to be preserved")
}
}
func TestScrubMissingPath(t *testing.T) {
m := map[string]any{"foo": "bar"}
scrub(m, []string{"nonexistent", "render_time.total"})
if m["foo"] != "bar" {
t.Error("expected foo to be preserved")
}
}
func TestScrubNestedParentNotMap(t *testing.T) {
m := map[string]any{"render_time": "not-a-map"}
scrub(m, []string{"render_time.total"})
if m["render_time"] != "not-a-map" {
t.Error("expected render_time to be unchanged when it is not a map")
}
}

View File

@@ -3,6 +3,7 @@ module fuj-management/go
go 1.26.1
require (
github.com/google/go-cmp v0.7.0
github.com/invopop/jsonschema v0.14.0
golang.org/x/net v0.53.0
golang.org/x/text v0.36.0

View File

@@ -50,7 +50,7 @@ func Load() Config {
return Config{
CredentialsPath: env("CREDENTIALS_PATH", ".secret/fuj-management-bot-credentials.json"),
BankAccount: env("BANK_ACCOUNT", "CZ8520100000002800359168"),
CacheDir: env("CACHE_DIR", "tmp"),
CacheDir: env("CACHE_DIR", "tmp/go"),
CacheTTL: envDuration("CACHE_TTL_SECONDS", 300),
CacheAPICheckTTL: envDuration("CACHE_API_CHECK_TTL_SECONDS", 300),
DriveTimeout: envDuration("DRIVE_TIMEOUT_SECONDS", 10),

View File

@@ -142,7 +142,13 @@ func parseDates(header []string) []struct {
}
var dt time.Time
var err error
for _, fmt_ := range []string{"02.01.2006", "01/02/2006"} {
// Use the unpadded reference forms ("2.1" and "1/2"): Go's time.Parse
// accepts both single-digit and zero-padded inputs against them, so
// "1.6.2026", "01.06.2026", "23.3.2026" all parse. Czech sheet authors
// drop the leading zero on dates ≤ 9 — Python's strptime is lenient
// the same way; the previous "02.01.2006" form silently dropped those
// columns and undercounted attendance.
for _, fmt_ := range []string{"2.1.2006", "1/2/2006"} {
dt, err = time.Parse(fmt_, raw)
if err == nil {
break
@@ -394,9 +400,19 @@ func parseTransactionRows(rows [][]any) ([]reconcile.Transaction, error) {
return fmt.Sprint(row[i])
}
// getRaw returns row[i] without stringifying — needed for FormatDate to
// dispatch on the underlying numeric type (Sheets returns serial-day
// numbers as float64). Stringifying first defeats that dispatch.
getRaw := func(row []any, i int) any {
if i < 0 || i >= len(row) {
return nil
}
return row[i]
}
var txns []reconcile.Transaction
for _, row := range rows[1:] {
dateStr := matching.FormatDate(getVal(row, idxDate))
dateStr := matching.FormatDate(getRaw(row, idxDate))
amountRaw := row[idxAmount]
if idxAmount < 0 || idxAmount >= len(row) {
amountRaw = ""

View File

@@ -114,12 +114,15 @@ func TestLoadJuniors(t *testing.T) {
func TestLoadTransactions(t *testing.T) {
// Sheets fake keyed by "<spreadsheetID>/<range>" — use the real constant.
// Row 1 uses a pre-formatted date string; row 2 uses the numeric Sheets
// serial-day form (float64) — the API returns either depending on cell
// formatting, and FormatDate must handle both.
paymentsKey := config.PaymentsSheetID + "/A1:Z"
sh := &sheets.Fake{Values: map[string][][]any{
paymentsKey: {
{"Date", "Amount", "manual fix", "Person", "Purpose", "Inferred Amount", "Sender", "VS", "Message", "Bank ID", "Sync ID"},
{"2026-04-01", 700.0, "", "Alice", "2026-04", "", "Alice Bank", "", "fee", "", "abc"},
{"2026-05-01", 500.0, "", "", "", "", "Bob Bank", "", "platba", "", "def"},
{46147.0, 500.0, "", "", "", "", "Bob Bank", "", "platba", "", "def"}, // 46147 serial-day = 2026-05-05
},
}}
s := buildSources(t, &attendance.Fake{}, sh)
@@ -137,6 +140,12 @@ func TestLoadTransactions(t *testing.T) {
if txns[0].Amount != 700 {
t.Errorf("txn[0].Amount: want 700, got %v", txns[0].Amount)
}
if txns[0].Date != "2026-04-01" {
t.Errorf("txn[0].Date: want 2026-04-01, got %q", txns[0].Date)
}
if txns[1].Date != "2026-05-05" {
t.Errorf("txn[1].Date (numeric serial-day): want 2026-05-05, got %q", txns[1].Date)
}
}
func TestLoadExceptions(t *testing.T) {
@@ -165,6 +174,28 @@ func TestLoadExceptions(t *testing.T) {
}
}
// TestParseDates_SingleDigitDayMonth covers the regression where Go's strict
// "02.01.2006" format dropped header cells written without leading zeros
// (e.g. "1.6.2026", "23.3.2026"), causing attendance undercounts and missing
// months on the /api/juniors response. Czech sheet authors drop the zero
// pad freely; Python's strptime tolerates it, so the parsers must match.
func TestParseDates_SingleDigitDayMonth(t *testing.T) {
// Czech form ("DD.MM.YYYY", with leading zeros optional) is the primary
// path. The "M/D/YYYY" fallback mirrors Python's %m/%d/%Y secondary
// strptime branch — month-first, day-second.
header := []string{"Jméno", "Tier", "", "01.06.2026", "1.6.2026", "23.3.2026", "6.4.2026", "01/02/2026", "1/2/2026"}
got := parseDates(header)
want := []string{"2026-06", "2026-06", "2026-03", "2026-04", "2026-01", "2026-01"}
if len(got) != len(want) {
t.Fatalf("parseDates: got %d entries, want %d (%v)", len(got), len(want), got)
}
for i, e := range got {
if e.month != want[i] {
t.Errorf("parseDates[%d].month = %q, want %q (raw=%q)", i, e.month, want[i], header[e.col])
}
}
}
// TTL smoke test: second call within TTL must not call fetch again.
func TestLoadAdults_CacheHit(t *testing.T) {
dir := t.TempDir()

View File

@@ -236,6 +236,8 @@ def fetch_sheet_data(spreadsheet_id: str, credentials_path: str) -> list[dict]:
idx_sender = get_col_index("Sender")
idx_message = get_col_index("Message")
idx_bank_id = get_col_index("Bank ID")
idx_vs = get_col_index("VS")
idx_sync_id = get_col_index("Sync ID")
required = {"Date": idx_date, "Amount": idx_amount, "Person": idx_person, "Purpose": idx_purpose}
missing = [name for name, idx in required.items() if idx == -1]
@@ -247,16 +249,33 @@ def fetch_sheet_data(spreadsheet_id: str, credentials_path: str) -> list[dict]:
def get_val(idx):
return row[idx] if idx != -1 and idx < len(row) else ""
def get_str(idx):
v = get_val(idx)
if isinstance(v, float) and v.is_integer():
return str(int(v))
return str(v)
def get_float(idx):
v = get_val(idx)
if isinstance(v, (int, float)):
return float(v)
try:
return float(str(v).strip())
except (ValueError, TypeError):
return 0.0
tx = {
"date": format_date(get_val(idx_date)),
"amount": get_val(idx_amount),
"amount": get_float(idx_amount),
"manual_fix": get_val(idx_manual),
"person": get_val(idx_person),
"purpose": get_val(idx_purpose),
"inferred_amount": get_val(idx_inferred_amount),
"sender": get_val(idx_sender),
"message": get_val(idx_message),
"vs": get_str(idx_vs),
"message": get_str(idx_message),
"bank_id": get_val(idx_bank_id),
"sync_id": get_val(idx_sync_id),
}
transactions.append(tx)

View File

@@ -310,8 +310,9 @@ def build_juniors_view_model(
cell_text = "-"
amount_to_pay = 0
if expected == "?" or (isinstance(expected, int) and expected > 0):
if expected == "?":
is_unknown = original_expected == "?"
if is_unknown or (isinstance(expected, int) and expected > 0):
if is_unknown:
status = "empty"
cell_text = f"?{count_str}"
elif paid >= expected:
@@ -339,7 +340,7 @@ def build_juniors_view_model(
status = "surplus"
cell_text = f"PAID {paid}"
if (isinstance(expected, int) and expected > 0) or paid > 0:
if (not is_unknown and isinstance(expected, int) and expected > 0) or paid > 0:
tooltip = f"Received: {paid}, Expected: {expected}"
else:
tooltip = ""

View File

@@ -129,6 +129,7 @@ class TestWebApp(unittest.TestCase):
'date': '2026-01-01', 'amount': 750, 'person': 'Test Member',
'purpose': '2026-01', 'message': 'test payment',
'sender': 'External Bank User', 'inferred_amount': 750,
'vs': '', 'sync_id': 'abc123',
}]
response = self.client.get('/api/adults')
self.assertEqual(response.status_code, 200)
@@ -155,6 +156,7 @@ class TestWebApp(unittest.TestCase):
mock_fetch_sheet.return_value = [{
'date': '2026-01-15', 'amount': 500, 'person': 'Junior One',
'purpose': '2026-01', 'message': '', 'sender': 'Parent', 'inferred_amount': 500,
'vs': '', 'sync_id': 'def456',
}]
response = self.client.get('/api/juniors')
self.assertEqual(response.status_code, 200)
@@ -172,6 +174,7 @@ class TestWebApp(unittest.TestCase):
mock_fetch_sheet.return_value = [{
'date': '2026-01-01', 'amount': 750, 'person': 'Test Member',
'purpose': '2026-01', 'message': 'test', 'sender': 'Someone',
'vs': '', 'sync_id': 'ghi789',
}]
response = self.client.get('/api/payments')
self.assertEqual(response.status_code, 200)