fuj-management/docs/plans/2026-05-03-2349-go-backend-rewrite.md

# Plan: Full Go rewrite of the Python/Flask backend

## Context

The current Flask app ([app.py](app.py) + [scripts/](scripts/), ~2400 LOC of
Python) handles attendance-based fee calculation, Fio bank sync, payment
reconciliation, and a server-rendered dashboard. The user wants a full
rewrite in Go with two goals:

1. **Quality Go code** as the primary outcome — idiomatic stdlib-first
   design, strong typing, proper layering. The Python codebase grew
   organically and mixes domain logic, IO, and HTTP concerns.
2. **Feature-parity certainty** — no behavioural drift between the Python
   and Go versions on anything that touches money. Reconciliation is real
   money; silent divergence is unacceptable.

**Switchable runtime**: both backends run on different TCP ports, started
independently via Makefile targets (`make web-py` on :5001, `make web-go` on
:8080). The user opens whichever they want in a browser. No reverse proxy,
no traffic-splitting, no shared frontend constraint — just two services
that read the same Google Sheets and the same `tmp/` cache.

**Frontends are allowed to diverge.** The Go web layer is designed cleanly
in its own right rather than as a byte-compatible Jinja port. Both backends
expose a JSON API (`/api/...`) with an identical contract — that's what
parity testing locks down. Rendered HTML and inline JS can be different.

## Versioning policy

- **Go**: latest stable release at project start. Pin in `go.mod` via the
  `go` directive (e.g. `go 1.X`) and use the matching `golang:1.X` builder
  image. Bump on each new minor as it lands stable.
- **Go libraries**: latest stable for every dependency in `go.mod`; run
  `go get -u ./... && go mod tidy` at the start and quarterly thereafter.
- **Python deps** (during the parallel-run period): keep
  [pyproject.toml](pyproject.toml) on its current versions to avoid
  destabilizing the parity baseline; bump only after Python retires.
- **Base images**: `golang:latest-stable` builder → `gcr.io/distroless/static:latest`
  runtime, both pinned by digest in CI for reproducibility.
- **CI runners**: latest stable Linux image on Gitea Actions.

The plan does not hardcode specific version numbers below — implementation
picks current-stable at the time M1 starts.

## Approach summary

- **Three-layer Go architecture**: pure domain (no IO) → IO clients (behind
  interfaces, easily faked) → HTTP/services (composition).
- **Capture-then-port**: dump current Python outputs as JSON fixtures, port
  Go function-by-function, assert byte-equality with `cmp.Diff`.
- **JSON contract is the spec, not the templates.** Each Python route gets
  an `/api/X` shadow that returns the dict already passed to the template.
  Go defines typed structs matching that shape; both sides validate against
  generated JSON Schema.
- **Money is integer CZK**: existing fees are integer CZK (750/200/500);
  keep it that way to avoid float drift in reconcile allocation. Where
  Sheets returns floats, parse and round at the boundary.
- **Frontend rewrite, not port**: Go uses `html/template` with cleanly
  organized templates and JS extracted into static files served via
  `embed.FS`. Same UX (filterable table, member-detail modal, QR launcher)
  but designed natively, no Jinja-port baggage.

## Go project layout

`go/` lives at the repo root alongside `scripts/` and `templates/` so both
backends share the same git history during migration.

```
go/
  cmd/
    fuj/main.go                    # single binary, subcommands: server | fees | sync | infer | reconcile
    parity/main.go                 # diff tool: hits both backends' /api/X, prints JSON diff
  internal/
    domain/                        # pure, no IO, no net/*
      czech/                       # normalize, parse_month_references
      fees/                        # calculate_fee, calculate_junior_fee, "?" sentinel type
      money/                       # parse_czk_amount, format helpers
      reconcile/                   # reconcile() + Ledger, MemberResult types
      matching/                    # _build_name_variants, match_members, infer_transaction_details
      synch/                       # generate_sync_id (pure hash)
    io/                            # IO behind interfaces, all impls have an in-memory fake
      sheets/                      # SheetsClient + Google impl + fake
      drive/                       # DriveClient for modifiedTime
      fio/                         # FioClient: API JSON impl + transparent-page HTML scraper
      cache/                       # FileCache with modifiedTime gating + two TTL knobs
    services/                      # composition layer; pure + IO, no HTTP
      attendance/                  # GetMembersWithFees, GetJuniorMembersWithFees
      payments/                    # FetchTransactions, FetchExceptions, BuildView
      banksync/                    # SyncToSheets, InferPayments (write ops)
    web/
      handlers/                    # one file per route family
      view/                        # HTML view-model structs (per route)
      api/                         # JSON view-model structs (the parity-locked contract)
      templates/                   # *.tmpl, embed.FS — designed natively, not a Jinja port
      static/                      # js/*.js, css/*.css served via embed.FS
      middleware/                  # request timer, recovery, slog
    config/                        # mirrors scripts/config.py (env loading)
    qr/                            # SPD string builder + PNG via go-qrcode
  tests/
    fixtures/                      # JSON fixtures captured from Python (PII-scrubbed)
    parity/                        # Go-side characterization tests (replay fixtures)
  build/Dockerfile                 # multi-stage: latest-stable golang builder → distroless static
  go.mod
```

## Library choices

All on latest stable as per the versioning policy above.

| Concern | Pick | Rationale |
|---|---|---|
| HTTP routing | `net/http` ServeMux | 8 static routes; no need for chi/gin given modern stdlib pattern matching |
| Templates | `html/template` | Auto-escaping; native Go feel |
| Static assets | `embed.FS` | Single binary, no loose files |
| Sheets/Drive | `google.golang.org/api/{sheets/v4,drive/v3}` + `option` | Official client; service-account auth via `option.WithCredentialsFile` |
| OAuth | `golang.org/x/oauth2/google` (token only; drop installed-app flow + pickle) | Production already uses service accounts |
| QR PNG | `github.com/skip2/go-qrcode` | Mature, byte-stable PNG output |
| NFKD | `golang.org/x/text/unicode/norm` + `unicode.IsMark` | Direct equivalent of `unicodedata.normalize("NFKD", ...)` |
| HTML scrape | `golang.org/x/net/html` token visitor | Counts `<table class="table">` to target the second one |
| CSV | `encoding/csv` (stdlib) | Match for Python `csv.reader` |
| Logging | `log/slog` (stdlib) | Honors `LOG_LEVEL` env |
| Diff/testing | `testing` + `github.com/google/go-cmp/cmp` | Readable `cmp.Diff` for parity assertions |
| Lint | `golangci-lint` (govet, staticcheck, errcheck, gofumpt, unused) | Standard quality gate |

## Migration sequencing — eight milestones with hard gates

**M1 — Skeleton + tooling.** Create `go/` tree, `go.mod` (latest stable
Go), Makefile targets (`go-build`, `go-test`, `go-run`, `web-go`),
`golangci-lint` config. `cmd/fuj server` prints a hello + version and
listens on :8080.
*Gate:* `make go-build` succeeds; `make web-go` serves a "hello" page on
:8080 in parallel with `make web` on :5001; lint clean.

**M2 — Pure-domain helpers, port leaf-first.** Order:
[czech_utils.py](scripts/czech_utils.py) `normalize` → `parse_month_references` →
[attendance.py](scripts/attendance.py) `calculate_fee`/`calculate_junior_fee` →
[infer_payments.py](scripts/infer_payments.py) `parse_czk_amount` →
[sync_fio_to_sheets.py](scripts/sync_fio_to_sheets.py) `generate_sync_id` →
[match_payments.py](scripts/match_payments.py) helpers (`_build_name_variants`,
`match_members`, `infer_transaction_details`, `format_date`) → `reconcile`.
Each gets a Go unit test plus a parity test driven by JSON fixtures from M3.
Also: `fuj fees` and `fuj reconcile` subcommands wired up (pure-domain CLIs).
*Gate:* All ported helpers pass parity tests.

**M3 — Fixture capture + characterization framework.** Build
`scripts/capture_fixtures.py` (Python helper that prints function results as
JSON to stdout — user pipes to disk) and `scripts/scrub_fixtures.py`
(replaces member names with deterministic pseudonyms `Member_<8hex>`,
scrambles sender/account/VS/bank_id while preserving structural
relationships, dates, amounts, exception keys). Capture ~10 reconcile
fixtures spanning every code path: greedy, proportional with float
remainder, even-split fallback, out-of-window credit, exception override,
`other:` purpose, junior `"?"`, comma-separated multi-person, multi-month
range, unmatched.
*Gate:* `tests/fixtures/` populated and committed; M2 parity tests green.

**M4 — IO layer behind interfaces.** Implement Sheets/Drive/Fio clients
matching Python return shapes. Drop the OAuth+pickle path entirely (service
account only). All clients have in-memory fakes for tests. Wire `fuj sync`
and `fuj infer` subcommands.
*Gate:* `go test -tags=integration ./internal/io/...` round-trips against a
test sheet (separate from prod); default-tag tests use fakes.

**M5 — JSON-only `/api/...` routes.** Add 8 Go route handlers that return
JSON. Add symmetric `/api/X` shadow endpoints in [app.py](app.py) that
`jsonify` the existing view-model dict (no transformation).
*Gate:* For each route, `cmd/parity` asserts
`cmp.Diff(python.json, go.json) == ""` modulo allowlist
(`render_time.total`, `build_meta`).

**M6 — Go-native HTML frontend.** Design Go templates cleanly (not a Jinja
port). Extract JS from inline into `internal/web/static/js/*.js` served via
`embed.FS`. Vanilla JS, no framework — same UX as Python (sortable table,
member-detail modal, name filter, month range filter, QR launcher) but
organized as proper modules. Templates render the JSON API response into
HTML; frontend JS fetches additional data from `/api/X` for the modal
rather than embedding `member_data` in `<script>`.
*Gate:* Browser smoke test of all routes on :8080 covers: name filter,
month filter, modal opens with correct months/transactions/exceptions, QR
modal renders, navigation between adults/juniors/payments works.

**M7 — Parallel-running watch period.** Both `make web-py` and `make web-go`
running locally (and in production via two containers on different ports).
Daily/manual `cmd/parity` runs catch any JSON drift. The user verifies the
Go UI matches what they expect feature-by-feature against the Python UI.
Run 1–2 weeks.
*Gate:* Zero non-allowlisted JSON diffs over 7 consecutive days, including
a sync-bank execution, a flush, and an attendance update. User sign-off
that the Go UI is feature-complete.

**M8 — Cutover + Python retirement.** Switch the bookmarked URL / docs to
the Go port. Keep Python container running but unrouted (or stopped) for
1 week as rollback. Then delete [app.py](app.py), [scripts/](scripts/),
the Python `Dockerfile`, and the Python tests. Update
[CLAUDE.md](CLAUDE.md) to reflect the Go-only state.
*Gate:* Two consecutive months of Go-only operation including end-of-month
settlement.

## CLI port (decided: port as Go subcommands)

Single Go binary `fuj` with subcommands replacing the existing Makefile
targets. Each reuses the domain layer directly:

| Old | New | Backed by | Milestone |
|---|---|---|---|
| `make fees` | `fuj fees` | `domain/fees` + `services/attendance` | M2 |
| `make reconcile` | `fuj reconcile` | `domain/reconcile` | M2 |
| `make sync-2026` | `fuj sync --year=2026` | `services/banksync.SyncToSheets` | M4 |
| `make infer` | `fuj infer [--dry-run]` | `services/banksync.InferPayments` | M4 |
| `make web` (py) | stays as Python `make web-py` until M8 | — | — |
| `make web-go` | `fuj server` | `web/handlers` | M1 |

Makefile targets get rewritten to invoke `./bin/fuj <subcommand>` once each
is ported. The Python `make` targets for already-ported commands stay as
`make X-py` aliases until M8, so you can run either side for cross-checks.

## JSON API contract strategy

**Go-defines, Python-conforms** with a 1-step bootstrap:

1. Run Python locally and dump `result["members"]`, `formatted_results`,
   `monthly_totals`, etc., to JSON. This is the spec.
2. Hand-author Go structs with explicit `json:` tags matching exact Python
   keys (`total_balance`, `original_expected`, `attendance_count` — no
   reliance on default lowercasing).
3. Generate `tests/fixtures/api-schema/*.schema.json` from the Go structs
   using `github.com/invopop/jsonschema`. Commit them.
4. Add a Python-side schema validator running in CI against the new
   `/api/X` responses.

**Two known-tricky shapes:**

- Junior `expected: int | "?"` →
  ```go
  type Expected struct{ Value int; Unknown bool }
  // MarshalJSON emits 42 or "?"
  ```
  Same for `original_expected`.
- Tuple dict keys `(normalize(name), normalize(period))` for exceptions —
  internal only, never crosses JSON. Use
  `map[ExceptionKey]Exception` with `ExceptionKey struct{ Name, Period string }`.

## Characterization test harness — two tiers

(HTML rendering parity dropped: frontends are intentionally different.)

**Tier 1 — Pure-function parity** (fast, every commit). Fixtures at
`tests/fixtures/pure/<func>/<case>.json` containing `{input, output}`,
captured once via `scripts/capture_fixtures.py`. Go test reads each, calls
the ported function, asserts deep equality with `cmp.Diff`. Functions in
scope: `normalize`, `parse_month_references`, `parse_czk_amount`,
`parse_czech_amount`, `parse_czech_date`, `format_date`,
`_build_name_variants`, `match_members`, `infer_transaction_details`,
`generate_sync_id`, `calculate_fee`, `calculate_junior_fee`, `reconcile`.

**Tier 2 — JSON API parity** (medium, on PR + nightly). `cmd/parity/main.go`
hits both `:5001/api/X` and `:8080/api/X` with a fixture-seeded `tmp/`
cache, normalizes volatile fields (`render_time`, build metadata), asserts
byte-equality. Cache freezing: pre-populate `tmp/*_cache.json` from
scrubbed snapshots so both backends read identical data.

**PII scrubbing** is mandatory ([CLAUDE.md](CLAUDE.md): "Member data must
never be committed"). `scripts/scrub_fixtures.py` produces deterministic
pseudonyms preserving uniqueness and structural relationships. Only
scrubbed fixtures land in `tests/fixtures/`; raw `tmp/*.json` stays
gitignored.

## Side-by-side runtime

Two services on different ports, started independently. No reverse proxy.

```
make web-py    # Python on :5001 (existing target, perhaps renamed from `make web`)
make web-go    # Go on :8080
```

Both read the same Google Sheets and write to the same `tmp/` cache
directory. The user opens `localhost:5001` or `localhost:8080` directly to
A/B compare.

**Cache directory coordination**: both backends use `tmp/`. Go writes via
`os.WriteFile` to `tmp/<key>_cache.json.tmp` then `os.Rename` (atomic on
Linux). Python's writes are pre-existing-non-atomic; accept until Python
retires.

**Sync coordination**: `/sync-bank` is non-idempotent under concurrency.
Both backends `flock` on `tmp/sync.lock`; Go uses `syscall.Flock`. (In
practice the user is unlikely to trigger sync from both UIs at once, but
the lock is cheap insurance.)

**Production deployment**: keep the existing Python container; add a Go
container in `docker-compose.yml` exposed on a different port. After M8,
remove the Python service.

## CI/CD

Currently zero test CI ([.gitea/workflows/build.yaml](.gitea/workflows/build.yaml)
only does `docker build`/`push`). Add `/.gitea/workflows/test.yml`:

```yaml
jobs:
  python-tests:    # fix M3 broken-test references first
    - uv sync && pytest tests/
  go-tests:
    - cd go && go test -race ./...
    - cd go && golangci-lint run
  parity-pure:     # Tier 1
    - cd go && go test -tags=parity ./tests/parity/...
```

Branch protection: `python-tests`, `go-tests`, `parity-pure` block merge.
Tier-2 parity runs nightly via `parity-nightly.yml` (boots both servers
via docker-compose with seeded caches, replays a fixed transaction script,
fails on any non-allowlisted diff).

A new Go `build/Dockerfile` (multi-stage: latest-stable `golang` builder →
`gcr.io/distroless/static:latest`, both pinned by digest) mirrors the
existing Python build job and produces a single static binary image.

## Risk register (top 4)

(Template auto-escape divergence dropped: irrelevant when frontends differ.)

1. **Sync ID hash drift** — HIGH/HIGH. Python builds the SHA-256 input by
   `str()`-ing each field then `.lower()`-ing the joined string;
   `str(750.0) == "750.0"`, `str(750) == "750"`. If Sheets API returns
   floats in Python but Go unmarshals as int, `750` vs `750.0` → different
   hash → duplicate rows. *Mitigation:* dedicated parity test with ~50
   real-row fixtures; if Go can't reproduce Python's float string format,
   normalize at the boundary (round to 2 decimals, format with explicit
   precision).
2. **Float allocation in `reconcile()` proportional phase** — HIGH/MEDIUM.
   Python's "last month absorbs remainder" depends on dict iteration order;
   Go map iteration is randomized. *Mitigation:* always iterate
   `sorted_months` explicitly in Go, never the map. Lock the distribution
   with a parity test on (300, 300, 150) months × 751-CZK payment.
3. **NFKD edge cases** — MEDIUM/MEDIUM. Python `unicodedata` and Go
   `golang.org/x/text` use the same algorithm but can differ on niche
   compatibility decompositions if `x/text` is older than CPython's tables.
   *Mitigation:* parity test with every distinct character ever observed in
   member names; pin `x/text` version explicitly.
4. **Czech month parser semantics** — MEDIUM/MEDIUM. Wrap-around year
   inference (`if start_m > end_m and m >= start_m: year = default_year - 1`)
   plus the "month >= 10 → previous year" heuristic are easy to mis-port.
   *Mitigation:* port table and algorithm verbatim line-for-line; parity
   test with ~30 real `message`-field fixture strings.

## Cutover plan

Simpler without a proxy in the middle:

1. After M7's 7-day clean window + user sign-off, treat Go as primary.
   Update bookmarks, docs, `make web` to point at Go.
2. Keep `make web-py` available for 1-week rollback. Run both containers
   in production but only point users at the Go one.
3. Watch 2 weeks including a month-end settlement on Go-only.
4. Decommission Python: remove from `docker-compose.yml`, delete
   [app.py](app.py) and [scripts/](scripts/), update
   [CLAUDE.md](CLAUDE.md). Keep image tagged `python-final` in registry as
   a 6-month rollback option.

**Retirement criteria:** zero parity-diff incidents in last 30 days, zero
rollbacks, two month-end settlements completed Go-only, manual
reconciliation review against `python-final` signed off.

## Critical files

- [scripts/match_payments.py](scripts/match_payments.py) — `reconcile()` is
  the single most load-bearing function (~200 lines of allocation logic)
  that must port byte-equivalently.
- [scripts/czech_utils.py](scripts/czech_utils.py) — `normalize` and
  `parse_month_references` underpin every member/month match across the
  system. 45 Czech month declensions, range wrap-around, year inference.
- [app.py](app.py) — defines the 8-route HTTP surface and view-model
  shapes. The spec for the Go web layer's JSON API.
- [scripts/sync_fio_to_sheets.py](scripts/sync_fio_to_sheets.py) —
  `generate_sync_id` defines the dedup contract against existing rows in
  the live sheet. Any drift creates duplicates.
- [scripts/attendance.py](scripts/attendance.py) — fee math + merged-month
  logic + junior `"?"` sentinel.
- [scripts/cache_utils.py](scripts/cache_utils.py) — Drive `modifiedTime`
  gating + two-TTL fallback that must be reproduced for shared-cache
  safety.
- [templates/adults.html](templates/adults.html) — read for the JSON shape
  the existing inline JS consumes (`member_data`); the Go frontend doesn't
  have to mirror the template, but the JSON contract derived from this
  page's data injection is the parity spec.

## Verification

End-to-end checks per milestone:

- **M1**: `make go-build && ./bin/fuj server --help` prints subcommand
  list. `make web-go` serves :8080 in parallel with `make web-py` on :5001.
- **M2-M3**: `cd go && go test -tags=parity ./tests/parity/pure/...` green.
  Spot-check: feed a known Czech-message string through both
  `parse_month_references` implementations, diff outputs.
- **M4**: `go test -tags=integration ./internal/io/sheets/...` round-trips
  against a test sheet (separate from prod).
- **M5**: `curl localhost:5001/api/adults | jq -S . > py.json && curl
  localhost:8080/api/adults | jq -S . > go.json && diff py.json go.json` —
  empty diff modulo allowlist.
- **M6**: Browser open `localhost:8080/adults`, click a member row, modal
  opens with all months / transactions / exceptions correctly populated.
  Same on `/juniors`. Click a Pay button → QR loads. Name filter and month
  range filter work.
- **M7**: Run `cd go && ./bin/parity --base http://localhost:5001
  --candidate http://localhost:8080 --routes adults,juniors,payments`
  daily for 7 days, zero non-allowlisted diffs. User confirms Go UI is
  feature-complete vs Python UI side-by-side.
- **M8**: `make web-py` removed from Makefile; `make web` points at Go;
  manual end-of-month settlement on Go matches the prior month's
  Python-produced report.

## Open questions / forks the user can override at review

- **Frontend JS organization in M6**: default is vanilla JS in separate
  files via `embed.FS`. If the user wants HTMX, Alpine.js, or a small
  framework, raise it before M6.
- **CI host**: Gitea Actions assumed (matches existing
  [.gitea/workflows/build.yaml](.gitea/workflows/build.yaml)).
- **Test sheet for M4 integration tests**: would need provisioning.
  Confirm whether to use a copy of the production sheet (PII!) or a
  synthetic one seeded by the fixture-capture process.