Jan Novak
723152cdad
All checks were successful
Deploy to K8s / deploy (push) Successful in 11s
The transaction-row parser in services/membership/sources.go used a helper (`getVal`) that did `fmt.Sprint(row[i])` before passing to `matching.FormatDate`. The Sheets API returns date-formatted cells as `float64` (Sheets serial-day numbers); pre-stringifying defeated `FormatDate`'s `case float64:` dispatch, so values like 46147 leaked through unchanged as the string "46147" instead of being converted to "2026-05-05". Surfaced by `make parity` (M5.4) — every `transactions[].date` on /api/adults and /api/juniors differed between Python and Go. Python side passes the raw value through directly (`isinstance(val, (int, float))` in scripts/match_payments.py format_date), so it was always correct. Added a `getRaw` helper that returns row[i] without stringifying; only the date column needs it. Extended TestLoadTransactions with a numeric-serial-day row to lock in the regression. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
175 lines
17 KiB
Markdown
175 lines
17 KiB
Markdown
# Changelog
|
||
|
||
## 2026-05-07 23:17 CEST — fix(go): pass raw value to FormatDate so numeric serial-day dates format
|
||
|
||
- `go/internal/services/membership/sources.go`: transaction-row parser now passes `row[idxDate]` directly to `matching.FormatDate` (via a new `getRaw` helper) instead of stringifying first via `getVal`. The Sheets API returns numeric serial-day values as `float64` for date-formatted cells; pre-stringifying them defeated `FormatDate`'s `case float64:` dispatch, causing all numeric dates to leak through as `"46147"` style strings instead of `"2026-05-05"`.
|
||
- Surfaced by `make parity` (M5.4): every `transactions[].date` field on `/api/adults` and `/api/juniors` differed between Python and Go.
|
||
- `sources_test.go::TestLoadTransactions` extended with a numeric-serial-day row covering the regression.
|
||
|
||
## 2026-05-07 23:05 CEST — fix(go): default CacheDir to `tmp/go` to avoid Python collision
|
||
|
||
- `go/internal/config/config.go`: `CacheDir` default changed from `tmp` to `tmp/go`. Override via `CACHE_DIR` env var still works.
|
||
- Why: both backends used `tmp/<key>_cache.json` with the same keys (`attendance_regular`, `attendance_juniors`, `payments_transactions`, `exceptions_dict`) but different shapes — Python caches post-processed view-model tuples, Go caches raw rows. Whichever wrote last poisoned the cache; running both in parallel produced `ValueError: too many values to unpack (expected 2, got 68)` on Python's `/adults` after the Go server populated `attendance_regular_cache.json` with raw CSV rows.
|
||
- After upgrading: stop the Go server, hit `/flush-cache` on the Python side once (rewrites `tmp/*.json` with correct shapes), then restart `make web-go` — it will use `tmp/go/` going forward. Required for the M5.4 `make parity` workflow which assumes both backends run side-by-side.
|
||
|
||
## 2026-05-07 22:37 CEST — feat(py): M5.3 — Python /api/* shadow endpoints
|
||
|
||
- `app.py`: four new JSON routes (`/api/version`, `/api/adults`, `/api/juniors`, `/api/payments`) mirroring the Go `/api/*` handlers; `_unwrap_view_model_for_api()` helper expands pre-serialised JSON strings and renames `month_labels_json` → `month_labels`, `raw_payments_json` → `raw_payments` to match Go wire contract.
|
||
- `tests/test_app.py`: four new smoke tests asserting top-level key sets and that unwrapped fields are objects (not strings).
|
||
|
||
## 2026-05-07 20:13 CEST — feat(go): M5.2 — HTTP handlers for /api/adults, /api/juniors, /api/payments, /api/version
|
||
|
||
- `web/api/handler.go`: `Handler` struct + `ServeAdults`, `ServeJuniors`, `ServePayments`, `ServeVersion` using `membership.Sources`.
|
||
- `web/api/build_{adults,juniors,payments,common}.go`: ports of `scripts/views.py` view-model builders; `buildJuniorMemberRow` handles `"?"` sentinel, `:NJ,MA` breakdown, unknown-month skip.
|
||
- Extended `reconcile.FeeData`/`MonthData` with `IsUnknown`, `JuniorAttendance`, `AdultAttendance`; `Transaction` with `ManualFix`, `VS`, `BankID`, `SyncID`.
|
||
- `sources.go` exports `AdultMergedMonths`/`JuniorMergedMonths`; parses new FeeData and transaction columns.
|
||
- `web/server.go` + `cmd/fuj/main.go` wired to register `/api/*` routes.
|
||
- PR #17.
|
||
|
||
## 2026-05-07 17:37 CEST — feat(go): M5.1 — /api/* wire types + JSON Schemas
|
||
|
||
- New `go/internal/web/api/` package: `AdultsResponse`, `JuniorsResponse`, `PaymentsResponse`, `VersionResponse` with explicit `json:` tags matching Python view-model keys.
|
||
- `Expected{Value int; Unknown bool}` custom `MarshalJSON` emits integer or `"?"` for junior single-attendance months.
|
||
- `schemagen_test.go` golden-tests four JSON Schemas committed to `go/tests/fixtures/api-schema/`. `JSONSchema()` on `Expected` lives in the test file — production binary has no jsonschema dep.
|
||
- PR #16.
|
||
|
||
## 2026-05-07 15:26 CEST — refactor(app): extract view-model builders into scripts/views.py
|
||
|
||
- Pulled ~350 lines of inline per-row computation out of `adults_view`, `juniors_view`, and `payments` into three pure functions in `scripts/views.py`: `build_adults_view_model`, `build_juniors_view_model`, `build_payments_view_model`.
|
||
- Moved `get_month_labels`, `group_payments_by_person`, `adapt_junior_members` from `app.py` to `scripts/views.py`. Route handlers now ~25 lines each.
|
||
- Hotfixed missing `import re` that caused 500 on `/qr` after the refactor.
|
||
- No behaviour change; all 27 tests pass. Prep for `/api/*` shadow endpoints (M5).
|
||
|
||
## 2026-05-07 14:13 CEST — feat(go): --print-fio-table + Fio debug logging + date parser fix
|
||
|
||
- Added `--print-fio-table` flag to `fuj sync --dry-run`: prints an aligned table of every Fio transaction in the window with `STATUS=NEW/DUP`, using `text/tabwriter`. Key files: `go/internal/services/banksync/fio_table.go`, `sync.go`, `cmd/fuj/main.go`.
|
||
- Added `LOG_LEVEL=DEBUG` debug logging on the Fio fetch path: client variant selected, full GET URL (token redacted on API path), HTTP status, body bytes, and per-parse drop-reason counters (`raw_rows`, `kept`, `dropped_bad_date`, `dropped_nonpositive_amount`). Key files: `go/internal/io/fio/{client,api,transparent}.go`.
|
||
- Fixed `parseCzechDate` to accept `DD.MM.YY` (2-digit year) in addition to the 4-digit variant — Fio's transparent page now serves this format. Key file: `go/internal/io/fio/transparent.go`.
|
||
- Added `make go-sync-debug [DAYS=N]` Makefile target (default 30 days).
|
||
|
||
## 2026-05-07 10:32 CEST — feat(go): --dry-run for fuj sync
|
||
|
||
- `SyncOpts.DryRun bool` added; when true, `SyncToSheets` prints planned writes (`would write header row`, `would append date=… amount=… sender=…`, `would sort by date`) and returns without calling `WriteHeader`, `AppendValues`, or `SortByDateColumn`.
|
||
- `fuj sync --dry-run` flag wired in `cmd/fuj/main.go`; mirrors existing `fuj infer --dry-run` behaviour.
|
||
- `TestSyncToSheets_DryRun` added to banksync test suite.
|
||
|
||
## 2026-05-07 01:03 CEST — feat(go/M4): IO layer behind interfaces
|
||
|
||
- `go/internal/io/attendance`: CSV-over-public-URL client + `Fake` for both adult and junior tabs.
|
||
- `go/internal/io/drive`: thin Drive v3 wrapper for `modifiedTime` reads + `Fake`.
|
||
- `go/internal/io/sheets`: Sheets v4 client (`GetValues`, `AppendValues`, `BatchUpdateValues`, `WriteHeader`, `SortByDateColumn`) + `Fake` with call-capture for assertions.
|
||
- `go/internal/io/cache`: Drive-modifiedTime-gated `FileCache` with two TTL knobs, atomic writes, and generic `Get[T]`; Python-compatible JSON format; `Flush()` support.
|
||
- `go/internal/io/fio`: `Client` interface backed by Fio REST API (`apiClient`) and HTML-scraper (`transparentClient`); `Fake` for tests. Fixtures in `testdata/`.
|
||
- `go/internal/services/membership/sources.go`: `NewSources` wires attendance CSV + Sheets + cache into `LoadAdults`, `LoadJuniors`, `LoadTransactions`, `LoadExceptions`. Includes Czech month/merged-month parsing logic.
|
||
- `go/internal/services/banksync`: `SyncToSheets` (dedup via SHA-256 Sync ID, optional sort) and `InferPayments` (name-match + `[?]` review prefix, dry-run) — fully tested with fakes.
|
||
- `go/cmd/fuj/main.go`: `sync` and `infer` subcommands wired to real clients; `fees` and `reconcile` now use real `NewSources`.
|
||
- All packages lint-clean (golangci-lint v1.64.8, gofumpt extra-rules).
|
||
|
||
## 2026-05-06 23:25 CEST — feat(go/M3): fixture capture + parity test framework
|
||
|
||
- `scripts/capture_fixtures.py`: dispatcher CLI that calls each ported function with seeded inputs and emits captured output as JSON fixtures.
|
||
- `scripts/scrub_fixtures.py`: deterministic PII scrubber (SHA-256 pseudonyms, digit-preserving account/VS hashes, name-sweep in free text).
|
||
- `scripts/_fixture_seeds.py`: handcrafted seed registry for all 10 pure functions + 10 reconcile branch-coverage cases.
|
||
- 98 fixture files committed under `go/tests/fixtures/pure/<func>/` and `go/tests/fixtures/reconcile/`; all PII-free.
|
||
- `go/tests/parity/parityio.go`: shared loader with generic `LoadDir`/`RunAll` helpers and typed `In`/`Out` structs for all 10 functions.
|
||
- 11 parity test packages under `//go:build parity`: 10 pure-function tests + bespoke reconcile test with per-cell float tolerance.
|
||
- Makefile: `go-parity`, `go-test-all`, `capture-fixtures` targets.
|
||
- `go/tests/fixtures/README.md`: refresh workflow, PII audit guide, adding-a-fixture steps.
|
||
|
||
## 2026-05-06 17:49 CEST — feat(go/M2.11-12): wire fuj fees + fuj reconcile subcommands
|
||
|
||
- New `go/internal/services/membership` package: `AttendanceLoader`, `TransactionLoader`, `ExceptionLoader` interfaces, a stub (`NewStubSources`) that returns `ErrIOPending`, and `FeesReport` / `ReconcileReport` orchestration functions backed by real `domain/fees` + `domain/reconcile` logic.
|
||
- Text formatters `printFeesTable` / `printReconcileReport` port the output of `calculate_fees.py` and `match_payments.py print_report` verbatim.
|
||
- `cmd/fuj/main.go`: `fuj fees` and `fuj reconcile` subcommands now dispatch properly; `fuj sync` / `fuj infer` retain the [M4] placeholder.
|
||
- Both subcommands exit 1 with a clean `"io layer not yet wired up; lands in milestone M4"` message until real Sheets loaders are injected in M4.
|
||
- 13 unit tests covering stubs, all formatter branches (OK/partial/UNPAID/dash cells, credits, debts, unmatched, review annotation), and orchestration wiring via fake loaders.
|
||
|
||
## 2026-05-06 16:38 CEST — fix: include juniors in payment-inference roster
|
||
|
||
- `scripts/infer_payments.py`: union adults + junior rosters so junior-only members are visible to the matcher.
|
||
- Root cause: `get_members_with_fees()` reads only the adults sheet; junior-only kids like Jáchym Kubík were absent from `member_names`, causing the exact-match short-circuit to never fire and a different adult sharing the first name to win via fuzzy review.
|
||
- Two regression tests added to `tests/test_match_members.py`.
|
||
|
||
## 2026-05-06 16:05 CEST — feat(go/M2.10): port domain/reconcile.Reconcile
|
||
|
||
- New `go/internal/domain/reconcile` package porting the three-phase payment allocation from `scripts/match_payments.py reconcile()`.
|
||
- 12 unit tests covering all Python test cases plus Go-only extras (diacritics tolerance, `[?]` stripping, `other:` purpose, out-of-window credit, inference fallback, unmatched, no-transaction guard).
|
||
|
||
## 2026-05-06 13:18 CEST — feat(go/M2.7-2.9): port domain/matching package
|
||
|
||
- New `go/internal/domain/matching` package porting three helpers from `scripts/match_payments.py`.
|
||
- `BuildNameVariants` — extracts normalized ASCII search variants from a member name, including nickname (from parens) and separate first/last; filters variants shorter than 3 chars; `variants[0]` is always the full normalized base name.
|
||
- `MatchMembers` — finds members in free text with `"auto"` or `"review"` confidence; exact-name short-circuit prevents nickname substrings (e.g. `tov`) from matching inside surnames (e.g. `ottova`).
|
||
- `FormatDate` — normalizes Google Sheets date values: handles nil, empty, int/float64 serial-days since 1899-12-30 (supports fractional serials), pre-formatted `YYYY-MM-DD` strings, and garbage input — never errors.
|
||
- `InferTransactionDetails` — composes name + month matching over sender/message/user_id; falls back to sender-only member match and date-derived month when text gives no signal.
|
||
- 21 table-driven tests; all expected values verified against live Python on 2026-05-06.
|
||
|
||
## 2026-05-06 12:43 CEST — feat(go/M2.6): port domain/synch.GenerateSyncID
|
||
|
||
- New `go/internal/domain/synch` package with `GenerateSyncID(Transaction) string` ported from `scripts/sync_fio_to_sheets.py` `generate_sync_id`.
|
||
- Byte-stable SHA-256 hash over `date|amount|currency|sender|vs|message|bank_id` (lowercased, UTF-8); `Currency: ""` defaults to `"CZK"` matching the Python missing-key fallback.
|
||
- Key subtlety: Python's `str(float)` emits `"500.0"` for whole-valued floats and switches to scientific notation at `|f| >= 1e16` or `|f| < 1e-4` — replicated in `formatAmount` using `'f'`/`'e'` format selection.
|
||
- 6 table-driven hash tests + 9 `formatAmount` tests; all expected values verified against live Python on 2026-05-06.
|
||
|
||
## 2026-05-06 09:38 CEST — feat(go/M2.5): port domain/money.ParseCZK
|
||
|
||
- New `go/internal/domain/money` package with `ParseCZK(string) (float64, error)` ported from `scripts/infer_payments.py` `parse_czk_amount`.
|
||
- Preserves the Czech-locale heuristic: comma → decimal sep; 2+ dots → thousand seps; single dot → decimal (so `"1.500"` → `1.5`).
|
||
- Returns `(0, ErrInvalidAmount)` on parse failure; callers wanting Python's silent-zero contract use `v, _ := ParseCZK(s)`.
|
||
- 15 table-driven tests plus a silent-zero contract test; all expected values verified against live Python on 2026-05-06.
|
||
|
||
## 2026-05-06 09:24 CEST — feat(go/M2.3+M2.4): port domain/fees.CalculateFee and CalculateJuniorFee
|
||
|
||
- New `go/internal/domain/fees` package with adult and junior fee calculators ported from `scripts/attendance.py`.
|
||
- `CalculateFee(count, monthKey) int` — `0→0`, `1→200`, `2+→AdultFeeMonthlyRate[month]` (fallback 700 CZK).
|
||
- `CalculateJuniorFee(count, monthKey) Expected` — `0→{0}`, `1→{Unknown:true}` (the `"?"` sentinel, now strictly typed), `2+→JuniorFeeMonthlyRate[month]` (fallback 500 CZK).
|
||
- 20 table-driven tests, all verified against live Python; `-race` clean; `golangci-lint` clean.
|
||
|
||
## 2026-05-06 00:07 CEST — feat(go/M2.2): port czech.ParseMonthReferences
|
||
|
||
- `internal/domain/czech.ParseMonthReferences`: three-pass regex (numeric slash, dot, Czech month names) with range wrap-around and `m≥10 → previousYear` heuristic, byte-equivalent to Python.
|
||
- 35 table-driven tests; all expected outputs verified against live Python before locking (addresses risk #4 from the rewrite plan).
|
||
|
||
## 2026-05-05 23:33 CEST — feat(go/M2.1): port czech.Normalize
|
||
|
||
- First M2 pure-domain task: `internal/domain/czech.Normalize` (NFKD + Mn-strip + lowercase), byte-equivalent to Python `czech_utils.normalize`.
|
||
- Adds `golang.org/x/text v0.36.0` as first external Go dependency.
|
||
- 13-case table-driven test, all spot-checked against Python before locking.
|
||
|
||
## 2026-05-04 23:08 CEST — fix: payment inference exact-match short-circuit
|
||
|
||
- `match_members()` now short-circuits on whole-word full-name hits; nickname/partial checks only run when no full name is present.
|
||
- Replaced bare `in` substring checks with `_word_in()` word-boundary regex throughout, closing the class of bugs where a short nickname (e.g. `tov`) matches inside another member's surname (`ottova`).
|
||
- Added `tests/test_match_members.py` (6 cases). Affects `scripts/match_payments.py`.
|
||
|
||
## 2026-05-04 23:08 CEST — feat: lower adult monthly fee to 700 CZK from April 2026
|
||
|
||
- `ADULT_FEE_DEFAULT` reduced from 750 → 700 CZK.
|
||
- `ADULT_FEE_MONTHLY_RATE` now pins Sep 2025 – Feb 2026 at 750 to preserve historical billing; Mar 2026 stays 350; Apr–May 2026 at 700. Affects `scripts/attendance.py`.
|
||
|
||
## 2026-05-04 12:02 CEST — Go rewrite M1: skeleton + tooling
|
||
|
||
- Created `go/` tree with module `fuj-management/go` (Go 1.26).
|
||
- `cmd/fuj`: stdlib-flag subcommand dispatcher; `server` and `version` implemented, stubs for M2/M4 commands.
|
||
- `internal/config`: env loader mirroring `scripts/config.py` (same env var names and defaults).
|
||
- `internal/logging`: slog setup accepting log level from config.
|
||
- `internal/web`: `net/http` ServeMux on `:8080`; `middleware/timer.go` logs method/path/status/ms.
|
||
- `go/build/Dockerfile`: multi-stage (`golang:1.26` → `alpine:3`) producing a static binary image.
|
||
- Makefile: `web` → `web-py` alias; added `web-go`, `go-build`, `go-test`, `go-run`, `go-lint`.
|
||
- `.gitea/workflows/build.yaml`: parallel `build-go` job pushing `<tag>-go` image.
|
||
- Gate: `make go-build`, `make go-lint`, `make go-test`, `curl :8080` all pass.
|
||
|
||
## 2026-05-03 20:37 CEST — Fix Balance column to correctly reflect past-month debt
|
||
|
||
- Balance (and Pay-All) are now computed as `sum(paid − expected)` over past months only, iterating directly over the ledger entries from `reconcile()`.
|
||
- Previously the balance used `total_balance` (which includes current/future-month activity and out-of-window credits) plus a one-sided current-month debt adjustment. Current-month *surplus* leaked through, making the balance appear less negative than the actual past-month debt.
|
||
- Pay-All is now `max(0, −balance)` so the two values are derived from a single source and can never disagree.
|
||
- Affected: `adults_view()` and `juniors_view()` in `app.py`.
|
||
|
||
## 2026-05-03 19:26 CEST — Fee-aware allocation for multi-month payments
|
||
|
||
- `reconcile()` no longer splits a multi-month payment evenly. Allocation is now per-member with two phases: greedy (if amount ≥ total expected, each month gets exactly its expected fee and overflow → credit) and proportional (otherwise distribute by each month's expected). Fixes the case where e.g. 1250 CZK covering 3 months with mixed fees (750/350/150) marked two months red.
|
||
- Out-of-window months keep the previous even-split-to-credit behavior. Fallback to even split when all matched months have `expected = 0` (prepayment before attendance is recorded).
|
||
- Display layer only — no changes to how payments are stored in Google Sheets; `Inferred Amount` still holds the full bank amount.
|
||
- Files: [scripts/match_payments.py](scripts/match_payments.py), [tests/test_reconcile_exceptions.py](tests/test_reconcile_exceptions.py) (6 new test cases).
|