Compare commits

..

6 Commits

Author SHA1 Message Date
10e2e9dc04 Merge pull request 'fix(reconcile): fill earliest month deficit first in multi-month allocations' (#35) from fix/fill-first-multi-month-allocation into main
All checks were successful
Deploy to K8s / deploy (push) Successful in 13s
Build and Push / build (push) Successful in 6s
Build and Push / build-go (push) Successful in 52s
Reviewed-on: #35
2026-05-11 22:01:36 +00:00
8734089223 fix(reconcile): fill earliest month deficit first in multi-month allocations
All checks were successful
Deploy to K8s / deploy (push) Successful in 9s
Replace proportional split with a fill-first loop that allocates
min(remaining, deficit) to each matched month in user-supplied order,
where deficit = expected - already_paid. Prior transactions' contributions
are now properly accounted for, so a second payment on overlapping months
fills only what's still owed instead of splitting proportionally by total
expected. Surplus after all deficits are covered goes to the credit bucket.

Fixes: Matyáš Thér 200+550 showing 566/183 instead of 500/250.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-11 23:59:36 +02:00
aaa876e593 fix(python): parse Fio 2-digit-year dates + add make sync-debug
All checks were successful
Deploy to K8s / deploy (push) Successful in 8s
Build and Push / build (push) Successful in 6s
Build and Push / build-go (push) Successful in 59s
Fio's transparent page now serves recent rows as DD.MM.YY while older
rows stay DD.MM.YYYY. parse_czech_date only knew the 4-digit form so
every recent transfer was silently dropped — make sync-2026 reported
zero new transactions. Adds %d.%m.%y and %d/%m/%y to the format list,
mirroring the Go-side fix from 2026-05-07.

Also adds a Python analog of make go-sync-debug:
- --dry-run skips header write / append / sort and prints "would …" lines
- --print-fio-table prints aligned per-txn table with NEW/DUP status
- make sync-debug [DAYS=N] wrapper (default DAYS=30)
- always-on stderr diagnostics in fio_utils: which fetcher was chosen
  (with FIO_API_TOKEN-unset lag warning) + raw-vs-filtered counts, so
  this class of "scraper drops everything" bug surfaces immediately.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-11 22:56:49 +02:00
f25552eef2 chore: CHANGELOG + tick M6.6 (f6ba85b) and M6.6.1 (4276d7b) in progress tracker
All checks were successful
Deploy to K8s / deploy (push) Successful in 6s
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-08 14:56:57 +02:00
4276d7b915 Merge pull request 'feat(go): M6.6.1 — QR payment popup modal on /adults and /juniors' (#33) from feat/go-m6-6-1-payment-qr-modal into main
All checks were successful
Deploy to K8s / deploy (push) Successful in 9s
Reviewed-on: #33
2026-05-08 12:54:23 +00:00
f6ba85b18f Merge pull request 'feat(go): M6.6 — /qr, /sync-bank, /flush-cache, /version pages' (#32) from feat/go-m6-6-action-pages into main
Some checks failed
Deploy to K8s / deploy (push) Has been cancelled
Reviewed-on: #32
2026-05-08 12:54:16 +00:00
12 changed files with 474 additions and 142 deletions

View File

@@ -1,5 +1,29 @@
# Changelog
## 2026-05-11 23:58 CEST — fix(reconcile): fill earliest month deficit first in multi-month allocations
- Multi-month payment allocation now fills the earliest in-window deficit first and spills
any remainder to later months, accounting for prior transactions' contributions to each month.
Previously a single transaction was split proportionally to each month's total expected fee,
ignoring what earlier transactions had already paid — surfaced by Matyáš Thér's 200+550 case
showing 566/183 instead of 500/250.
- Files: `scripts/match_payments.py`, `go/internal/domain/reconcile/reconcile.go`, tests, parity fixtures.
## 2026-05-11 22:56 CEST — fix(python): parse Fio 2-digit-year dates + add `make sync-debug` dry-run tool
- Fix: `scripts/fio_utils.py` `parse_czech_date` now accepts `DD.MM.YY` / `D.M.YY` in addition to the 4-digit-year variants. Fio's transparent page now mixes both forms in the same response; the 2-digit rows were being silently dropped, which caused `make sync-2026` to miss every recent transfer. Mirrors the Go-side fix from 2026-05-07 (CHANGELOG entry below).
- Added `--dry-run` and `--print-fio-table` flags to `scripts/sync_fio_to_sheets.py`, plus a `make sync-debug [DAYS=N]` Makefile target. Mirrors `make go-sync-debug`: fetches from Fio and dedupes against the sheet, prints `STATUS=NEW/DUP` per transaction, and prints per-row `Dry run: would append …` lines + `would sort by date` instead of touching the sheet.
- Added always-on stderr diagnostics in `scripts/fio_utils.py`: which fetcher was selected (authenticated API vs. transparent-page scraper with `FIO_API_TOKEN`-unset warning), and raw-vs-after-filter transaction counts on both paths — so this class of "scraper drops everything" bug surfaces immediately.
## 2026-05-08 14:55 CEST — feat(go): M6.6.1 — Pay-button QR popup modal
- Restored the Python `showPayQR` in-page modal UX that was lost in M6.6 (Pay buttons were navigating the tab to the raw `/qr` PNG).
- Replaced `<a href="{{qrHref ...}}">Pay</a>` with `<button data-name|amount|month|raw-month>` on `/adults` and `/juniors`; click is handled by a new `static/js/payment-qr.js` IIFE module that opens `#qrModal` with title, account, amount, message, and the QR image.
- Added `#qrModal` markup to both templates; CSS `display:none` / `.active{display:flex}` rules added (content rules were already present from M6.1). `Esc`, `[close]`, and outside-click all dismiss; coexists with the M6.5 member-detail modal.
- Removed the now-dead `qrHref` / `qrHrefAll` template helpers from `render.go`.
- Markup tests in `html_handler_test.go` assert modal IDs, script tag, `data-bank-account`, and that no bare `href="/qr"` links remain.
- Key files: `go/internal/web/static/js/payment-qr.js`, `go/internal/web/templates/adults.tmpl`, `go/internal/web/templates/juniors.tmpl`, `go/internal/web/render.go`, `go/internal/web/static/css/app.css`.
## 2026-05-08 13:57 CEST — feat(go): M6.6 — /qr, /sync-bank, /flush-cache, /version
- Added `GET /qr`: generates Czech QR Platba PNG from SPD payload (account, amount, message query params); ports Python's `qr_code()` handler exactly including account validation, amount clamping, and `*` stripping.

View File

@@ -35,6 +35,7 @@ help:
@echo " make sync - Sync Fio transactions to Google Sheets"
@echo " make sync-2025 - Sync Fio transactions for Q4 2025 (Oct-Dec)"
@echo " make sync-2026 - Sync Fio transactions for the whole year of 2026"
@echo " make sync-debug [DAYS=N] - Dry-run Python sync with Fio diagnostics and txn table (default DAYS=30)"
@echo " make infer - Infer payment details (Person, Purpose, Amount) in the sheet"
@echo " make reconcile - Show balance report using Google Sheets data"
@echo " make venv - Sync virtual environment with pyproject.toml"
@@ -125,6 +126,9 @@ sync-2025: $(PYTHON)
sync-2026: $(PYTHON)
$(PYTHON) scripts/sync_fio_to_sheets.py --credentials .secret/fuj-management-bot-credentials.json --from 2026-01-01 --to 2026-12-31 --sort-by-date
sync-debug: $(PYTHON) ## Dry-run Python sync with Fio diagnostics and txn table (default DAYS=30)
$(PYTHON) scripts/sync_fio_to_sheets.py --credentials .secret/fuj-management-bot-credentials.json --days $(DAYS) --dry-run --print-fio-table
infer: $(PYTHON)
$(PYTHON) scripts/infer_payments.py --credentials $(CREDENTIALS)

View File

@@ -4,7 +4,7 @@ Companion to [2026-05-03-2349-go-backend-rewrite.md](2026-05-03-2349-go-backend-
**Current milestone:** M6 — Go-native HTML frontend
**Started:** 2026-05-04
**Last updated:** 2026-05-08 (M6.4 merged)
**Last updated:** 2026-05-08 (M6.6 + M6.6.1 merged)
## How to use
@@ -115,7 +115,8 @@ Goal: feature-equivalent UX on the Go side, designed cleanly. Not a Jinja port.
- [x] **M6.3** `/juniors` page: same structure + per-month J/A attendance breakdown + `"?"` sentinel rendering — `9564103`
- [x] **M6.4** `/payments` page: grouped-by-person ledger view — `689f1c0`
- [x] **M6.5** Modal JS module (`static/js/member-detail.js`): fetches `/api/adults` (or juniors), renders status/exceptions/transactions on row click; keyboard nav (Esc, ↑/↓) — `e53e238`
- [x] **M6.6** `/qr`, `/sync-bank`, `/flush-cache`, `/version` pages
- [x] **M6.6** `/qr`, `/sync-bank`, `/flush-cache`, `/version` pages — `f6ba85b`
- [x] **M6.6.1** Pay-button QR popup modal (`payment-qr.js`); restores Python `showPayQR` UX lost in M6.6 — `4276d7b`
- [ ] **M6.7** Wire `embed.FS` into handlers; verify single-binary deployment includes all assets
**Gate:** Browser smoke on :8080: all pages render, name+month filters work, modal opens with correct data, QR loads, sync/flush work end-to-end.

View File

@@ -0,0 +1,184 @@
# Fill-first multi-month payment allocation
## Context
Matyáš Thér paid in two transactions:
| # | Amount | Purpose |
|---|--------|---------------------|
| 1 | 200 | `2026-02` |
| 2 | 550 | `2026-02, 2026-03` |
Total 750 = his expected fee for the two months (likely 2026-02 = 500, 2026-03 = 250 — junior tier or exception-adjusted). The app currently shows 2026-02 = **566** paid, 2026-03 = **183** paid. The user wants:
> First use the second payment for the rest of 2026-02 (no more, no less), then put the remainder toward 2026-03.
Both Python ([scripts/match_payments.py](scripts/match_payments.py)) and Go ([go/internal/domain/reconcile/reconcile.go](go/internal/domain/reconcile/reconcile.go)) have the same bug: when a single transaction's `in_window_share` is less than the sum of in-window expected fees, both fall into a **proportional** branch that splits the new transaction across months in proportion to each month's *total* expected fee — never consulting what prior transactions already paid into earlier months.
Trace for txn 2:
- `in_window = [(2026-02, 500), (2026-03, 250)]`, `total_expected = 750`, `in_window_share = 550`.
- `550 < 750` → proportional:
- 02 alloc = `550 × 500 / 750 = 366.67`
- 03 alloc = `550 366.67 = 183.33`
- Combined with txn 1's 200 → 02 = 566.67, 03 = 183.33 → display `566 / 183`. ✓ matches reported numbers.
The fix: replace the greedy + proportional branches with a single **fill-first** loop that iterates `in_window` in user-supplied order (already chronological by convention from [scripts/infer_payments.py:151](scripts/infer_payments.py#L151)) and allocates `min(remaining, max(0, expected paid_so_far))` to each month, with any final surplus going to the credit bucket. This collapses three cases (greedy / proportional / pure overflow) into one and naturally consults the ledger's `paid` field which is already updated by prior transactions in the same reconcile pass.
The even-split branch (`total_expected == 0`, prepayment before fees known) stays untouched — different semantic, folding it in would silently change behavior.
## Changes
### Python — [scripts/match_payments.py:471-498](scripts/match_payments.py#L471-L498)
Replace both `total_expected > 0` branches (current lines 471498) with a single unified loop. Keep lines 466469 above and the even-split fallback below (lines 499510) as-is.
```python
if total_expected > 0:
# Fill-first: iterate in_window in matched_months order (chronological by
# convention), allocate min(remaining, deficit) to each month, where
# deficit accounts for prior transactions already credited to that month.
# Any surplus after all in-window deficits are covered → credit bucket.
remaining = in_window_share
for m, exp in in_window:
paid_so_far = ledger[member_name][m]["paid"]
deficit = max(0.0, float(exp) - paid_so_far)
alloc = min(remaining, deficit)
if alloc <= 0:
continue
ledger[member_name][m]["paid"] += alloc
ledger[member_name][m]["transactions"].append({
"amount": alloc,
"date": tx["date"],
"sender": tx["sender"],
"message": tx["message"],
"confidence": confidence,
})
remaining -= alloc
if remaining > 0:
credits[member_name] = credits.get(member_name, 0) + int(remaining)
else:
# … existing even-split branch (lines 499510) unchanged …
```
Note: skipping the `transactions.append` when `alloc <= 0` (e.g. month already fully paid by a prior txn) avoids zero-amount ghost rows in the per-month transaction list. This is a small UI-visible side effect; before committing, grep tests for assertions on `len(transactions)` per month to confirm nothing relies on the current "one row per (txn, month) regardless of alloc" behavior.
### Go — [go/internal/domain/reconcile/reconcile.go:320-357](go/internal/domain/reconcile/reconcile.go#L320-L357)
Same shape — replace both `totalExpected > 0` branches. Even-split branch (lines 358372) stays.
```go
if totalExpected > 0 {
// Fill-first; see Python reconcile() for rationale.
remaining := inWindowShare
for _, mw := range inWindow {
md := ledger[memberName][mw.month]
deficit := float64(mw.expected) - md.Paid
if deficit < 0 {
deficit = 0
}
alloc := remaining
if deficit < alloc {
alloc = deficit
}
if alloc <= 0 {
continue
}
md.Paid += alloc
md.Transactions = append(md.Transactions, TxEntry{
Amount: alloc,
Date: tx.Date,
Sender: tx.Sender,
Message: tx.Message,
Confidence: string(m.Confidence),
})
ledger[memberName][mw.month] = md
remaining -= alloc
}
if remaining > 0 {
credits[memberName] += int(remaining)
}
} else {
// … existing even-split branch (lines 358372) unchanged …
}
```
### Tests
#### Python — [tests/test_reconcile_exceptions.py](tests/test_reconcile_exceptions.py)
1. **Rewrite `test_proportional_underpayment`** ([line 96](tests/test_reconcile_exceptions.py#L96)) — its current assertions (`paid_02 < 750`, `paid_03 < 350`, `paid_04 < 750`, and 02/04 equal allocation) are incompatible with fill-first. Under fill-first with the same fixture (1250 across `02:750, 03:350, 04:750`):
- 02: `min(1250, 750) = 750` (full) → remaining 500
- 03: `min(500, 350) = 350` (full) → remaining 150
- 04: `min(150, 750) = 150` (partial) → remaining 0
Replace assertions with these exact expected values, rename to `test_underpayment_fills_earliest_first`.
2. **Add `test_fill_first_across_two_transactions`** — the Matyáš regression:
```python
def test_fill_first_across_two_transactions(self):
"""Prior txn fills 02 partially; later txn finishes 02 then spills to 03."""
members = [('Matyáš', 'A', {'2026-02': (500, 2), '2026-03': (250, 1)})]
sorted_months = ['2026-02', '2026-03']
tx1 = _tx('Matyáš', '2026-02', 200)
tx2 = _tx('Matyáš', '2026-02, 2026-03', 550)
result = reconcile(members, sorted_months, [tx1, tx2])
months = result['members']['Matyáš']['months']
self.assertAlmostEqual(months['2026-02']['paid'], 500, places=2)
self.assertAlmostEqual(months['2026-03']['paid'], 250, places=2)
```
3. `test_greedy_exact_match`, `test_greedy_overpayment_goes_to_credit`, `test_single_month_unchanged`, `test_two_members_multi_month` should pass unchanged — fill-first agrees with greedy when payment ≥ total expected.
#### Go — [go/internal/domain/reconcile/reconcile_test.go](go/internal/domain/reconcile/reconcile_test.go)
- Add `TestUnderpaymentFillsEarliestFirst` mirroring the rewritten Python test.
- Add `TestFillFirstAcrossTwoTransactions` mirroring the Matyáš scenario.
#### Parity — [go/tests/parity/reconcile/reconcile_parity_test.go](go/tests/parity/reconcile/reconcile_parity_test.go)
Add a fixture for the Matyáš two-transaction case. Since both implementations change together and Python remains canonical, existing parity fixtures should continue to pass; verify after edits.
### Changelog — [CHANGELOG.md](CHANGELOG.md)
Append top entry (use `date "+%Y-%m-%d %H:%M %Z"` at commit time):
```markdown
## 2026-05-11 23:55 CET — fill-first multi-month payment allocation
- Multi-month payment allocation now fills the earliest in-window deficit first
and spills the remainder to later months, accounting for prior transactions'
contributions. Previously a single transaction was split proportionally to
each month's total expected fee, ignoring earlier payments — surfaced by
Matyáš Thér's two-payment 200+550 case showing 566/183 instead of 500/250.
- scripts/match_payments.py, go/internal/domain/reconcile/reconcile.go, tests.
```
## Critical files
- [scripts/match_payments.py](scripts/match_payments.py) — Python reconcile (canonical)
- [go/internal/domain/reconcile/reconcile.go](go/internal/domain/reconcile/reconcile.go) — Go reconcile (mirrors Python)
- [tests/test_reconcile_exceptions.py](tests/test_reconcile_exceptions.py) — rewrite `test_proportional_underpayment` + add Matyáš test
- [go/internal/domain/reconcile/reconcile_test.go](go/internal/domain/reconcile/reconcile_test.go) — add Go tests
- [go/tests/parity/reconcile/reconcile_parity_test.go](go/tests/parity/reconcile/reconcile_parity_test.go) — add parity fixture
- [CHANGELOG.md](CHANGELOG.md) — top entry
## Verification
1. `make test` — Python unit tests including the rewritten + new fill-first tests.
2. `cd go && go test ./internal/domain/reconcile/... ./tests/parity/reconcile/...` — Go unit + parity tests.
3. `make web` → load `/adults` or `/juniors` (whichever lists Matyáš Thér) → his 2026-02 row should be fully paid (no shortfall), 2026-03 fully paid, no leftover credit.
4. Spot-check one other multi-month-purpose member to make sure fully-covered cases still look right.
## Branch & MR (per CLAUDE.md)
1. `git checkout -b fix/fill-first-multi-month-allocation`
2. Apply edits + tests + CHANGELOG entry.
3. `make test && (cd go && go test ./...)` — both green.
4. Commit: `fix(reconcile): fill earliest month deficit first in multi-month allocations` with `Co-Authored-By` trailer.
5. `git push -u origin fix/fill-first-multi-month-allocation`
6. `tea pr create --title "fix(reconcile): fill earliest month deficit first" --description "<body>" --base main --head fix/fill-first-multi-month-allocation`
7. Print MR URL. Do not merge from CLI.

View File

@@ -115,10 +115,11 @@ type monthExpected struct {
expected int
}
// Reconcile matches transactions to members and months using three allocation phases:
// 1. Greedy: payment ≥ total expected → fill each month exactly; overflow → credit.
// 2. Proportional: payment < total → distribute by each month's share; last absorbs float remainder.
// 3. Even-split fallback: all expected fees are 0 (prepayment) → divide equally.
// Reconcile matches transactions to members and months using two allocation phases:
// 1. Fill-first: iterate matched months in user-supplied order, allocating min(remaining,
// deficit) to each month where deficit = expected already-paid. Surplus → credit.
// Handles both the "greedy" (payment covers all) and "partial" cases in one pass.
// 2. Even-split fallback: all expected fees are 0 (prepayment) → divide equally.
//
// defaultYear seeds czech.ParseMonthReferences in the inference fallback.
// Pass time.Now().Year() in production; pass a fixed year in tests.
@@ -317,34 +318,26 @@ func Reconcile(
totalExpected += mw.expected
}
if totalExpected > 0 && inWindowShare >= float64(totalExpected) {
// Greedy: payment covers all expected fees; overflow → credit
credits[memberName] += int(inWindowShare - float64(totalExpected))
for _, mw := range inWindow {
alloc := float64(mw.expected)
md := ledger[memberName][mw.month]
md.Paid += alloc
md.Transactions = append(md.Transactions, TxEntry{
Amount: alloc,
Date: tx.Date,
Sender: tx.Sender,
Message: tx.Message,
Confidence: string(m.Confidence),
})
ledger[memberName][mw.month] = md
}
} else if totalExpected > 0 {
// Proportional: distribute by each month's share; last month absorbs float remainder
if totalExpected > 0 {
// Fill-first: iterate inWindow in matched-months order (chronological by
// convention), allocating min(remaining, deficit) to each month. Deficit
// is net of what prior transactions already paid, so a second payment on
// the same months correctly fills only what remains due. Any surplus after
// all deficits are covered goes to the credit bucket.
remaining := inWindowShare
for i, mw := range inWindow {
var alloc float64
if i == len(inWindow)-1 {
alloc = remaining
} else {
alloc = inWindowShare * float64(mw.expected) / float64(totalExpected)
}
remaining -= alloc
for _, mw := range inWindow {
md := ledger[memberName][mw.month]
deficit := float64(mw.expected) - md.Paid
if deficit < 0 {
deficit = 0
}
alloc := remaining
if deficit < alloc {
alloc = deficit
}
if alloc <= 0 {
continue
}
md.Paid += alloc
md.Transactions = append(md.Transactions, TxEntry{
Amount: alloc,
@@ -354,6 +347,10 @@ func Reconcile(
Confidence: string(m.Confidence),
})
ledger[memberName][mw.month] = md
remaining -= alloc
}
if remaining > 0 {
credits[memberName] += int(remaining)
}
} else {
// Even-split fallback: prepayment before attendance recorded

View File

@@ -111,36 +111,26 @@ func TestReconcileGreedyOverpaymentGoesToCredit(t *testing.T) {
}
}
func TestReconcileProportionalUnderpayment(t *testing.T) {
func TestReconcileUnderpaymentFillsEarliestFirst(t *testing.T) {
t.Parallel()
members := []Member{{
Name: "Alice", Tier: "A",
Fees: map[string]FeeData{"2026-02": {Expected: 750, Attendance: 3}, "2026-03": {Expected: 350, Attendance: 3}, "2026-04": {Expected: 750, Attendance: 3}},
}}
sortedMonths := []string{"2026-02", "2026-03", "2026-04"}
amount := 1250.0
result := Reconcile(members, sortedMonths, []Transaction{tx("Alice", "2026-02, 2026-03, 2026-04", amount)}, nil, defaultYear)
result := Reconcile(members, sortedMonths, []Transaction{tx("Alice", "2026-02, 2026-03, 2026-04", 1250)}, nil, defaultYear)
months := result.Members["Alice"].Months
paid02 := months["2026-02"].Paid
paid03 := months["2026-03"].Paid
paid04 := months["2026-04"].Paid
if paid02 >= 750 {
t.Errorf("2026-02 should be underpaid, got %f", paid02)
// 02 filled first (750), then 03 (350), then remainder 150 to 04
if math.Abs(months["2026-02"].Paid-750) > 0.01 {
t.Errorf("02: want 750, got %f", months["2026-02"].Paid)
}
if paid03 >= 350 {
t.Errorf("2026-03 should be underpaid, got %f", paid03)
if math.Abs(months["2026-03"].Paid-350) > 0.01 {
t.Errorf("03: want 350, got %f", months["2026-03"].Paid)
}
if paid04 >= 750 {
t.Errorf("2026-04 should be underpaid, got %f", paid04)
}
if math.Abs(paid02+paid03+paid04-amount) > 0.01 {
t.Errorf("sum of paid want %f, got %f", amount, paid02+paid03+paid04)
}
if math.Abs(paid02-paid04) > 0.01 {
t.Errorf("02 and 04 have equal expected, want equal paid: %f vs %f", paid02, paid04)
if math.Abs(months["2026-04"].Paid-150) > 0.01 {
t.Errorf("04: want 150, got %f", months["2026-04"].Paid)
}
}
@@ -374,3 +364,52 @@ func TestReconcileNoTransactionsAllUnpaid(t *testing.T) {
t.Errorf("no txs: want empty unmatched, got %v", result.Unmatched)
}
}
// Payment < total expected → fill earliest months first, spill remainder to later.
func TestUnderpaymentFillsEarliestFirst(t *testing.T) {
t.Parallel()
members := []Member{{Name: "Alice", Tier: "A", Fees: map[string]FeeData{
"2026-02": {Expected: 750, Attendance: 3},
"2026-03": {Expected: 350, Attendance: 3},
"2026-04": {Expected: 750, Attendance: 3},
}}}
txs := []Transaction{tx("Alice", "2026-02, 2026-03, 2026-04", 1250)}
result := Reconcile(members, []string{"2026-02", "2026-03", "2026-04"}, txs, nil, defaultYear)
months := result.Members["Alice"].Months
// 02 filled first (750), then 03 (350), then remainder 150 to 04
if math.Abs(months["2026-02"].Paid-750) > 0.01 {
t.Errorf("02: want 750, got %f", months["2026-02"].Paid)
}
if math.Abs(months["2026-03"].Paid-350) > 0.01 {
t.Errorf("03: want 350, got %f", months["2026-03"].Paid)
}
if math.Abs(months["2026-04"].Paid-150) > 0.01 {
t.Errorf("04: want 150, got %f", months["2026-04"].Paid)
}
}
// Prior txn fills 02 partially; later txn finishes 02 then spills to 03.
func TestFillFirstAcrossTwoTransactions(t *testing.T) {
t.Parallel()
members := []Member{{Name: "Matyáš", Tier: "A", Fees: map[string]FeeData{
"2026-02": {Expected: 500, Attendance: 2},
"2026-03": {Expected: 250, Attendance: 1},
}}}
sortedMonths := []string{"2026-02", "2026-03"}
txs := []Transaction{
tx("Matyáš", "2026-02", 200),
tx("Matyáš", "2026-02, 2026-03", 550),
}
result := Reconcile(members, sortedMonths, txs, nil, defaultYear)
months := result.Members["Matyáš"].Months
if math.Abs(months["2026-02"].Paid-500) > 0.01 {
t.Errorf("02: want 500, got %f", months["2026-02"].Paid)
}
if math.Abs(months["2026-03"].Paid-250) > 0.01 {
t.Errorf("03: want 250, got %f", months["2026-03"].Paid)
}
}

View File

@@ -1,7 +1,7 @@
{
"case": "03_proportional_remainder",
"func": "scripts.match_payments.reconcile",
"captured_at": "2026-05-06",
"captured_at": "2026-05-11",
"input": {
"members": [
{
@@ -54,10 +54,10 @@
"original_expected": 750,
"attendance_count": 3,
"exception": null,
"paid": 324.3243243243243,
"paid": 750.0,
"transactions": [
{
"amount": 324.3243243243243,
"amount": 750.0,
"date": "2026-03-10",
"sender": "Member_d035d9f9",
"message": "",
@@ -70,10 +70,10 @@
"original_expected": 750,
"attendance_count": 2,
"exception": null,
"paid": 324.3243243243243,
"paid": 50.0,
"transactions": [
{
"amount": 324.3243243243243,
"amount": 50.0,
"date": "2026-03-10",
"sender": "Member_d035d9f9",
"message": "",
@@ -86,25 +86,17 @@
"original_expected": 350,
"attendance_count": 2,
"exception": null,
"paid": 151.35135135135135,
"transactions": [
{
"amount": 151.35135135135135,
"date": "2026-03-10",
"sender": "Member_d035d9f9",
"message": "",
"confidence": "auto"
}
]
"paid": 0,
"transactions": []
}
},
"other_transactions": [],
"total_balance": -1051
"total_balance": -1050
}
},
"unmatched": [],
"credits": {
"Member_d035d9f9": -1051
"Member_d035d9f9": -1050
}
}
}

View File

@@ -1,7 +1,7 @@
{
"case": "09_multiperson_multimonth",
"func": "scripts.match_payments.reconcile",
"captured_at": "2026-05-06",
"captured_at": "2026-05-11",
"input": {
"members": [
{
@@ -63,10 +63,10 @@
"original_expected": 750,
"attendance_count": 3,
"exception": null,
"paid": 500.0,
"paid": 750.0,
"transactions": [
{
"amount": 500.0,
"amount": 750.0,
"date": "2026-02-15",
"sender": "Member_d035d9f9",
"message": "",
@@ -79,10 +79,10 @@
"original_expected": 750,
"attendance_count": 2,
"exception": null,
"paid": 500.0,
"paid": 250.0,
"transactions": [
{
"amount": 500.0,
"amount": 250.0,
"date": "2026-02-15",
"sender": "Member_d035d9f9",
"message": "",
@@ -102,10 +102,10 @@
"original_expected": 750,
"attendance_count": 2,
"exception": null,
"paid": 681.8181818181819,
"paid": 750.0,
"transactions": [
{
"amount": 681.8181818181819,
"amount": 750.0,
"date": "2026-02-15",
"sender": "Member_d035d9f9",
"message": "",
@@ -118,10 +118,10 @@
"original_expected": 350,
"attendance_count": 2,
"exception": null,
"paid": 318.18181818181813,
"paid": 250.0,
"transactions": [
{
"amount": 318.18181818181813,
"amount": 250.0,
"date": "2026-02-15",
"sender": "Member_d035d9f9",
"message": "",
@@ -131,13 +131,13 @@
}
},
"other_transactions": [],
"total_balance": -101
"total_balance": -100
}
},
"unmatched": [],
"credits": {
"Member_d035d9f9": -500,
"Member_f4a93e46": -101
"Member_f4a93e46": -100
}
}
}

View File

@@ -4,6 +4,7 @@
import json
import os
import re
import sys
import urllib.request
from datetime import datetime
from html.parser import HTMLParser
@@ -89,9 +90,11 @@ def parse_czech_amount(s: str) -> float | None:
def parse_czech_date(s: str) -> str | None:
"""Parse 'DD.MM.YYYY' to 'YYYY-MM-DD'."""
"""Parse a Czech date to 'YYYY-MM-DD'. Accepts 4-digit and 2-digit years
with dot or slash separators; Fio's transparent page mixes 'DD.MM.YYYY'
and 'DD.MM.YY' in the same response."""
s = s.strip()
for fmt in ("%d.%m.%Y", "%d/%m/%Y"):
for fmt in ("%d.%m.%Y", "%d/%m/%Y", "%d.%m.%y", "%d/%m/%y"):
try:
return datetime.strptime(s, fmt).strftime("%Y-%m-%d")
except ValueError:
@@ -146,6 +149,7 @@ def fetch_transactions_transparent(
"bank_id": "", # HTML scraping doesn't give stable ID
})
print(f"fio: transparent fetched {len(rows)} raw rows, {len(transactions)} transaction(s) after filtering", file=sys.stderr)
return transactions
@@ -169,7 +173,8 @@ def fetch_transactions_api(
transactions = []
tx_list = data.get("accountStatement", {}).get("transactionList", {})
for tx in (tx_list.get("transaction") or []):
raw_list = tx_list.get("transaction") or []
for tx in raw_list:
# Each field is {"value": ..., "name": ..., "id": ...} or null
def val(col_id):
col = tx.get(f"column{col_id}")
@@ -197,6 +202,7 @@ def fetch_transactions_api(
"currency": str(val(14) or "CZK"), # column14 = Currency
})
print(f"fio: api fetched {len(raw_list)} raw transaction(s), {len(transactions)} after filtering", file=sys.stderr)
return transactions
@@ -204,8 +210,14 @@ def fetch_transactions(date_from: str, date_to: str) -> list[dict]:
"""Fetch transactions, using API if token available, else transparent page."""
token = os.environ.get("FIO_API_TOKEN", "").strip()
if token:
print(f"fio: using authenticated API, window {date_from}..{date_to}", file=sys.stderr)
return fetch_transactions_api(token, date_from, date_to)
print(
f"fio: using transparent page (FIO_API_TOKEN unset — expect publishing lag), "
f"window {date_from}..{date_to}, account=2800359168",
file=sys.stderr,
)
# Convert YYYY-MM-DD to DD.MM.YYYY for the transparent page URL
from_dt = datetime.strptime(date_from, "%Y-%m-%d")
to_dt = datetime.strptime(date_to, "%Y-%m-%d")

View File

@@ -468,26 +468,19 @@ def reconcile(
total_expected = sum(e for _, e in in_window)
if total_expected > 0 and in_window_share >= total_expected:
# Greedy phase: payment covers all in-window fees; overflow → credit.
credits[member_name] = credits.get(member_name, 0) + int(in_window_share - total_expected)
for m, exp in in_window:
alloc = float(exp)
ledger[member_name][m]["paid"] += alloc
ledger[member_name][m]["transactions"].append({
"amount": alloc,
"date": tx["date"],
"sender": tx["sender"],
"message": tx["message"],
"confidence": confidence,
})
elif total_expected > 0:
# Proportional phase: distribute in_window_share by each month's expected fee.
# Last month absorbs any float remainder so the sum equals in_window_share exactly.
if total_expected > 0:
# Fill-first: iterate in_window in matched_months order (chronological by
# convention from infer_payments.py), allocating min(remaining, deficit) to
# each month. Deficit is net of what prior transactions already paid, so a
# second payment on the same months correctly fills only what remains due.
# Any surplus after all deficits are covered goes to the credit bucket.
remaining = in_window_share
for i, (m, exp) in enumerate(in_window):
alloc = remaining if i == len(in_window) - 1 else in_window_share * exp / total_expected
remaining -= alloc
for m, exp in in_window:
paid_so_far = ledger[member_name][m]["paid"]
deficit = max(0.0, float(exp) - paid_so_far)
alloc = min(remaining, deficit)
if alloc <= 0:
continue
ledger[member_name][m]["paid"] += alloc
ledger[member_name][m]["transactions"].append({
"amount": alloc,
@@ -496,6 +489,9 @@ def reconcile(
"message": tx["message"],
"confidence": confidence,
})
remaining -= alloc
if remaining > 0:
credits[member_name] = credits.get(member_name, 0) + int(remaining)
else:
# Fallback: no expected fees (prepayment before attendance recorded); even split.
per_month = in_window_share / len(in_window)

View File

@@ -77,6 +77,35 @@ def generate_sync_id(tx: dict) -> str:
return hashlib.sha256(raw_str.encode("utf-8")).hexdigest()
def _trunc(s: str, n: int = 40) -> str:
s = str(s)
return s if len(s) <= n else s[: n - 1] + ""
def _print_fio_table(transactions: list[dict], statuses: list[str]) -> None:
headers = ["DATE", "AMOUNT", "SENDER", "VS", "MESSAGE", "BANKID", "STATUS"]
rows = [
[
str(tx.get("date", "")),
f"{float(tx.get('amount', 0)):.2f}",
str(tx.get("sender", "")),
str(tx.get("vs", "")),
_trunc(str(tx.get("message", ""))),
str(tx.get("bank_id", "")),
status,
]
for tx, status in zip(transactions, statuses)
]
widths = [
max(len(headers[i]), max((len(r[i]) for r in rows), default=0))
for i in range(len(headers))
]
sep = " "
print(sep.join(h.ljust(w) for h, w in zip(headers, widths)))
for row in rows:
print(sep.join(cell.ljust(w) for cell, w in zip(row, widths)))
def sort_sheet_by_date(service, spreadsheet_id):
"""Sort the sheet by the Date column (Column B)."""
# Get the sheet ID (gid) of the first sheet
@@ -104,12 +133,21 @@ def sort_sheet_by_date(service, spreadsheet_id):
print("Sheet sorted by date.")
def sync_to_sheets(spreadsheet_id: str, credentials_path: str, days: int = None, date_from_str: str = None, date_to_str: str = None, sort_by_date: bool = False):
def sync_to_sheets(
spreadsheet_id: str,
credentials_path: str,
days: int = None,
date_from_str: str = None,
date_to_str: str = None,
sort_by_date: bool = False,
dry_run: bool = False,
print_fio_table: bool = False,
):
print(f"Connecting to Google Sheets using {credentials_path}...")
service = get_sheets_service(credentials_path)
sheet = service.spreadsheets()
# 1. Fetch existing IDs from Column G (last column in A-G range)
# 1. Read existing sync IDs from Column K
print(f"Reading existing sync IDs from sheet...")
try:
result = sheet.values().get(
@@ -117,19 +155,22 @@ def sync_to_sheets(spreadsheet_id: str, credentials_path: str, days: int = None,
range="A1:K" # Include header and all columns to check Sync ID
).execute()
values = result.get("values", [])
# Check and insert labels if missing
if not values or values[0] != COLUMN_LABELS:
print("Inserting column labels...")
sheet.values().update(
spreadsheetId=spreadsheet_id,
range="A1",
valueInputOption="USER_ENTERED",
body={"values": [COLUMN_LABELS]}
).execute()
if dry_run:
print("Dry run: would write header row")
else:
print("Inserting column labels...")
sheet.values().update(
spreadsheetId=spreadsheet_id,
range="A1",
valueInputOption="USER_ENTERED",
body={"values": [COLUMN_LABELS]}
).execute()
existing_ids = set()
else:
# Sync ID is now the last column (index 10)
# Sync ID is the last column (index 10)
existing_ids = {row[10] for row in values[1:] if len(row) > 10}
except Exception as e:
print(f"Error reading sheet (maybe empty?): {e}")
@@ -150,8 +191,12 @@ def sync_to_sheets(spreadsheet_id: str, credentials_path: str, days: int = None,
transactions = fetch_transactions(df_str, dt_str)
print(f"Found {len(transactions)} transactions.")
# 3. Filter for new transactions
if dry_run:
print(f"Dry run: window {df_str} to {dt_str}, fetched {len(transactions)} transaction(s) from Fio")
# 3. Determine NEW/DUP for each transaction
new_rows = []
tx_statuses = []
for tx in transactions:
sync_id = generate_sync_id(tx)
if sync_id not in existing_ids:
@@ -169,24 +214,48 @@ def sync_to_sheets(spreadsheet_id: str, credentials_path: str, days: int = None,
tx.get("bank_id", ""),
sync_id,
])
tx_statuses.append("NEW")
else:
tx_statuses.append("DUP")
# 4. Print table (before early-return so all transactions are shown including DUPs)
if print_fio_table and transactions:
_print_fio_table(transactions, tx_statuses)
if not new_rows:
print("No new transactions to sync.")
if dry_run:
print("Dry run: would sync 0 new transaction(s).")
else:
print("No new transactions to sync.")
return
# 4. Append to sheet
print(f"Appending {len(new_rows)} new transactions to the sheet...")
body = {"values": new_rows}
sheet.values().append(
spreadsheetId=spreadsheet_id,
range="A2", # Appends to the end of the sheet
valueInputOption="USER_ENTERED",
body=body
).execute()
print("Sync completed successfully.")
if sort_by_date:
sort_sheet_by_date(service, spreadsheet_id)
# 5. Append to sheet or print dry-run would-write lines
if dry_run:
for tx, status in zip(transactions, tx_statuses):
if status == "NEW":
print(
f"Dry run: would append"
f" date={tx.get('date', '')}"
f" amount={tx.get('amount', '')}"
f" sender={tx.get('sender', '')}"
f" vs={tx.get('vs', '')}"
f" message={tx.get('message', '')}"
)
if sort_by_date:
print("Dry run: would sort by date")
print(f"Dry run: would sync {len(new_rows)} new transaction(s).")
else:
print(f"Appending {len(new_rows)} new transactions to the sheet...")
body = {"values": new_rows}
sheet.values().append(
spreadsheetId=spreadsheet_id,
range="A2", # Appends to the end of the sheet
valueInputOption="USER_ENTERED",
body=body
).execute()
print("Sync completed successfully.")
if sort_by_date:
sort_sheet_by_date(service, spreadsheet_id)
def main():
@@ -197,16 +266,20 @@ def main():
parser.add_argument("--from", dest="date_from", help="Start date YYYY-MM-DD")
parser.add_argument("--to", dest="date_to", help="End date YYYY-MM-DD")
parser.add_argument("--sort-by-date", action="store_true", help="Sort the sheet by date after sync")
parser.add_argument("--dry-run", action="store_true", help="Fetch and dedup without writing to the sheet")
parser.add_argument("--print-fio-table", action="store_true", help="Print aligned table of all fetched transactions with NEW/DUP status (use with --dry-run)")
args = parser.parse_args()
try:
sync_to_sheets(
spreadsheet_id=args.sheet_id,
spreadsheet_id=args.sheet_id,
credentials_path=args.credentials,
days=args.days,
date_from_str=args.date_from,
date_to_str=args.date_to,
sort_by_date=args.sort_by_date
sort_by_date=args.sort_by_date,
dry_run=args.dry_run,
print_fio_table=args.print_fio_table,
)
except Exception as e:
print(f"Sync failed: {e}")

View File

@@ -93,8 +93,8 @@ class TestMultiMonthAllocation(unittest.TestCase):
self.assertEqual(int(months['2026-02']['paid']), 750)
self.assertEqual(result['credits'].get('Alice', 0), 500)
def test_proportional_underpayment(self):
"""Payment < total expected → proportional split; sum of paid == payment amount."""
def test_underpayment_fills_earliest_first(self):
"""Payment < total expected → fill earliest months first, spill remainder to later."""
members = [('Alice', 'A', {'2026-02': (750, 3), '2026-03': (350, 3), '2026-04': (750, 3)})]
sorted_months = ['2026-02', '2026-03', '2026-04']
amount = 1250
@@ -103,18 +103,28 @@ class TestMultiMonthAllocation(unittest.TestCase):
result = reconcile(members, sorted_months, [tx])
months = result['members']['Alice']['months']
paid_02 = months['2026-02']['paid']
paid_03 = months['2026-03']['paid']
paid_04 = months['2026-04']['paid']
# 02 filled first (750), then 03 (350), then remainder 150 to 04
self.assertAlmostEqual(months['2026-02']['paid'], 750, places=2)
self.assertAlmostEqual(months['2026-03']['paid'], 350, places=2)
self.assertAlmostEqual(months['2026-04']['paid'], 150, places=2)
# No CZK lost
self.assertAlmostEqual(
months['2026-02']['paid'] + months['2026-03']['paid'] + months['2026-04']['paid'],
amount, places=2,
)
# All months should be partial (underpaid)
self.assertLess(paid_02, 750)
self.assertLess(paid_03, 350)
self.assertLess(paid_04, 750)
# Sum must equal the original payment (no CZK lost)
self.assertAlmostEqual(paid_02 + paid_03 + paid_04, amount, places=2)
# 02 and 04 have equal expected → equal allocation
self.assertAlmostEqual(paid_02, paid_04, places=2)
def test_fill_first_across_two_transactions(self):
"""Prior txn fills 02 partially; later txn finishes 02 then spills to 03."""
members = [('Matyáš', 'A', {'2026-02': (500, 2), '2026-03': (250, 1)})]
sorted_months = ['2026-02', '2026-03']
tx1 = _tx('Matyáš', '2026-02', 200)
tx2 = _tx('Matyáš', '2026-02, 2026-03', 550)
result = reconcile(members, sorted_months, [tx1, tx2])
months = result['members']['Matyáš']['months']
self.assertAlmostEqual(months['2026-02']['paid'], 500, places=2)
self.assertAlmostEqual(months['2026-03']['paid'], 250, places=2)
def test_single_month_unchanged(self):
"""Single-month payment: full amount goes to that month (regression guard)."""