Compare commits
1 Commits
0.34
...
feat/go-m6
| Author | SHA1 | Date | |
|---|---|---|---|
| d981392593 |
@@ -1,10 +1,11 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
## 2026-05-11 22:56 CEST — fix(python): parse Fio 2-digit-year dates + add `make sync-debug` dry-run tool
|
## 2026-05-08 15:24 CEST — feat(go): M6.7 — single-binary embed verification
|
||||||
|
|
||||||
- Fix: `scripts/fio_utils.py` `parse_czech_date` now accepts `DD.MM.YY` / `D.M.YY` in addition to the 4-digit-year variants. Fio's transparent page now mixes both forms in the same response; the 2-digit rows were being silently dropped, which caused `make sync-2026` to miss every recent transfer. Mirrors the Go-side fix from 2026-05-07 (CHANGELOG entry below).
|
- Confirmed `embed.FS` wiring is complete: templates parsed via `template.ParseFS(templateFS, ...)`, static assets served via `http.FileServerFS(fs.Sub(staticFS, "static"))`.
|
||||||
- Added `--dry-run` and `--print-fio-table` flags to `scripts/sync_fio_to_sheets.py`, plus a `make sync-debug [DAYS=N]` Makefile target. Mirrors `make go-sync-debug`: fetches from Fio and dedupes against the sheet, prints `STATUS=NEW/DUP` per transaction, and prints per-row `Dry run: would append …` lines + `would sort by date` instead of touching the sheet.
|
- Added `go/internal/web/assets_test.go` with two tests: `TestEmbedCompleteness` (walks disk vs embed.FS to catch forgotten files) and `TestStaticAssetsServed` (hits `/static/css/app.css` and all JS files through the mux, asserts 200 + Content-Type + non-empty body + 404 for unknown paths).
|
||||||
- Added always-on stderr diagnostics in `scripts/fio_utils.py`: which fetcher was selected (authenticated API vs. transparent-page scraper with `FIO_API_TOKEN`-unset warning), and raw-vs-after-filter transaction counts on both paths — so this class of "scraper drops everything" bug surfaces immediately.
|
- Closes M6; single binary confirmed self-contained with no adjacent `templates/` or `static/` required at runtime.
|
||||||
|
- Key files: `go/internal/web/assets_test.go` (new).
|
||||||
|
|
||||||
## 2026-05-08 14:55 CEST — feat(go): M6.6.1 — Pay-button QR popup modal
|
## 2026-05-08 14:55 CEST — feat(go): M6.6.1 — Pay-button QR popup modal
|
||||||
|
|
||||||
|
|||||||
4
Makefile
4
Makefile
@@ -35,7 +35,6 @@ help:
|
|||||||
@echo " make sync - Sync Fio transactions to Google Sheets"
|
@echo " make sync - Sync Fio transactions to Google Sheets"
|
||||||
@echo " make sync-2025 - Sync Fio transactions for Q4 2025 (Oct-Dec)"
|
@echo " make sync-2025 - Sync Fio transactions for Q4 2025 (Oct-Dec)"
|
||||||
@echo " make sync-2026 - Sync Fio transactions for the whole year of 2026"
|
@echo " make sync-2026 - Sync Fio transactions for the whole year of 2026"
|
||||||
@echo " make sync-debug [DAYS=N] - Dry-run Python sync with Fio diagnostics and txn table (default DAYS=30)"
|
|
||||||
@echo " make infer - Infer payment details (Person, Purpose, Amount) in the sheet"
|
@echo " make infer - Infer payment details (Person, Purpose, Amount) in the sheet"
|
||||||
@echo " make reconcile - Show balance report using Google Sheets data"
|
@echo " make reconcile - Show balance report using Google Sheets data"
|
||||||
@echo " make venv - Sync virtual environment with pyproject.toml"
|
@echo " make venv - Sync virtual environment with pyproject.toml"
|
||||||
@@ -126,9 +125,6 @@ sync-2025: $(PYTHON)
|
|||||||
sync-2026: $(PYTHON)
|
sync-2026: $(PYTHON)
|
||||||
$(PYTHON) scripts/sync_fio_to_sheets.py --credentials .secret/fuj-management-bot-credentials.json --from 2026-01-01 --to 2026-12-31 --sort-by-date
|
$(PYTHON) scripts/sync_fio_to_sheets.py --credentials .secret/fuj-management-bot-credentials.json --from 2026-01-01 --to 2026-12-31 --sort-by-date
|
||||||
|
|
||||||
sync-debug: $(PYTHON) ## Dry-run Python sync with Fio diagnostics and txn table (default DAYS=30)
|
|
||||||
$(PYTHON) scripts/sync_fio_to_sheets.py --credentials .secret/fuj-management-bot-credentials.json --days $(DAYS) --dry-run --print-fio-table
|
|
||||||
|
|
||||||
infer: $(PYTHON)
|
infer: $(PYTHON)
|
||||||
$(PYTHON) scripts/infer_payments.py --credentials $(CREDENTIALS)
|
$(PYTHON) scripts/infer_payments.py --credentials $(CREDENTIALS)
|
||||||
|
|
||||||
|
|||||||
@@ -117,7 +117,7 @@ Goal: feature-equivalent UX on the Go side, designed cleanly. Not a Jinja port.
|
|||||||
- [x] **M6.5** Modal JS module (`static/js/member-detail.js`): fetches `/api/adults` (or juniors), renders status/exceptions/transactions on row click; keyboard nav (Esc, ↑/↓) — `e53e238`
|
- [x] **M6.5** Modal JS module (`static/js/member-detail.js`): fetches `/api/adults` (or juniors), renders status/exceptions/transactions on row click; keyboard nav (Esc, ↑/↓) — `e53e238`
|
||||||
- [x] **M6.6** `/qr`, `/sync-bank`, `/flush-cache`, `/version` pages — `f6ba85b`
|
- [x] **M6.6** `/qr`, `/sync-bank`, `/flush-cache`, `/version` pages — `f6ba85b`
|
||||||
- [x] **M6.6.1** Pay-button QR popup modal (`payment-qr.js`); restores Python `showPayQR` UX lost in M6.6 — `4276d7b`
|
- [x] **M6.6.1** Pay-button QR popup modal (`payment-qr.js`); restores Python `showPayQR` UX lost in M6.6 — `4276d7b`
|
||||||
- [ ] **M6.7** Wire `embed.FS` into handlers; verify single-binary deployment includes all assets
|
- [ ] **M6.7** Wire `embed.FS` into handlers; verify single-binary deployment includes all assets — (pending merge)
|
||||||
|
|
||||||
**Gate:** Browser smoke on :8080: all pages render, name+month filters work, modal opens with correct data, QR loads, sync/flush work end-to-end.
|
**Gate:** Browser smoke on :8080: all pages render, name+month filters work, modal opens with correct data, QR loads, sync/flush work end-to-end.
|
||||||
|
|
||||||
|
|||||||
184
docs/plans/2026-05-08-1457-go-m6-7-single-binary-embed-verify.md
Normal file
184
docs/plans/2026-05-08-1457-go-m6-7-single-binary-embed-verify.md
Normal file
@@ -0,0 +1,184 @@
|
|||||||
|
# M6.7 — Single-binary embed verification
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
M6.7 is the final task in M6 (Go-native HTML frontend). Per the
|
||||||
|
[progress tracker](2026-05-03-2349-go-backend-rewrite-progress.md): "Wire
|
||||||
|
`embed.FS` into handlers; verify single-binary deployment includes all
|
||||||
|
assets."
|
||||||
|
|
||||||
|
The wiring is **already in place** from M6.1 onward:
|
||||||
|
|
||||||
|
- [go/internal/web/assets.go](../../go/internal/web/assets.go) declares
|
||||||
|
`//go:embed templates` → `templateFS` and `//go:embed static` → `staticFS`.
|
||||||
|
- [go/internal/web/render.go:66](../../go/internal/web/render.go#L66)
|
||||||
|
parses every page template via `template.New(...).ParseFS(templateFS, ...)`.
|
||||||
|
- [go/internal/web/server.go:48-68](../../go/internal/web/server.go#L48-L68)
|
||||||
|
serves `/static/*` via `http.FileServerFS(fs.Sub(staticFS, "static"))`.
|
||||||
|
- [go/build/Dockerfile](../../go/build/Dockerfile) copies only the compiled
|
||||||
|
binary into the `alpine:3` runtime image — no `templates/` or `static/`
|
||||||
|
directory ever lands beside it.
|
||||||
|
|
||||||
|
What is missing is **proof** the embed is complete and stays complete:
|
||||||
|
|
||||||
|
1. Nothing fails the build/test if a contributor adds a new file under
|
||||||
|
`internal/web/templates/` or `internal/web/static/` that isn't matched
|
||||||
|
by an `//go:embed` glob (or, more realistically, adds a sibling
|
||||||
|
directory like `static/img/` and the glob still picks it up — but a
|
||||||
|
typo'd directive would silently drop it).
|
||||||
|
2. No automated test exercises the `/static/*` route against the embedded
|
||||||
|
FS — current tests in
|
||||||
|
[html_handler_test.go](../../go/internal/web/html_handler_test.go)
|
||||||
|
render templates (which proves `templateFS` is good) but never hit a
|
||||||
|
static URL through the mux.
|
||||||
|
3. The "single binary, no working-dir assets" property is undocumented —
|
||||||
|
if it ever broke, no one would notice until the Docker image started
|
||||||
|
500'ing in prod.
|
||||||
|
|
||||||
|
The intended outcome: a small test file plus a documented manual
|
||||||
|
verification step, after which M6.7 can be ticked and M6 closed.
|
||||||
|
|
||||||
|
## Plan
|
||||||
|
|
||||||
|
### 1. Add `go/internal/web/assets_test.go`
|
||||||
|
|
||||||
|
One new file, two tests, no production code changes.
|
||||||
|
|
||||||
|
**Test A — embed completeness regression guard.** Walks the on-disk
|
||||||
|
`templates/` and `static/` directories and asserts every regular file is
|
||||||
|
also present in the corresponding embedded FS. Catches:
|
||||||
|
|
||||||
|
- A new template added without updating the `//go:embed` directive
|
||||||
|
(current globs are `templates` and `static` — recursive by default for
|
||||||
|
directories, so this is a low-probability regression, but the test
|
||||||
|
doubles as living documentation of the contract).
|
||||||
|
- A typo in the directive (e.g. someone renames `static` → `assets` in
|
||||||
|
one place but not the other).
|
||||||
|
|
||||||
|
Implementation sketch:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func TestEmbedCompleteness(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
diskFS fs.FS // os.DirFS("templates") / os.DirFS("static")
|
||||||
|
embed fs.FS // exported helper or via internal test in package web
|
||||||
|
root string
|
||||||
|
}{...}
|
||||||
|
for _, tc := range cases {
|
||||||
|
_ = fs.WalkDir(tc.diskFS, ".", func(path string, d fs.DirEntry, err error) error {
|
||||||
|
if err != nil || d.IsDir() { return err }
|
||||||
|
embPath := tc.root + "/" + path
|
||||||
|
if _, err := fs.Stat(tc.embed, embPath); err != nil {
|
||||||
|
t.Errorf("file %q on disk but missing in embed.FS: %v", embPath, err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Because `templateFS` and `staticFS` are unexported package vars, this
|
||||||
|
test lives in `package web` (not `web_test`) — sibling to
|
||||||
|
[assets.go](../../go/internal/web/assets.go). All the existing handler
|
||||||
|
tests are in `package web_test`; that's fine, this one is internal.
|
||||||
|
|
||||||
|
**Test B — `/static/*` end-to-end via the mux.** Builds an `http.ServeMux`
|
||||||
|
with the same wiring as
|
||||||
|
[server.go:68](../../go/internal/web/server.go#L68), fires httptest
|
||||||
|
requests, asserts:
|
||||||
|
|
||||||
|
- `GET /static/css/app.css` → 200, `Content-Type: text/css; charset=utf-8`,
|
||||||
|
body contains a known string from app.css (e.g. a CSS selector).
|
||||||
|
- `GET /static/js/member-detail.js` → 200, `Content-Type` starts with
|
||||||
|
`text/javascript` or `application/javascript`, body non-empty.
|
||||||
|
- `GET /static/js/payment-qr.js` → 200, body non-empty.
|
||||||
|
- `GET /static/css/missing.css` → 404 (sanity: the file server actually
|
||||||
|
rejects unknown paths instead of returning some default).
|
||||||
|
|
||||||
|
Rather than duplicate the mux assembly, factor a tiny helper (or test the
|
||||||
|
existing mux). The cleanest move: extract `staticHandler()` from
|
||||||
|
[server.go:48-50,68](../../go/internal/web/server.go#L48-L68) into a small
|
||||||
|
exported-from-package function or just `staticFS` / `fs.Sub` helper, and
|
||||||
|
have the test call it. Smallest delta: keep production code unchanged and
|
||||||
|
replicate the two-line wiring inside the test file (acceptable — it's
|
||||||
|
two lines and the test exists precisely to lock that contract).
|
||||||
|
|
||||||
|
### 2. Manual / one-shot verification (no code; documented in plan only)
|
||||||
|
|
||||||
|
Run once locally and tick M6.7. Command transcript:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make go-build # → ./bin/fuj
|
||||||
|
cp bin/fuj /tmp/fuj-standalone
|
||||||
|
cd /tmp # working dir has no templates/ or static/
|
||||||
|
./fuj-standalone server &
|
||||||
|
SERVER_PID=$!
|
||||||
|
sleep 1
|
||||||
|
curl -sf http://localhost:8080/adults | grep -q "Adults Dashboard"
|
||||||
|
curl -sf http://localhost:8080/juniors | grep -q "Juniors"
|
||||||
|
curl -sf http://localhost:8080/payments | grep -q "Payments Ledger"
|
||||||
|
curl -sf -o /tmp/app.css http://localhost:8080/static/css/app.css \
|
||||||
|
&& test -s /tmp/app.css
|
||||||
|
curl -sf -o /tmp/qr.js http://localhost:8080/static/js/payment-qr.js \
|
||||||
|
&& test -s /tmp/qr.js
|
||||||
|
kill $SERVER_PID
|
||||||
|
```
|
||||||
|
|
||||||
|
`fuj server` will fail to talk to Sheets without credentials, so the
|
||||||
|
`/adults` etc. pages will render with the `Error` field set — that's
|
||||||
|
fine; the assertion is that the **template + static asset pipeline** is
|
||||||
|
self-contained, not that data loads. Each curl above only checks for
|
||||||
|
markup present in every render path (header text and stylesheet body).
|
||||||
|
|
||||||
|
### 3. Tracker + changelog
|
||||||
|
|
||||||
|
- Tick `M6.7` in
|
||||||
|
[docs/plans/2026-05-03-2349-go-backend-rewrite-progress.md:120](2026-05-03-2349-go-backend-rewrite-progress.md#L120),
|
||||||
|
append the merge SHA on the line.
|
||||||
|
- Mark "Last updated" date and bump milestone status: M6 complete, next is
|
||||||
|
M7.
|
||||||
|
- Append a `CHANGELOG.md` entry per CLAUDE.md convention (`date "+%Y-%m-%d %H:%M %Z"`).
|
||||||
|
|
||||||
|
## Files touched
|
||||||
|
|
||||||
|
| File | Change |
|
||||||
|
| --- | --- |
|
||||||
|
| [go/internal/web/assets_test.go](../../go/internal/web/assets_test.go) | **new** — two tests (embed completeness + `/static/*` mux) |
|
||||||
|
| [docs/plans/2026-05-03-2349-go-backend-rewrite-progress.md](2026-05-03-2349-go-backend-rewrite-progress.md) | tick M6.7, bump "last updated" |
|
||||||
|
| [CHANGELOG.md](../../CHANGELOG.md) | new top entry |
|
||||||
|
|
||||||
|
No production source files change. (If extracting the static handler
|
||||||
|
reads cleaner, a 4-line refactor in
|
||||||
|
[server.go](../../go/internal/web/server.go) is acceptable but optional.)
|
||||||
|
|
||||||
|
## Branch + MR
|
||||||
|
|
||||||
|
Per project convention this is a feature, so:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git checkout -b feat/go-m6-7-embed-verify
|
||||||
|
# … commits …
|
||||||
|
git push -u origin feat/go-m6-7-embed-verify
|
||||||
|
tea pr create --title "feat(go): M6.7 — single-binary embed verification" \
|
||||||
|
--description "<short body referencing M6.7>" --base main \
|
||||||
|
--head feat/go-m6-7-embed-verify
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
After implementation:
|
||||||
|
|
||||||
|
1. `make go-test` → green (new `TestEmbedCompleteness` and `TestStaticAssetsServed` pass).
|
||||||
|
2. `make go-lint` → clean.
|
||||||
|
3. Run the manual transcript in §2 above — all curls succeed, no
|
||||||
|
"template not found" or 404 on static assets.
|
||||||
|
4. `make go-build && docker build -f go/build/Dockerfile -t fuj-go:m6-7 go/` →
|
||||||
|
succeeds; `docker run --rm -p 8080:8080 fuj-go:m6-7` serves `/adults`
|
||||||
|
with stylesheet attached (visual smoke test in browser).
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- Re-architecting how templates are parsed or served.
|
||||||
|
- Compressing / fingerprinting static assets (a separate concern).
|
||||||
|
- Live integration test with real Sheets data — covered later in M7.
|
||||||
93
go/internal/web/assets_test.go
Normal file
93
go/internal/web/assets_test.go
Normal file
@@ -0,0 +1,93 @@
|
|||||||
|
package web
|
||||||
|
|
||||||
|
import (
|
||||||
|
"io/fs"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestEmbedCompleteness guards against a new template or static file being
|
||||||
|
// added to disk but missing from the embedded FS (e.g. a new directory that
|
||||||
|
// the //go:embed glob does not match).
|
||||||
|
func TestEmbedCompleteness(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
diskDir string
|
||||||
|
embedFS fs.FS
|
||||||
|
embedRoot string
|
||||||
|
}{
|
||||||
|
{"templates", "templates", templateFS, "templates"},
|
||||||
|
{"static", "static", staticFS, "static"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range cases {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
diskFS := os.DirFS(tc.diskDir)
|
||||||
|
_ = fs.WalkDir(diskFS, ".", func(path string, d fs.DirEntry, err error) error {
|
||||||
|
if err != nil || d.IsDir() {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
embPath := tc.embedRoot + "/" + path
|
||||||
|
if _, statErr := fs.Stat(tc.embedFS, embPath); statErr != nil {
|
||||||
|
t.Errorf("file %q exists on disk but is missing from embed.FS (%v)", embPath, statErr)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestStaticAssetsServed verifies that /static/* is served from the embedded
|
||||||
|
// FS through the same mux wiring used in server.go, so a standalone binary
|
||||||
|
// with no adjacent static/ directory still delivers assets.
|
||||||
|
func TestStaticAssetsServed(t *testing.T) {
|
||||||
|
subFS, err := fs.Sub(staticFS, "static")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("fs.Sub static: %v", err)
|
||||||
|
}
|
||||||
|
mux := http.NewServeMux()
|
||||||
|
mux.Handle("/static/", http.StripPrefix("/static/", http.FileServerFS(subFS)))
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
path string
|
||||||
|
wantCT string
|
||||||
|
wantSnippet string
|
||||||
|
}{
|
||||||
|
{"/static/css/app.css", "text/css", "body {"},
|
||||||
|
{"/static/js/member-detail.js", "javascript", "Member-detail modal"},
|
||||||
|
{"/static/js/filters.js", "javascript", ""},
|
||||||
|
{"/static/js/payment-qr.js", "javascript", ""},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range cases {
|
||||||
|
t.Run(tc.path, func(t *testing.T) {
|
||||||
|
req := httptest.NewRequest(http.MethodGet, tc.path, nil)
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
mux.ServeHTTP(w, req)
|
||||||
|
|
||||||
|
if w.Code != http.StatusOK {
|
||||||
|
t.Fatalf("GET %s: status %d, want 200", tc.path, w.Code)
|
||||||
|
}
|
||||||
|
ct := w.Header().Get("Content-Type")
|
||||||
|
if !strings.Contains(ct, tc.wantCT) {
|
||||||
|
t.Errorf("GET %s: Content-Type %q, want it to contain %q", tc.path, ct, tc.wantCT)
|
||||||
|
}
|
||||||
|
if tc.wantSnippet != "" && !strings.Contains(w.Body.String(), tc.wantSnippet) {
|
||||||
|
t.Errorf("GET %s: body missing expected snippet %q", tc.path, tc.wantSnippet)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sanity: unknown path → 404 (file server doesn't fall through silently)
|
||||||
|
t.Run("missing-file", func(t *testing.T) {
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/static/css/nonexistent.css", nil)
|
||||||
|
w := httptest.NewRecorder()
|
||||||
|
mux.ServeHTTP(w, req)
|
||||||
|
if w.Code != http.StatusNotFound {
|
||||||
|
t.Errorf("unknown static path: status %d, want 404", w.Code)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -4,7 +4,6 @@
|
|||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
import sys
|
|
||||||
import urllib.request
|
import urllib.request
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from html.parser import HTMLParser
|
from html.parser import HTMLParser
|
||||||
@@ -90,11 +89,9 @@ def parse_czech_amount(s: str) -> float | None:
|
|||||||
|
|
||||||
|
|
||||||
def parse_czech_date(s: str) -> str | None:
|
def parse_czech_date(s: str) -> str | None:
|
||||||
"""Parse a Czech date to 'YYYY-MM-DD'. Accepts 4-digit and 2-digit years
|
"""Parse 'DD.MM.YYYY' to 'YYYY-MM-DD'."""
|
||||||
with dot or slash separators; Fio's transparent page mixes 'DD.MM.YYYY'
|
|
||||||
and 'DD.MM.YY' in the same response."""
|
|
||||||
s = s.strip()
|
s = s.strip()
|
||||||
for fmt in ("%d.%m.%Y", "%d/%m/%Y", "%d.%m.%y", "%d/%m/%y"):
|
for fmt in ("%d.%m.%Y", "%d/%m/%Y"):
|
||||||
try:
|
try:
|
||||||
return datetime.strptime(s, fmt).strftime("%Y-%m-%d")
|
return datetime.strptime(s, fmt).strftime("%Y-%m-%d")
|
||||||
except ValueError:
|
except ValueError:
|
||||||
@@ -149,7 +146,6 @@ def fetch_transactions_transparent(
|
|||||||
"bank_id": "", # HTML scraping doesn't give stable ID
|
"bank_id": "", # HTML scraping doesn't give stable ID
|
||||||
})
|
})
|
||||||
|
|
||||||
print(f"fio: transparent fetched {len(rows)} raw rows, {len(transactions)} transaction(s) after filtering", file=sys.stderr)
|
|
||||||
return transactions
|
return transactions
|
||||||
|
|
||||||
|
|
||||||
@@ -173,8 +169,7 @@ def fetch_transactions_api(
|
|||||||
|
|
||||||
transactions = []
|
transactions = []
|
||||||
tx_list = data.get("accountStatement", {}).get("transactionList", {})
|
tx_list = data.get("accountStatement", {}).get("transactionList", {})
|
||||||
raw_list = tx_list.get("transaction") or []
|
for tx in (tx_list.get("transaction") or []):
|
||||||
for tx in raw_list:
|
|
||||||
# Each field is {"value": ..., "name": ..., "id": ...} or null
|
# Each field is {"value": ..., "name": ..., "id": ...} or null
|
||||||
def val(col_id):
|
def val(col_id):
|
||||||
col = tx.get(f"column{col_id}")
|
col = tx.get(f"column{col_id}")
|
||||||
@@ -202,7 +197,6 @@ def fetch_transactions_api(
|
|||||||
"currency": str(val(14) or "CZK"), # column14 = Currency
|
"currency": str(val(14) or "CZK"), # column14 = Currency
|
||||||
})
|
})
|
||||||
|
|
||||||
print(f"fio: api fetched {len(raw_list)} raw transaction(s), {len(transactions)} after filtering", file=sys.stderr)
|
|
||||||
return transactions
|
return transactions
|
||||||
|
|
||||||
|
|
||||||
@@ -210,14 +204,8 @@ def fetch_transactions(date_from: str, date_to: str) -> list[dict]:
|
|||||||
"""Fetch transactions, using API if token available, else transparent page."""
|
"""Fetch transactions, using API if token available, else transparent page."""
|
||||||
token = os.environ.get("FIO_API_TOKEN", "").strip()
|
token = os.environ.get("FIO_API_TOKEN", "").strip()
|
||||||
if token:
|
if token:
|
||||||
print(f"fio: using authenticated API, window {date_from}..{date_to}", file=sys.stderr)
|
|
||||||
return fetch_transactions_api(token, date_from, date_to)
|
return fetch_transactions_api(token, date_from, date_to)
|
||||||
|
|
||||||
print(
|
|
||||||
f"fio: using transparent page (FIO_API_TOKEN unset — expect publishing lag), "
|
|
||||||
f"window {date_from}..{date_to}, account=2800359168",
|
|
||||||
file=sys.stderr,
|
|
||||||
)
|
|
||||||
# Convert YYYY-MM-DD to DD.MM.YYYY for the transparent page URL
|
# Convert YYYY-MM-DD to DD.MM.YYYY for the transparent page URL
|
||||||
from_dt = datetime.strptime(date_from, "%Y-%m-%d")
|
from_dt = datetime.strptime(date_from, "%Y-%m-%d")
|
||||||
to_dt = datetime.strptime(date_to, "%Y-%m-%d")
|
to_dt = datetime.strptime(date_to, "%Y-%m-%d")
|
||||||
|
|||||||
@@ -77,35 +77,6 @@ def generate_sync_id(tx: dict) -> str:
|
|||||||
return hashlib.sha256(raw_str.encode("utf-8")).hexdigest()
|
return hashlib.sha256(raw_str.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
def _trunc(s: str, n: int = 40) -> str:
|
|
||||||
s = str(s)
|
|
||||||
return s if len(s) <= n else s[: n - 1] + "…"
|
|
||||||
|
|
||||||
|
|
||||||
def _print_fio_table(transactions: list[dict], statuses: list[str]) -> None:
|
|
||||||
headers = ["DATE", "AMOUNT", "SENDER", "VS", "MESSAGE", "BANKID", "STATUS"]
|
|
||||||
rows = [
|
|
||||||
[
|
|
||||||
str(tx.get("date", "")),
|
|
||||||
f"{float(tx.get('amount', 0)):.2f}",
|
|
||||||
str(tx.get("sender", "")),
|
|
||||||
str(tx.get("vs", "")),
|
|
||||||
_trunc(str(tx.get("message", ""))),
|
|
||||||
str(tx.get("bank_id", "")),
|
|
||||||
status,
|
|
||||||
]
|
|
||||||
for tx, status in zip(transactions, statuses)
|
|
||||||
]
|
|
||||||
widths = [
|
|
||||||
max(len(headers[i]), max((len(r[i]) for r in rows), default=0))
|
|
||||||
for i in range(len(headers))
|
|
||||||
]
|
|
||||||
sep = " "
|
|
||||||
print(sep.join(h.ljust(w) for h, w in zip(headers, widths)))
|
|
||||||
for row in rows:
|
|
||||||
print(sep.join(cell.ljust(w) for cell, w in zip(row, widths)))
|
|
||||||
|
|
||||||
|
|
||||||
def sort_sheet_by_date(service, spreadsheet_id):
|
def sort_sheet_by_date(service, spreadsheet_id):
|
||||||
"""Sort the sheet by the Date column (Column B)."""
|
"""Sort the sheet by the Date column (Column B)."""
|
||||||
# Get the sheet ID (gid) of the first sheet
|
# Get the sheet ID (gid) of the first sheet
|
||||||
@@ -133,21 +104,12 @@ def sort_sheet_by_date(service, spreadsheet_id):
|
|||||||
print("Sheet sorted by date.")
|
print("Sheet sorted by date.")
|
||||||
|
|
||||||
|
|
||||||
def sync_to_sheets(
|
def sync_to_sheets(spreadsheet_id: str, credentials_path: str, days: int = None, date_from_str: str = None, date_to_str: str = None, sort_by_date: bool = False):
|
||||||
spreadsheet_id: str,
|
|
||||||
credentials_path: str,
|
|
||||||
days: int = None,
|
|
||||||
date_from_str: str = None,
|
|
||||||
date_to_str: str = None,
|
|
||||||
sort_by_date: bool = False,
|
|
||||||
dry_run: bool = False,
|
|
||||||
print_fio_table: bool = False,
|
|
||||||
):
|
|
||||||
print(f"Connecting to Google Sheets using {credentials_path}...")
|
print(f"Connecting to Google Sheets using {credentials_path}...")
|
||||||
service = get_sheets_service(credentials_path)
|
service = get_sheets_service(credentials_path)
|
||||||
sheet = service.spreadsheets()
|
sheet = service.spreadsheets()
|
||||||
|
|
||||||
# 1. Read existing sync IDs from Column K
|
# 1. Fetch existing IDs from Column G (last column in A-G range)
|
||||||
print(f"Reading existing sync IDs from sheet...")
|
print(f"Reading existing sync IDs from sheet...")
|
||||||
try:
|
try:
|
||||||
result = sheet.values().get(
|
result = sheet.values().get(
|
||||||
@@ -155,22 +117,19 @@ def sync_to_sheets(
|
|||||||
range="A1:K" # Include header and all columns to check Sync ID
|
range="A1:K" # Include header and all columns to check Sync ID
|
||||||
).execute()
|
).execute()
|
||||||
values = result.get("values", [])
|
values = result.get("values", [])
|
||||||
|
|
||||||
# Check and insert labels if missing
|
# Check and insert labels if missing
|
||||||
if not values or values[0] != COLUMN_LABELS:
|
if not values or values[0] != COLUMN_LABELS:
|
||||||
if dry_run:
|
print("Inserting column labels...")
|
||||||
print("Dry run: would write header row")
|
sheet.values().update(
|
||||||
else:
|
spreadsheetId=spreadsheet_id,
|
||||||
print("Inserting column labels...")
|
range="A1",
|
||||||
sheet.values().update(
|
valueInputOption="USER_ENTERED",
|
||||||
spreadsheetId=spreadsheet_id,
|
body={"values": [COLUMN_LABELS]}
|
||||||
range="A1",
|
).execute()
|
||||||
valueInputOption="USER_ENTERED",
|
|
||||||
body={"values": [COLUMN_LABELS]}
|
|
||||||
).execute()
|
|
||||||
existing_ids = set()
|
existing_ids = set()
|
||||||
else:
|
else:
|
||||||
# Sync ID is the last column (index 10)
|
# Sync ID is now the last column (index 10)
|
||||||
existing_ids = {row[10] for row in values[1:] if len(row) > 10}
|
existing_ids = {row[10] for row in values[1:] if len(row) > 10}
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"Error reading sheet (maybe empty?): {e}")
|
print(f"Error reading sheet (maybe empty?): {e}")
|
||||||
@@ -191,12 +150,8 @@ def sync_to_sheets(
|
|||||||
transactions = fetch_transactions(df_str, dt_str)
|
transactions = fetch_transactions(df_str, dt_str)
|
||||||
print(f"Found {len(transactions)} transactions.")
|
print(f"Found {len(transactions)} transactions.")
|
||||||
|
|
||||||
if dry_run:
|
# 3. Filter for new transactions
|
||||||
print(f"Dry run: window {df_str} to {dt_str}, fetched {len(transactions)} transaction(s) from Fio")
|
|
||||||
|
|
||||||
# 3. Determine NEW/DUP for each transaction
|
|
||||||
new_rows = []
|
new_rows = []
|
||||||
tx_statuses = []
|
|
||||||
for tx in transactions:
|
for tx in transactions:
|
||||||
sync_id = generate_sync_id(tx)
|
sync_id = generate_sync_id(tx)
|
||||||
if sync_id not in existing_ids:
|
if sync_id not in existing_ids:
|
||||||
@@ -214,48 +169,24 @@ def sync_to_sheets(
|
|||||||
tx.get("bank_id", ""),
|
tx.get("bank_id", ""),
|
||||||
sync_id,
|
sync_id,
|
||||||
])
|
])
|
||||||
tx_statuses.append("NEW")
|
|
||||||
else:
|
|
||||||
tx_statuses.append("DUP")
|
|
||||||
|
|
||||||
# 4. Print table (before early-return so all transactions are shown including DUPs)
|
|
||||||
if print_fio_table and transactions:
|
|
||||||
_print_fio_table(transactions, tx_statuses)
|
|
||||||
|
|
||||||
if not new_rows:
|
if not new_rows:
|
||||||
if dry_run:
|
print("No new transactions to sync.")
|
||||||
print("Dry run: would sync 0 new transaction(s).")
|
|
||||||
else:
|
|
||||||
print("No new transactions to sync.")
|
|
||||||
return
|
return
|
||||||
|
|
||||||
# 5. Append to sheet or print dry-run would-write lines
|
# 4. Append to sheet
|
||||||
if dry_run:
|
print(f"Appending {len(new_rows)} new transactions to the sheet...")
|
||||||
for tx, status in zip(transactions, tx_statuses):
|
body = {"values": new_rows}
|
||||||
if status == "NEW":
|
sheet.values().append(
|
||||||
print(
|
spreadsheetId=spreadsheet_id,
|
||||||
f"Dry run: would append"
|
range="A2", # Appends to the end of the sheet
|
||||||
f" date={tx.get('date', '')}"
|
valueInputOption="USER_ENTERED",
|
||||||
f" amount={tx.get('amount', '')}"
|
body=body
|
||||||
f" sender={tx.get('sender', '')}"
|
).execute()
|
||||||
f" vs={tx.get('vs', '')}"
|
print("Sync completed successfully.")
|
||||||
f" message={tx.get('message', '')}"
|
|
||||||
)
|
if sort_by_date:
|
||||||
if sort_by_date:
|
sort_sheet_by_date(service, spreadsheet_id)
|
||||||
print("Dry run: would sort by date")
|
|
||||||
print(f"Dry run: would sync {len(new_rows)} new transaction(s).")
|
|
||||||
else:
|
|
||||||
print(f"Appending {len(new_rows)} new transactions to the sheet...")
|
|
||||||
body = {"values": new_rows}
|
|
||||||
sheet.values().append(
|
|
||||||
spreadsheetId=spreadsheet_id,
|
|
||||||
range="A2", # Appends to the end of the sheet
|
|
||||||
valueInputOption="USER_ENTERED",
|
|
||||||
body=body
|
|
||||||
).execute()
|
|
||||||
print("Sync completed successfully.")
|
|
||||||
if sort_by_date:
|
|
||||||
sort_sheet_by_date(service, spreadsheet_id)
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
@@ -266,20 +197,16 @@ def main():
|
|||||||
parser.add_argument("--from", dest="date_from", help="Start date YYYY-MM-DD")
|
parser.add_argument("--from", dest="date_from", help="Start date YYYY-MM-DD")
|
||||||
parser.add_argument("--to", dest="date_to", help="End date YYYY-MM-DD")
|
parser.add_argument("--to", dest="date_to", help="End date YYYY-MM-DD")
|
||||||
parser.add_argument("--sort-by-date", action="store_true", help="Sort the sheet by date after sync")
|
parser.add_argument("--sort-by-date", action="store_true", help="Sort the sheet by date after sync")
|
||||||
parser.add_argument("--dry-run", action="store_true", help="Fetch and dedup without writing to the sheet")
|
|
||||||
parser.add_argument("--print-fio-table", action="store_true", help="Print aligned table of all fetched transactions with NEW/DUP status (use with --dry-run)")
|
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
try:
|
try:
|
||||||
sync_to_sheets(
|
sync_to_sheets(
|
||||||
spreadsheet_id=args.sheet_id,
|
spreadsheet_id=args.sheet_id,
|
||||||
credentials_path=args.credentials,
|
credentials_path=args.credentials,
|
||||||
days=args.days,
|
days=args.days,
|
||||||
date_from_str=args.date_from,
|
date_from_str=args.date_from,
|
||||||
date_to_str=args.date_to,
|
date_to_str=args.date_to,
|
||||||
sort_by_date=args.sort_by_date,
|
sort_by_date=args.sort_by_date
|
||||||
dry_run=args.dry_run,
|
|
||||||
print_fio_table=args.print_fio_table,
|
|
||||||
)
|
)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"Sync failed: {e}")
|
print(f"Sync failed: {e}")
|
||||||
|
|||||||
Reference in New Issue
Block a user