Compare commits
1 Commits
feat/go-m6
...
0.34
| Author | SHA1 | Date | |
|---|---|---|---|
| aaa876e593 |
@@ -1,11 +1,10 @@
|
||||
# Changelog
|
||||
|
||||
## 2026-05-08 15:24 CEST — feat(go): M6.7 — single-binary embed verification
|
||||
## 2026-05-11 22:56 CEST — fix(python): parse Fio 2-digit-year dates + add `make sync-debug` dry-run tool
|
||||
|
||||
- Confirmed `embed.FS` wiring is complete: templates parsed via `template.ParseFS(templateFS, ...)`, static assets served via `http.FileServerFS(fs.Sub(staticFS, "static"))`.
|
||||
- Added `go/internal/web/assets_test.go` with two tests: `TestEmbedCompleteness` (walks disk vs embed.FS to catch forgotten files) and `TestStaticAssetsServed` (hits `/static/css/app.css` and all JS files through the mux, asserts 200 + Content-Type + non-empty body + 404 for unknown paths).
|
||||
- Closes M6; single binary confirmed self-contained with no adjacent `templates/` or `static/` required at runtime.
|
||||
- Key files: `go/internal/web/assets_test.go` (new).
|
||||
- Fix: `scripts/fio_utils.py` `parse_czech_date` now accepts `DD.MM.YY` / `D.M.YY` in addition to the 4-digit-year variants. Fio's transparent page now mixes both forms in the same response; the 2-digit rows were being silently dropped, which caused `make sync-2026` to miss every recent transfer. Mirrors the Go-side fix from 2026-05-07 (CHANGELOG entry below).
|
||||
- Added `--dry-run` and `--print-fio-table` flags to `scripts/sync_fio_to_sheets.py`, plus a `make sync-debug [DAYS=N]` Makefile target. Mirrors `make go-sync-debug`: fetches from Fio and dedupes against the sheet, prints `STATUS=NEW/DUP` per transaction, and prints per-row `Dry run: would append …` lines + `would sort by date` instead of touching the sheet.
|
||||
- Added always-on stderr diagnostics in `scripts/fio_utils.py`: which fetcher was selected (authenticated API vs. transparent-page scraper with `FIO_API_TOKEN`-unset warning), and raw-vs-after-filter transaction counts on both paths — so this class of "scraper drops everything" bug surfaces immediately.
|
||||
|
||||
## 2026-05-08 14:55 CEST — feat(go): M6.6.1 — Pay-button QR popup modal
|
||||
|
||||
|
||||
4
Makefile
4
Makefile
@@ -35,6 +35,7 @@ help:
|
||||
@echo " make sync - Sync Fio transactions to Google Sheets"
|
||||
@echo " make sync-2025 - Sync Fio transactions for Q4 2025 (Oct-Dec)"
|
||||
@echo " make sync-2026 - Sync Fio transactions for the whole year of 2026"
|
||||
@echo " make sync-debug [DAYS=N] - Dry-run Python sync with Fio diagnostics and txn table (default DAYS=30)"
|
||||
@echo " make infer - Infer payment details (Person, Purpose, Amount) in the sheet"
|
||||
@echo " make reconcile - Show balance report using Google Sheets data"
|
||||
@echo " make venv - Sync virtual environment with pyproject.toml"
|
||||
@@ -125,6 +126,9 @@ sync-2025: $(PYTHON)
|
||||
sync-2026: $(PYTHON)
|
||||
$(PYTHON) scripts/sync_fio_to_sheets.py --credentials .secret/fuj-management-bot-credentials.json --from 2026-01-01 --to 2026-12-31 --sort-by-date
|
||||
|
||||
sync-debug: $(PYTHON) ## Dry-run Python sync with Fio diagnostics and txn table (default DAYS=30)
|
||||
$(PYTHON) scripts/sync_fio_to_sheets.py --credentials .secret/fuj-management-bot-credentials.json --days $(DAYS) --dry-run --print-fio-table
|
||||
|
||||
infer: $(PYTHON)
|
||||
$(PYTHON) scripts/infer_payments.py --credentials $(CREDENTIALS)
|
||||
|
||||
|
||||
@@ -117,7 +117,7 @@ Goal: feature-equivalent UX on the Go side, designed cleanly. Not a Jinja port.
|
||||
- [x] **M6.5** Modal JS module (`static/js/member-detail.js`): fetches `/api/adults` (or juniors), renders status/exceptions/transactions on row click; keyboard nav (Esc, ↑/↓) — `e53e238`
|
||||
- [x] **M6.6** `/qr`, `/sync-bank`, `/flush-cache`, `/version` pages — `f6ba85b`
|
||||
- [x] **M6.6.1** Pay-button QR popup modal (`payment-qr.js`); restores Python `showPayQR` UX lost in M6.6 — `4276d7b`
|
||||
- [ ] **M6.7** Wire `embed.FS` into handlers; verify single-binary deployment includes all assets — (pending merge)
|
||||
- [ ] **M6.7** Wire `embed.FS` into handlers; verify single-binary deployment includes all assets
|
||||
|
||||
**Gate:** Browser smoke on :8080: all pages render, name+month filters work, modal opens with correct data, QR loads, sync/flush work end-to-end.
|
||||
|
||||
|
||||
@@ -1,184 +0,0 @@
|
||||
# M6.7 — Single-binary embed verification
|
||||
|
||||
## Context
|
||||
|
||||
M6.7 is the final task in M6 (Go-native HTML frontend). Per the
|
||||
[progress tracker](2026-05-03-2349-go-backend-rewrite-progress.md): "Wire
|
||||
`embed.FS` into handlers; verify single-binary deployment includes all
|
||||
assets."
|
||||
|
||||
The wiring is **already in place** from M6.1 onward:
|
||||
|
||||
- [go/internal/web/assets.go](../../go/internal/web/assets.go) declares
|
||||
`//go:embed templates` → `templateFS` and `//go:embed static` → `staticFS`.
|
||||
- [go/internal/web/render.go:66](../../go/internal/web/render.go#L66)
|
||||
parses every page template via `template.New(...).ParseFS(templateFS, ...)`.
|
||||
- [go/internal/web/server.go:48-68](../../go/internal/web/server.go#L48-L68)
|
||||
serves `/static/*` via `http.FileServerFS(fs.Sub(staticFS, "static"))`.
|
||||
- [go/build/Dockerfile](../../go/build/Dockerfile) copies only the compiled
|
||||
binary into the `alpine:3` runtime image — no `templates/` or `static/`
|
||||
directory ever lands beside it.
|
||||
|
||||
What is missing is **proof** the embed is complete and stays complete:
|
||||
|
||||
1. Nothing fails the build/test if a contributor adds a new file under
|
||||
`internal/web/templates/` or `internal/web/static/` that isn't matched
|
||||
by an `//go:embed` glob (or, more realistically, adds a sibling
|
||||
directory like `static/img/` and the glob still picks it up — but a
|
||||
typo'd directive would silently drop it).
|
||||
2. No automated test exercises the `/static/*` route against the embedded
|
||||
FS — current tests in
|
||||
[html_handler_test.go](../../go/internal/web/html_handler_test.go)
|
||||
render templates (which proves `templateFS` is good) but never hit a
|
||||
static URL through the mux.
|
||||
3. The "single binary, no working-dir assets" property is undocumented —
|
||||
if it ever broke, no one would notice until the Docker image started
|
||||
500'ing in prod.
|
||||
|
||||
The intended outcome: a small test file plus a documented manual
|
||||
verification step, after which M6.7 can be ticked and M6 closed.
|
||||
|
||||
## Plan
|
||||
|
||||
### 1. Add `go/internal/web/assets_test.go`
|
||||
|
||||
One new file, two tests, no production code changes.
|
||||
|
||||
**Test A — embed completeness regression guard.** Walks the on-disk
|
||||
`templates/` and `static/` directories and asserts every regular file is
|
||||
also present in the corresponding embedded FS. Catches:
|
||||
|
||||
- A new template added without updating the `//go:embed` directive
|
||||
(current globs are `templates` and `static` — recursive by default for
|
||||
directories, so this is a low-probability regression, but the test
|
||||
doubles as living documentation of the contract).
|
||||
- A typo in the directive (e.g. someone renames `static` → `assets` in
|
||||
one place but not the other).
|
||||
|
||||
Implementation sketch:
|
||||
|
||||
```go
|
||||
func TestEmbedCompleteness(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
diskFS fs.FS // os.DirFS("templates") / os.DirFS("static")
|
||||
embed fs.FS // exported helper or via internal test in package web
|
||||
root string
|
||||
}{...}
|
||||
for _, tc := range cases {
|
||||
_ = fs.WalkDir(tc.diskFS, ".", func(path string, d fs.DirEntry, err error) error {
|
||||
if err != nil || d.IsDir() { return err }
|
||||
embPath := tc.root + "/" + path
|
||||
if _, err := fs.Stat(tc.embed, embPath); err != nil {
|
||||
t.Errorf("file %q on disk but missing in embed.FS: %v", embPath, err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Because `templateFS` and `staticFS` are unexported package vars, this
|
||||
test lives in `package web` (not `web_test`) — sibling to
|
||||
[assets.go](../../go/internal/web/assets.go). All the existing handler
|
||||
tests are in `package web_test`; that's fine, this one is internal.
|
||||
|
||||
**Test B — `/static/*` end-to-end via the mux.** Builds an `http.ServeMux`
|
||||
with the same wiring as
|
||||
[server.go:68](../../go/internal/web/server.go#L68), fires httptest
|
||||
requests, asserts:
|
||||
|
||||
- `GET /static/css/app.css` → 200, `Content-Type: text/css; charset=utf-8`,
|
||||
body contains a known string from app.css (e.g. a CSS selector).
|
||||
- `GET /static/js/member-detail.js` → 200, `Content-Type` starts with
|
||||
`text/javascript` or `application/javascript`, body non-empty.
|
||||
- `GET /static/js/payment-qr.js` → 200, body non-empty.
|
||||
- `GET /static/css/missing.css` → 404 (sanity: the file server actually
|
||||
rejects unknown paths instead of returning some default).
|
||||
|
||||
Rather than duplicate the mux assembly, factor a tiny helper (or test the
|
||||
existing mux). The cleanest move: extract `staticHandler()` from
|
||||
[server.go:48-50,68](../../go/internal/web/server.go#L48-L68) into a small
|
||||
exported-from-package function or just `staticFS` / `fs.Sub` helper, and
|
||||
have the test call it. Smallest delta: keep production code unchanged and
|
||||
replicate the two-line wiring inside the test file (acceptable — it's
|
||||
two lines and the test exists precisely to lock that contract).
|
||||
|
||||
### 2. Manual / one-shot verification (no code; documented in plan only)
|
||||
|
||||
Run once locally and tick M6.7. Command transcript:
|
||||
|
||||
```bash
|
||||
make go-build # → ./bin/fuj
|
||||
cp bin/fuj /tmp/fuj-standalone
|
||||
cd /tmp # working dir has no templates/ or static/
|
||||
./fuj-standalone server &
|
||||
SERVER_PID=$!
|
||||
sleep 1
|
||||
curl -sf http://localhost:8080/adults | grep -q "Adults Dashboard"
|
||||
curl -sf http://localhost:8080/juniors | grep -q "Juniors"
|
||||
curl -sf http://localhost:8080/payments | grep -q "Payments Ledger"
|
||||
curl -sf -o /tmp/app.css http://localhost:8080/static/css/app.css \
|
||||
&& test -s /tmp/app.css
|
||||
curl -sf -o /tmp/qr.js http://localhost:8080/static/js/payment-qr.js \
|
||||
&& test -s /tmp/qr.js
|
||||
kill $SERVER_PID
|
||||
```
|
||||
|
||||
`fuj server` will fail to talk to Sheets without credentials, so the
|
||||
`/adults` etc. pages will render with the `Error` field set — that's
|
||||
fine; the assertion is that the **template + static asset pipeline** is
|
||||
self-contained, not that data loads. Each curl above only checks for
|
||||
markup present in every render path (header text and stylesheet body).
|
||||
|
||||
### 3. Tracker + changelog
|
||||
|
||||
- Tick `M6.7` in
|
||||
[docs/plans/2026-05-03-2349-go-backend-rewrite-progress.md:120](2026-05-03-2349-go-backend-rewrite-progress.md#L120),
|
||||
append the merge SHA on the line.
|
||||
- Mark "Last updated" date and bump milestone status: M6 complete, next is
|
||||
M7.
|
||||
- Append a `CHANGELOG.md` entry per CLAUDE.md convention (`date "+%Y-%m-%d %H:%M %Z"`).
|
||||
|
||||
## Files touched
|
||||
|
||||
| File | Change |
|
||||
| --- | --- |
|
||||
| [go/internal/web/assets_test.go](../../go/internal/web/assets_test.go) | **new** — two tests (embed completeness + `/static/*` mux) |
|
||||
| [docs/plans/2026-05-03-2349-go-backend-rewrite-progress.md](2026-05-03-2349-go-backend-rewrite-progress.md) | tick M6.7, bump "last updated" |
|
||||
| [CHANGELOG.md](../../CHANGELOG.md) | new top entry |
|
||||
|
||||
No production source files change. (If extracting the static handler
|
||||
reads cleaner, a 4-line refactor in
|
||||
[server.go](../../go/internal/web/server.go) is acceptable but optional.)
|
||||
|
||||
## Branch + MR
|
||||
|
||||
Per project convention this is a feature, so:
|
||||
|
||||
```bash
|
||||
git checkout -b feat/go-m6-7-embed-verify
|
||||
# … commits …
|
||||
git push -u origin feat/go-m6-7-embed-verify
|
||||
tea pr create --title "feat(go): M6.7 — single-binary embed verification" \
|
||||
--description "<short body referencing M6.7>" --base main \
|
||||
--head feat/go-m6-7-embed-verify
|
||||
```
|
||||
|
||||
## Verification
|
||||
|
||||
After implementation:
|
||||
|
||||
1. `make go-test` → green (new `TestEmbedCompleteness` and `TestStaticAssetsServed` pass).
|
||||
2. `make go-lint` → clean.
|
||||
3. Run the manual transcript in §2 above — all curls succeed, no
|
||||
"template not found" or 404 on static assets.
|
||||
4. `make go-build && docker build -f go/build/Dockerfile -t fuj-go:m6-7 go/` →
|
||||
succeeds; `docker run --rm -p 8080:8080 fuj-go:m6-7` serves `/adults`
|
||||
with stylesheet attached (visual smoke test in browser).
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Re-architecting how templates are parsed or served.
|
||||
- Compressing / fingerprinting static assets (a separate concern).
|
||||
- Live integration test with real Sheets data — covered later in M7.
|
||||
@@ -1,93 +0,0 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"io/fs"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestEmbedCompleteness guards against a new template or static file being
|
||||
// added to disk but missing from the embedded FS (e.g. a new directory that
|
||||
// the //go:embed glob does not match).
|
||||
func TestEmbedCompleteness(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
diskDir string
|
||||
embedFS fs.FS
|
||||
embedRoot string
|
||||
}{
|
||||
{"templates", "templates", templateFS, "templates"},
|
||||
{"static", "static", staticFS, "static"},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
diskFS := os.DirFS(tc.diskDir)
|
||||
_ = fs.WalkDir(diskFS, ".", func(path string, d fs.DirEntry, err error) error {
|
||||
if err != nil || d.IsDir() {
|
||||
return err
|
||||
}
|
||||
embPath := tc.embedRoot + "/" + path
|
||||
if _, statErr := fs.Stat(tc.embedFS, embPath); statErr != nil {
|
||||
t.Errorf("file %q exists on disk but is missing from embed.FS (%v)", embPath, statErr)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestStaticAssetsServed verifies that /static/* is served from the embedded
|
||||
// FS through the same mux wiring used in server.go, so a standalone binary
|
||||
// with no adjacent static/ directory still delivers assets.
|
||||
func TestStaticAssetsServed(t *testing.T) {
|
||||
subFS, err := fs.Sub(staticFS, "static")
|
||||
if err != nil {
|
||||
t.Fatalf("fs.Sub static: %v", err)
|
||||
}
|
||||
mux := http.NewServeMux()
|
||||
mux.Handle("/static/", http.StripPrefix("/static/", http.FileServerFS(subFS)))
|
||||
|
||||
cases := []struct {
|
||||
path string
|
||||
wantCT string
|
||||
wantSnippet string
|
||||
}{
|
||||
{"/static/css/app.css", "text/css", "body {"},
|
||||
{"/static/js/member-detail.js", "javascript", "Member-detail modal"},
|
||||
{"/static/js/filters.js", "javascript", ""},
|
||||
{"/static/js/payment-qr.js", "javascript", ""},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.path, func(t *testing.T) {
|
||||
req := httptest.NewRequest(http.MethodGet, tc.path, nil)
|
||||
w := httptest.NewRecorder()
|
||||
mux.ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("GET %s: status %d, want 200", tc.path, w.Code)
|
||||
}
|
||||
ct := w.Header().Get("Content-Type")
|
||||
if !strings.Contains(ct, tc.wantCT) {
|
||||
t.Errorf("GET %s: Content-Type %q, want it to contain %q", tc.path, ct, tc.wantCT)
|
||||
}
|
||||
if tc.wantSnippet != "" && !strings.Contains(w.Body.String(), tc.wantSnippet) {
|
||||
t.Errorf("GET %s: body missing expected snippet %q", tc.path, tc.wantSnippet)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Sanity: unknown path → 404 (file server doesn't fall through silently)
|
||||
t.Run("missing-file", func(t *testing.T) {
|
||||
req := httptest.NewRequest(http.MethodGet, "/static/css/nonexistent.css", nil)
|
||||
w := httptest.NewRecorder()
|
||||
mux.ServeHTTP(w, req)
|
||||
if w.Code != http.StatusNotFound {
|
||||
t.Errorf("unknown static path: status %d, want 404", w.Code)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -4,6 +4,7 @@
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.request
|
||||
from datetime import datetime
|
||||
from html.parser import HTMLParser
|
||||
@@ -89,9 +90,11 @@ def parse_czech_amount(s: str) -> float | None:
|
||||
|
||||
|
||||
def parse_czech_date(s: str) -> str | None:
|
||||
"""Parse 'DD.MM.YYYY' to 'YYYY-MM-DD'."""
|
||||
"""Parse a Czech date to 'YYYY-MM-DD'. Accepts 4-digit and 2-digit years
|
||||
with dot or slash separators; Fio's transparent page mixes 'DD.MM.YYYY'
|
||||
and 'DD.MM.YY' in the same response."""
|
||||
s = s.strip()
|
||||
for fmt in ("%d.%m.%Y", "%d/%m/%Y"):
|
||||
for fmt in ("%d.%m.%Y", "%d/%m/%Y", "%d.%m.%y", "%d/%m/%y"):
|
||||
try:
|
||||
return datetime.strptime(s, fmt).strftime("%Y-%m-%d")
|
||||
except ValueError:
|
||||
@@ -146,6 +149,7 @@ def fetch_transactions_transparent(
|
||||
"bank_id": "", # HTML scraping doesn't give stable ID
|
||||
})
|
||||
|
||||
print(f"fio: transparent fetched {len(rows)} raw rows, {len(transactions)} transaction(s) after filtering", file=sys.stderr)
|
||||
return transactions
|
||||
|
||||
|
||||
@@ -169,7 +173,8 @@ def fetch_transactions_api(
|
||||
|
||||
transactions = []
|
||||
tx_list = data.get("accountStatement", {}).get("transactionList", {})
|
||||
for tx in (tx_list.get("transaction") or []):
|
||||
raw_list = tx_list.get("transaction") or []
|
||||
for tx in raw_list:
|
||||
# Each field is {"value": ..., "name": ..., "id": ...} or null
|
||||
def val(col_id):
|
||||
col = tx.get(f"column{col_id}")
|
||||
@@ -197,6 +202,7 @@ def fetch_transactions_api(
|
||||
"currency": str(val(14) or "CZK"), # column14 = Currency
|
||||
})
|
||||
|
||||
print(f"fio: api fetched {len(raw_list)} raw transaction(s), {len(transactions)} after filtering", file=sys.stderr)
|
||||
return transactions
|
||||
|
||||
|
||||
@@ -204,8 +210,14 @@ def fetch_transactions(date_from: str, date_to: str) -> list[dict]:
|
||||
"""Fetch transactions, using API if token available, else transparent page."""
|
||||
token = os.environ.get("FIO_API_TOKEN", "").strip()
|
||||
if token:
|
||||
print(f"fio: using authenticated API, window {date_from}..{date_to}", file=sys.stderr)
|
||||
return fetch_transactions_api(token, date_from, date_to)
|
||||
|
||||
print(
|
||||
f"fio: using transparent page (FIO_API_TOKEN unset — expect publishing lag), "
|
||||
f"window {date_from}..{date_to}, account=2800359168",
|
||||
file=sys.stderr,
|
||||
)
|
||||
# Convert YYYY-MM-DD to DD.MM.YYYY for the transparent page URL
|
||||
from_dt = datetime.strptime(date_from, "%Y-%m-%d")
|
||||
to_dt = datetime.strptime(date_to, "%Y-%m-%d")
|
||||
|
||||
@@ -77,6 +77,35 @@ def generate_sync_id(tx: dict) -> str:
|
||||
return hashlib.sha256(raw_str.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def _trunc(s: str, n: int = 40) -> str:
|
||||
s = str(s)
|
||||
return s if len(s) <= n else s[: n - 1] + "…"
|
||||
|
||||
|
||||
def _print_fio_table(transactions: list[dict], statuses: list[str]) -> None:
|
||||
headers = ["DATE", "AMOUNT", "SENDER", "VS", "MESSAGE", "BANKID", "STATUS"]
|
||||
rows = [
|
||||
[
|
||||
str(tx.get("date", "")),
|
||||
f"{float(tx.get('amount', 0)):.2f}",
|
||||
str(tx.get("sender", "")),
|
||||
str(tx.get("vs", "")),
|
||||
_trunc(str(tx.get("message", ""))),
|
||||
str(tx.get("bank_id", "")),
|
||||
status,
|
||||
]
|
||||
for tx, status in zip(transactions, statuses)
|
||||
]
|
||||
widths = [
|
||||
max(len(headers[i]), max((len(r[i]) for r in rows), default=0))
|
||||
for i in range(len(headers))
|
||||
]
|
||||
sep = " "
|
||||
print(sep.join(h.ljust(w) for h, w in zip(headers, widths)))
|
||||
for row in rows:
|
||||
print(sep.join(cell.ljust(w) for cell, w in zip(row, widths)))
|
||||
|
||||
|
||||
def sort_sheet_by_date(service, spreadsheet_id):
|
||||
"""Sort the sheet by the Date column (Column B)."""
|
||||
# Get the sheet ID (gid) of the first sheet
|
||||
@@ -104,12 +133,21 @@ def sort_sheet_by_date(service, spreadsheet_id):
|
||||
print("Sheet sorted by date.")
|
||||
|
||||
|
||||
def sync_to_sheets(spreadsheet_id: str, credentials_path: str, days: int = None, date_from_str: str = None, date_to_str: str = None, sort_by_date: bool = False):
|
||||
def sync_to_sheets(
|
||||
spreadsheet_id: str,
|
||||
credentials_path: str,
|
||||
days: int = None,
|
||||
date_from_str: str = None,
|
||||
date_to_str: str = None,
|
||||
sort_by_date: bool = False,
|
||||
dry_run: bool = False,
|
||||
print_fio_table: bool = False,
|
||||
):
|
||||
print(f"Connecting to Google Sheets using {credentials_path}...")
|
||||
service = get_sheets_service(credentials_path)
|
||||
sheet = service.spreadsheets()
|
||||
|
||||
# 1. Fetch existing IDs from Column G (last column in A-G range)
|
||||
# 1. Read existing sync IDs from Column K
|
||||
print(f"Reading existing sync IDs from sheet...")
|
||||
try:
|
||||
result = sheet.values().get(
|
||||
@@ -117,19 +155,22 @@ def sync_to_sheets(spreadsheet_id: str, credentials_path: str, days: int = None,
|
||||
range="A1:K" # Include header and all columns to check Sync ID
|
||||
).execute()
|
||||
values = result.get("values", [])
|
||||
|
||||
|
||||
# Check and insert labels if missing
|
||||
if not values or values[0] != COLUMN_LABELS:
|
||||
print("Inserting column labels...")
|
||||
sheet.values().update(
|
||||
spreadsheetId=spreadsheet_id,
|
||||
range="A1",
|
||||
valueInputOption="USER_ENTERED",
|
||||
body={"values": [COLUMN_LABELS]}
|
||||
).execute()
|
||||
if dry_run:
|
||||
print("Dry run: would write header row")
|
||||
else:
|
||||
print("Inserting column labels...")
|
||||
sheet.values().update(
|
||||
spreadsheetId=spreadsheet_id,
|
||||
range="A1",
|
||||
valueInputOption="USER_ENTERED",
|
||||
body={"values": [COLUMN_LABELS]}
|
||||
).execute()
|
||||
existing_ids = set()
|
||||
else:
|
||||
# Sync ID is now the last column (index 10)
|
||||
# Sync ID is the last column (index 10)
|
||||
existing_ids = {row[10] for row in values[1:] if len(row) > 10}
|
||||
except Exception as e:
|
||||
print(f"Error reading sheet (maybe empty?): {e}")
|
||||
@@ -150,8 +191,12 @@ def sync_to_sheets(spreadsheet_id: str, credentials_path: str, days: int = None,
|
||||
transactions = fetch_transactions(df_str, dt_str)
|
||||
print(f"Found {len(transactions)} transactions.")
|
||||
|
||||
# 3. Filter for new transactions
|
||||
if dry_run:
|
||||
print(f"Dry run: window {df_str} to {dt_str}, fetched {len(transactions)} transaction(s) from Fio")
|
||||
|
||||
# 3. Determine NEW/DUP for each transaction
|
||||
new_rows = []
|
||||
tx_statuses = []
|
||||
for tx in transactions:
|
||||
sync_id = generate_sync_id(tx)
|
||||
if sync_id not in existing_ids:
|
||||
@@ -169,24 +214,48 @@ def sync_to_sheets(spreadsheet_id: str, credentials_path: str, days: int = None,
|
||||
tx.get("bank_id", ""),
|
||||
sync_id,
|
||||
])
|
||||
tx_statuses.append("NEW")
|
||||
else:
|
||||
tx_statuses.append("DUP")
|
||||
|
||||
# 4. Print table (before early-return so all transactions are shown including DUPs)
|
||||
if print_fio_table and transactions:
|
||||
_print_fio_table(transactions, tx_statuses)
|
||||
|
||||
if not new_rows:
|
||||
print("No new transactions to sync.")
|
||||
if dry_run:
|
||||
print("Dry run: would sync 0 new transaction(s).")
|
||||
else:
|
||||
print("No new transactions to sync.")
|
||||
return
|
||||
|
||||
# 4. Append to sheet
|
||||
print(f"Appending {len(new_rows)} new transactions to the sheet...")
|
||||
body = {"values": new_rows}
|
||||
sheet.values().append(
|
||||
spreadsheetId=spreadsheet_id,
|
||||
range="A2", # Appends to the end of the sheet
|
||||
valueInputOption="USER_ENTERED",
|
||||
body=body
|
||||
).execute()
|
||||
print("Sync completed successfully.")
|
||||
|
||||
if sort_by_date:
|
||||
sort_sheet_by_date(service, spreadsheet_id)
|
||||
# 5. Append to sheet or print dry-run would-write lines
|
||||
if dry_run:
|
||||
for tx, status in zip(transactions, tx_statuses):
|
||||
if status == "NEW":
|
||||
print(
|
||||
f"Dry run: would append"
|
||||
f" date={tx.get('date', '')}"
|
||||
f" amount={tx.get('amount', '')}"
|
||||
f" sender={tx.get('sender', '')}"
|
||||
f" vs={tx.get('vs', '')}"
|
||||
f" message={tx.get('message', '')}"
|
||||
)
|
||||
if sort_by_date:
|
||||
print("Dry run: would sort by date")
|
||||
print(f"Dry run: would sync {len(new_rows)} new transaction(s).")
|
||||
else:
|
||||
print(f"Appending {len(new_rows)} new transactions to the sheet...")
|
||||
body = {"values": new_rows}
|
||||
sheet.values().append(
|
||||
spreadsheetId=spreadsheet_id,
|
||||
range="A2", # Appends to the end of the sheet
|
||||
valueInputOption="USER_ENTERED",
|
||||
body=body
|
||||
).execute()
|
||||
print("Sync completed successfully.")
|
||||
if sort_by_date:
|
||||
sort_sheet_by_date(service, spreadsheet_id)
|
||||
|
||||
|
||||
def main():
|
||||
@@ -197,16 +266,20 @@ def main():
|
||||
parser.add_argument("--from", dest="date_from", help="Start date YYYY-MM-DD")
|
||||
parser.add_argument("--to", dest="date_to", help="End date YYYY-MM-DD")
|
||||
parser.add_argument("--sort-by-date", action="store_true", help="Sort the sheet by date after sync")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Fetch and dedup without writing to the sheet")
|
||||
parser.add_argument("--print-fio-table", action="store_true", help="Print aligned table of all fetched transactions with NEW/DUP status (use with --dry-run)")
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
sync_to_sheets(
|
||||
spreadsheet_id=args.sheet_id,
|
||||
spreadsheet_id=args.sheet_id,
|
||||
credentials_path=args.credentials,
|
||||
days=args.days,
|
||||
date_from_str=args.date_from,
|
||||
date_to_str=args.date_to,
|
||||
sort_by_date=args.sort_by_date
|
||||
sort_by_date=args.sort_by_date,
|
||||
dry_run=args.dry_run,
|
||||
print_fio_table=args.print_fio_table,
|
||||
)
|
||||
except Exception as e:
|
||||
print(f"Sync failed: {e}")
|
||||
|
||||
Reference in New Issue
Block a user