From 9b99f6d33b27842881d0c29249482eb87f9392e1 Mon Sep 17 00:00:00 2001 From: Jan Novak Date: Wed, 11 Mar 2026 11:57:30 +0100 Subject: [PATCH] docs: experiment with generated documentation, let's keep it in git for now --- docs/README.md | 15 ++ docs/_sidebar.md | 25 +++ docs/by-claude-opus/README.md | 214 ++++++++++++++++++ docs/by-claude-opus/architecture.md | 268 +++++++++++++++++++++++ docs/by-claude-opus/data-model.md | 201 +++++++++++++++++ docs/by-claude-opus/deployment.md | 198 +++++++++++++++++ docs/by-claude-opus/development.md | 228 +++++++++++++++++++ docs/by-claude-opus/scripts.md | 325 ++++++++++++++++++++++++++++ docs/by-claude-opus/testing.md | 145 +++++++++++++ docs/by-claude-opus/user-guide.md | 166 ++++++++++++++ docs/by-claude-opus/web-app.md | 256 ++++++++++++++++++++++ docs/by-gemini/README.md | 36 +++ docs/by-gemini/architecture.md | 48 ++++ docs/by-gemini/deployment.md | 72 ++++++ docs/by-gemini/scripts.md | 66 ++++++ docs/by-gemini/user-guide.md | 61 ++++++ docs/index.html | 43 ++++ 17 files changed, 2367 insertions(+) create mode 100644 docs/README.md create mode 100644 docs/_sidebar.md create mode 100644 docs/by-claude-opus/README.md create mode 100644 docs/by-claude-opus/architecture.md create mode 100644 docs/by-claude-opus/data-model.md create mode 100644 docs/by-claude-opus/deployment.md create mode 100644 docs/by-claude-opus/development.md create mode 100644 docs/by-claude-opus/scripts.md create mode 100644 docs/by-claude-opus/testing.md create mode 100644 docs/by-claude-opus/user-guide.md create mode 100644 docs/by-claude-opus/web-app.md create mode 100644 docs/by-gemini/README.md create mode 100644 docs/by-gemini/architecture.md create mode 100644 docs/by-gemini/deployment.md create mode 100644 docs/by-gemini/scripts.md create mode 100644 docs/by-gemini/user-guide.md create mode 100644 docs/index.html diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..3338afe --- /dev/null +++ b/docs/README.md @@ -0,0 +1,15 @@ +# FUJ Management Documentation + +Welcome to the documentation for the FUJ Management application. + +This project automates financial and operational management for the FUJ (Frisbee Ultimate Jablonec) club. + +## Navigation + +Use the sidebar to explore the documentation: + +* **[Project Notes](project-notes.md)**: Main brainstorming and domain model. +* **[Scripts](scripts.md)**: Details about available CLI tools. +* **[Fee Specification](fee-calculation-spec.md)**: Rules for fee calculation. + +For more technical details, check out the guides by Claude and Gemini in the sidebar. diff --git a/docs/_sidebar.md b/docs/_sidebar.md new file mode 100644 index 0000000..ed8215e --- /dev/null +++ b/docs/_sidebar.md @@ -0,0 +1,25 @@ +* [Home](README.md) +* [Project Notes](project-notes.md) +* [Scripts](scripts.md) +* [Fee Spec](fee-calculation-spec.md) + +* **By Claude Opus** + * [README](by-claude-opus/README.md) + * [User Guide](by-claude-opus/user-guide.md) + * [Web App](by-claude-opus/web-app.md) + * [Deployment](by-claude-opus/deployment.md) + * [Architecture](by-claude-opus/architecture.md) + * [Data Model](by-claude-opus/data-model.md) + * [Development](by-claude-opus/development.md) + * [Scripts](by-claude-opus/scripts.md) + * [Testing](by-claude-opus/testing.md) + +* **By Gemini** + * [README](by-gemini/README.md) + * [User Guide](by-gemini/user-guide.md) + * [Architecture](by-gemini/architecture.md) + * [Deployment](by-gemini/deployment.md) + * [Scripts](by-gemini/scripts.md) + +* **Specs** + * [Fio Sync](spec/fio_to_sheets_sync.md) diff --git a/docs/by-claude-opus/README.md b/docs/by-claude-opus/README.md new file mode 100644 index 0000000..fc95177 --- /dev/null +++ b/docs/by-claude-opus/README.md @@ -0,0 +1,214 @@ +# FUJ Management — Comprehensive Documentation + +> **FUJ = Frisbee Ultimate Jablonec** — a small sports club in the Czech Republic. + +## What Is This Project? + +FUJ Management is a purpose-built financial management system for a small ultimate frisbee club. It automates the tedious process of tracking **who attended practice**, **how much they owe**, **who has paid**, and **who still owes money** — a workflow that would otherwise require manual cross-referencing between attendance spreadsheets and bank statements. + +The system is built around two Google Sheets (one for attendance, one for payments) and a Fio bank transparent account. A set of Python scripts sync and process the data, while a Flask-based web dashboard provides real-time visibility into fees, payments, and reconciliation status. + +### The Problem It Solves + +Before this system, the club treasurer had to: + +1. **Manually count** attendance marks for each member each month +2. **Calculate** whether each person owes 0, 200, or 750 CZK based on how many times they showed up +3. **Cross-reference** bank statements to figure out who paid and for which month +4. **Chase** members who hadn't paid, often losing track of partial payments and advance payments +5. **Handle edge cases** like members paying for multiple months at once, using nicknames in payment messages, or paying via a family member's account + +This system automates steps 1–4 entirely, and provides tooling for step 5. + +## System Overview + +``` +┌──────────────────────────┐ ┌──────────────────────────┐ +│ Attendance Sheet │ │ Fio Bank Account │ +│ (Google Sheets) │ │ (transparent account) │ +│ │ │ │ +│ Members × Dates × ✓/✗ │ │ Incoming payments with │ +│ Tier (A/J/X) │ │ sender, amount, message │ +└──────────┬───────────────┘ └──────────┬───────────────┘ + │ │ + │ CSV export │ API / HTML scraping + │ │ + ▼ ▼ + ┌─────────────────┐ ┌───────────────────────┐ + │ attendance.py │ │ sync_fio_to_sheets.py │ + │ │ │ │ + │ Fetches sheet, │ │ Syncs bank txns to │ + │ computes fees │ │ Payments Google Sheet │ + └────────┬────────┘ └───────────┬────────────┘ + │ │ + │ ▼ + │ ┌───────────────────────┐ + │ │ Payments Sheet │ + │ │ (Google Sheets) │ + │ │ │ + │ │ Date|Amount|Person| │ + │ │ Purpose|Sender|etc. │ + │ └───────────┬────────────┘ + │ │ + │ ┌─────────────────────────┤ + │ │ │ + │ ▼ ▼ + │ ┌──────────────┐ ┌──────────────────┐ + │ │infer_payments│ │ match_payments.py │ + │ │ .py │ │ │ + │ │ │ │ Reconciliation │ + │ │ Auto-fills │ │ engine: matches │ + │ │ Person, │ │ payments against │ + │ │ Purpose, │ │ expected fees │ + │ │ Amount │ └────────┬──────────┘ + │ └──────────────┘ │ + │ │ + └────────────────┬───────────────┘ + │ + ▼ + ┌──────────────────────┐ + │ Flask Web App │ + │ (app.py) │ + │ │ + │ /fees – fee │ + │ table │ + │ /reconcile – balance │ + │ matrix │ + │ /payments – ledger │ + │ /qr – QR code │ + └───────────────────────┘ +``` + +## Quick Start + +### Prerequisites + +- **Python 3.13+** +- **[uv](https://docs.astral.sh/uv/)** — fast Python package manager +- **Google Sheets API credentials** — a service account JSON file placed at `.secret/fuj-management-bot-credentials.json` +- *Optional*: `FIO_API_TOKEN` environment variable for Fio REST API access (falls back to transparent page scraping) + +### Setup + +```bash +# Clone and install dependencies +git clone +cd fuj-management +uv sync # Installs all dependencies from pyproject.toml + +# Place your Google API credentials +mkdir -p .secret +cp /path/to/your/credentials.json .secret/fuj-management-bot-credentials.json +``` + +### Common Operations + +| Command | Purpose | +|---------|---------| +| `make web` | Start the web dashboard at `http://localhost:5001` | +| `make sync` | Pull new bank transactions into the Google Sheet | +| `make infer` | Auto-fill Person/Purpose/Amount for new transactions | +| `make reconcile` | Print a CLI balance report | +| `make fees` | Print fee calculation table from attendance | +| `make test` | Run the test suite | +| `make image` | Build the Docker container image | + +### Typical Workflow + +``` +make sync → make infer → (manual review in Google Sheets) → make web + ↓ ↓ ↓ ↓ +Pull new bank Auto-match Fix any [?] View live + transactions payments to flagged rows dashboard + into sheet members/months in the sheet +``` + +## Documentation Index + +| Document | Contents | +|----------|----------| +| [Architecture](architecture.md) | System design, data flow diagrams, module dependency graph | +| [Web Application](web-app.md) | Flask app architecture, routes, templates, interactive features | +| [User Guide](user-guide.md) | End-user guide for the web dashboard — what each page shows | +| [Scripts Reference](scripts.md) | Detailed reference for all CLI scripts and shared modules | +| [Data Model](data-model.md) | Google Sheets schemas, fee calculation rules, bank integration | +| [Deployment](deployment.md) | Docker containerization, Gitea CI/CD, Kubernetes deployment | +| [Testing](testing.md) | Test infrastructure, coverage, how to write new tests | +| [Development Guide](development.md) | Local setup, coding conventions, tooling, project history | + +## Technology Stack + +| Layer | Technology | +|-------|-----------| +| Language | Python 3.13+ | +| Web framework | Flask 3.1 | +| Package management | uv + pyproject.toml | +| Data sources | Google Sheets API, Fio Bank API / HTML scraping | +| QR codes | `qrcode` library (PIL backend) | +| Containerization | Docker (Alpine-based) | +| CI/CD | Gitea Actions | +| Deployment target | Self-hosted Kubernetes | +| Frontend | Server-rendered HTML/CSS/JS (terminal-aesthetic theme) | + +## Project Structure + +``` +fuj-management/ +├── app.py # Flask web application (4 routes) +├── Makefile # Build automation (13 targets) +├── pyproject.toml # Python dependencies and metadata +│ +├── scripts/ +│ ├── attendance.py # Shared: attendance data + fee calculation +│ ├── calculate_fees.py # CLI: print fee table +│ ├── match_payments.py # Core: reconciliation engine + CLI report +│ ├── infer_payments.py # Auto-fill Person/Purpose in Google Sheet +│ ├── sync_fio_to_sheets.py # Sync Fio bank → Google Sheet +│ ├── fio_utils.py # Shared: Fio bank data fetching +│ └── czech_utils.py # Shared: diacritics normalization + Czech month parsing +│ +├── templates/ +│ ├── fees.html # Attendance/fees dashboard +│ ├── reconcile.html # Payment reconciliation with modals + QR +│ └── payments.html # Payments ledger grouped by member +│ +├── tests/ +│ ├── test_app.py # Flask route tests (mocked data) +│ └── test_reconcile_exceptions.py # Reconciliation with fee exceptions +│ +├── build/ +│ ├── Dockerfile # Alpine-based container image +│ └── entrypoint.sh # Container entry point +│ +├── .gitea/workflows/ +│ ├── build.yaml # CI: build + push Docker image +│ └── kubernetes-deploy.yaml # CD: deploy to K8s cluster +│ +├── .secret/ # (gitignored) API credentials +├── docs/ # Project documentation +│ ├── project-notes.md # Original brainstorming and design notes +│ ├── fee-calculation-spec.md # Fee rules and payment matching spec +│ ├── scripts.md # Legacy scripts documentation +│ └── spec/ +│ └── fio_to_sheets_sync.md # Fio-to-Sheets sync specification +│ +└── CLAUDE.md # AI assistant context file +``` + +## Key Design Decisions + +1. **No database** — Google Sheets serves as both the data store and the manual editing interface. This keeps the system simple and accessible to non-technical club members who can review and edit data directly in the spreadsheet. + +2. **PII separation** — No member names or personal data are stored in the git repository. All data is fetched at runtime from Google Sheets and the bank account. + +3. **Idempotent sync** — The Fio-to-Sheets sync uses SHA-256 hashes as deduplication keys, making re-runs safe and append-only. + +4. **Graceful fallbacks** — Bank data can be fetched via the REST API (if a token is available) or by scraping the public transparent account page. The system doesn't break if the API token is missing. + +5. **Czech language support** — Payment messages are in Czech and use diacritics. The system normalizes text (strips diacritics) and understands Czech month names in all grammatical declensions. + +6. **Terminal aesthetic** — The web dashboard uses a monospace, dark-themed, terminal-inspired design that matches the project's pragmatic, CLI-first philosophy. + +--- + +*This documentation was generated on 2026-03-03 by Claude Opus, based on a comprehensive analysis of the complete codebase.* diff --git a/docs/by-claude-opus/architecture.md b/docs/by-claude-opus/architecture.md new file mode 100644 index 0000000..78cdc02 --- /dev/null +++ b/docs/by-claude-opus/architecture.md @@ -0,0 +1,268 @@ +# System Architecture + +## Overview + +FUJ Management follows a **pipeline architecture** where data flows from external sources (Google Sheets, Fio Bank) through processing scripts into a web dashboard. There is no central database — Google Sheets serves as the persistent data store, and the Flask app renders views by fetching and processing data on every request. + +## Component Architecture + +``` + ┌─────────────────────────────────────────────────┐ + │ EXTERNAL DATA SOURCES │ + │ │ + │ ┌──────────────────┐ ┌──────────────────────┐ │ + │ │ Attendance Sheet │ │ Fio Bank Account │ │ + │ │ (Google Sheets) │ │ │ │ + │ │ │ │ ┌────────────────┐ │ │ + │ │ ID: 1E2e_gT... │ │ │ REST API │ │ │ + │ │ │ │ │ (JSON, w/token)│ │ │ + │ │ CSV export (pub) │ │ ├────────────────┤ │ │ + │ │ │ │ │ Transparent │ │ │ + │ └────────┬─────────┘ │ │ page (HTML) │ │ │ + │ │ │ └───────┬────────┘ │ │ + │ │ └──────────┼──────────┘ │ + └───────────┼───────────────────────┼────────────┘ + │ │ + ─ ─ ─ ─ ─ ─ ┼ ─ ─ DATA INGESTION ─ ┼ ─ ─ ─ ─ ─ + │ │ + ┌───────────▼──────┐ ┌───────────▼──────────┐ + │ attendance.py │ │ fio_utils.py │ + │ │ │ │ + │ fetch_csv() │ │ fetch_transactions() │ + │ parse_dates() │ │ FioTableParser │ + │ group_by_month() │ │ parse_czech_amount() │ + │ calculate_fee() │ │ parse_czech_date() │ + │ get_members() │ │ │ + │ get_members_ │ │ API + HTML fallback │ + │ with_fees() │ │ │ + └───────────┬──────┘ └───────────┬──────────┘ + │ │ + ─ ─ ─ ─ ─ ─ ┼ ─ ─ PROCESSING ─ ─ ─ ┼ ─ ─ ─ ─ ─ + │ │ + │ ┌─────────────▼──────────┐ + │ │ sync_fio_to_sheets.py │ ──▶ Payments Sheet + │ │ │ (Google Sheets) + │ │ generate_sync_id() │ + │ │ sort_sheet_by_date() │ + │ │ get_sheets_service() │ + │ └────────────────────────┘ + │ │ + │ ┌─────────────▼──────────┐ + │ │ infer_payments.py │ ──▶ Writes back to + │ │ │ Payments Sheet + │ │ infer Person/Purpose/ │ + │ │ Amount for empty rows │ + │ └────────────────────────┘ + │ │ + │ ┌──────────────────▼──────────┐ + │ │ czech_utils.py │ + │ │ │ + │ │ normalize() — strip │ + │ │ diacritics │ + │ │ parse_month_references() │ + │ │ CZECH_MONTHS dict │ + │ └─────────────────────────────┘ + │ │ + ─ ─ ─ ─ ─ ─ ┼ ─ RECONCILIATION ─ ─┼ ─ ─ ─ ─ ─ + │ │ + ┌─────────▼───────────────────────▼───────────┐ + │ match_payments.py │ + │ │ + │ _build_name_variants() — name matching │ + │ match_members() — fuzzy match │ + │ infer_transaction_details() │ + │ fetch_sheet_data() — read payments │ + │ fetch_exceptions() — fee overrides │ + │ reconcile() — CORE ENGINE │ + │ print_report() — CLI output │ + └──────────────────────┬──────────────────────┘ + │ + ─ ─ ─ ─ ─ ─ ─ PRESENTATION ┼ ─ ─ ─ ─ ─ ─ ─ ─ ─ + │ + ┌──────────────────────▼──────────────────────┐ + │ app.py (Flask) │ + │ │ + │ GET / → redirect to /fees │ + │ GET /fees → fees.html │ + │ GET /reconcile → reconcile.html │ + │ GET /payments → payments.html │ + │ GET /qr → PNG QR code (SPD format) │ + └─────────────────────────────────────────────┘ +``` + +## Module Dependency Graph + +``` +app.py + ├── attendance.py + │ └── (stdlib: csv, urllib, datetime) + └── match_payments.py + ├── attendance.py + ├── czech_utils.py + │ └── (stdlib: re, unicodedata) + └── sync_fio_to_sheets.py (for get_sheets_service, DEFAULT_SPREADSHEET_ID) + └── fio_utils.py + └── (stdlib: json, urllib, html.parser, datetime) + +infer_payments.py + ├── sync_fio_to_sheets.py + ├── match_payments.py + └── attendance.py + +calculate_fees.py + └── attendance.py +``` + +### Import Relationships + +| Module | Imports from | +|--------|-------------| +| `app.py` | `attendance` (`get_members_with_fees`, `SHEET_ID`), `match_payments` (`reconcile`, `fetch_sheet_data`, `fetch_exceptions`, `normalize`, `DEFAULT_SPREADSHEET_ID`) | +| `match_payments.py` | `attendance` (`get_members_with_fees`), `czech_utils` (`normalize`, `parse_month_references`), `sync_fio_to_sheets` (`get_sheets_service`, `DEFAULT_SPREADSHEET_ID`) | +| `infer_payments.py` | `sync_fio_to_sheets` (`get_sheets_service`, `DEFAULT_SPREADSHEET_ID`), `match_payments` (`infer_transaction_details`), `attendance` (`get_members_with_fees`) | +| `sync_fio_to_sheets.py` | `fio_utils` (`fetch_transactions`) | +| `calculate_fees.py` | `attendance` (`get_members_with_fees`) | + +## Data Flow Patterns + +### Pattern 1: Sync & Enrich (Batch Pipeline) + +This is the primary workflow for keeping the payments ledger up to date: + +``` +1. make sync 2. make infer + ┌──────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ + │ Fio │───▶│ Payments │ │ Payments │───▶│ Payments │ + │ Bank │ │ Sheet │ │ Sheet │ │ Sheet │ + └──────┘ │ (append) │ │ (read) │ │ (update) │ + └──────────┘ └──────────┘ └──────────┘ + + - Fetches last 30 days - Reads empty Person/Purpose rows + - SHA-256 dedup prevents - Uses name matching + Czech month + duplicate entries parsing to auto-fill + - Marks uncertain matches with [?] +``` + +### Pattern 2: Real-Time Rendering (Web Dashboard) + +Every web request triggers a fresh data fetch — no caching layer exists: + +``` +Browser Request → Flask Route → Fetch (Google Sheets API/CSV) → Process → Render HTML + │ │ + │ attendance.py │ reconcile() + │ fetch_sheet_data() │ or direct + │ fetch_exceptions() │ formatting + ▼ ▼ + ~1-3 seconds Template with + (network I/O) inline CSS + JS +``` + +### Pattern 3: QR Code Generation (On-Demand) + +``` +Browser clicks "Pay" → GET /qr?account=...&amount=...&message=... → SPD QR PNG + │ + qrcode lib + generates + in-memory PNG +``` + +## Key Design Patterns + +### 1. Google Sheets as Database + +Instead of a traditional database, the system uses two Google Sheets: + +| Sheet | Purpose | Access Method | +|-------|---------|---------------| +| Attendance Sheet (`1E2e_gT...`) | Member names, tiers, practice dates, attendance marks | Public CSV export (no auth needed) | +| Payments Sheet (`1Om0YPo...`) | Bank transactions with Person/Purpose annotations | Google Sheets API (service account auth) | + +**Trade-offs**: +- ✅ Non-technical users can view and edit data directly +- ✅ No database setup or maintenance +- ✅ Built-in audit trail (Google Sheets version history) +- ❌ Every page load incurs 1-3s of API latency +- ❌ No complex queries or indexing +- ❌ Rate limits on Google Sheets API + +### 2. Dual-Mode Bank Access + +`fio_utils.py` implements a transparent fallback pattern: + +```python +def fetch_transactions(date_from, date_to): + token = os.environ.get("FIO_API_TOKEN", "").strip() + if token: + return fetch_transactions_api(token, date_from, date_to) # Structured JSON + return fetch_transactions_transparent(...) # HTML scraping +``` + +The API provides richer data (sender account numbers, stable bank IDs) but requires a token. The transparent page is always available but lacks some fields. + +### 3. Name Matching with Confidence Levels + +The reconciliation engine uses a multi-tier matching strategy: + +| Priority | Method | Confidence | Example | +|----------|--------|-----------|---------| +| 1 | Full name match | `auto` | "František Vrbík" in message | +| 2 | Both first + last name (any order) | `auto` | "Vrbík František" | +| 3 | Nickname match | `auto` | "(Štrúdl)" from member list | +| 4 | Last name only (≥4 chars, not common) | `review` | "Vrbík" alone | +| 5 | First name only (≥3 chars) | `review` | "František" alone | + +When both `auto` and `review` matches exist, `review` matches are discarded. This prevents false positives from generic first names. + +### 4. Exception System + +Fee overrides are managed through an `exceptions` sheet tab in the Payments Google Sheet: + +| Column | Content | +|--------|---------| +| Name | Member name | +| Period | Month (YYYY-MM) | +| Amount | Overridden fee in CZK | +| Note | Reason for the exception | + +Exceptions are applied during reconciliation, replacing the attendance-calculated fee with the manually specified amount. + +### 5. Render-Time Performance Tracking + +Every page includes a performance breakdown: + +```python +@app.before_request +def start_timer(): + g.start_time = time.perf_counter() + g.steps = [] + +def record_step(name): + g.steps.append((name, time.perf_counter())) +``` + +The footer displays total render time and, on click, reveals a detailed breakdown (e.g., `fetch_members:0.892s | fetch_payments:1.205s | reconcile:0.003s | render:0.015s`). + +## Security Considerations + +| Concern | Mitigation | +|---------|-----------| +| PII in git | `.secret/` is gitignored; all data fetched at runtime | +| Google API credentials | Service account JSON stored in `.secret/`, mounted as Docker secret | +| Bank API token | Passed via `FIO_API_TOKEN` environment variable, never committed | +| Web app authentication | **None currently** — the app has no auth layer | +| CSRF protection | **None currently** — Flask default (no POST routes exist) | + +## Scalability Notes + +This system is purpose-built for a small club (~20-40 members). It makes deliberate trade-offs favoring simplicity over scale: + +- **No caching**: Every page load fetches live data from Google Sheets (1-3s latency). For a single-user admin dashboard, this is acceptable. +- **No background workers**: Sync and inference are manual `make` commands, not scheduled jobs. +- **No database**: Google Sheets handles 10s of members and 100s of transactions with ease. +- **Single-process Flask**: The built-in development server runs directly in production (via Docker). For this use case, this is intentional — it's a personal tool, not a public service. + +--- + +*Architecture documentation generated from comprehensive code analysis on 2026-03-03.* diff --git a/docs/by-claude-opus/data-model.md b/docs/by-claude-opus/data-model.md new file mode 100644 index 0000000..bfe4321 --- /dev/null +++ b/docs/by-claude-opus/data-model.md @@ -0,0 +1,201 @@ +# Data Model + +## Overview + +FUJ Management operates on two Google Sheets and an external bank account. There is no local database — all persistent data lives in Google Sheets, and all member data is fetched at runtime (never committed to git). + +## External Data Sources + +### 1. Attendance Google Sheet + +| Property | Value | +|----------|-------| +| **Sheet ID** | `1E2e_gT_K5AwSRCDLDTa2UetZTkHmBOcz0kFbBUNUNBA` | +| **Access** | Public CSV export (no authentication required) | +| **Purpose** | Member roster, weekly practice attendance marks | +| **Scope** | Tuesday practices (20:30–22:00) | + +#### Schema + +``` +Row 1: [Title] [blank] [blank] [10/1/2025] [10/8/2025] [10/15/2025] ... +Row 2: Venue per date (ignored by the system) +Row 3: Subtotals per date (ignored by the system) +Row 4+: [Name] [Tier] [Total] [TRUE/FALSE] [TRUE/FALSE] ... +... +Row N: # last line (sentinel — stops parsing) +``` + +| Column | Index | Content | Example | +|--------|-------|---------|---------| +| A | 0 | Member name | `Jan Novák` | +| B | 1 | Tier code | `A`, `J`, or `X` | +| C | 2 | Total attendance (auto-calculated, ignored by the system) | `12` | +| D+ | 3+ | Attendance per date | `TRUE` or `FALSE` | + +#### Tier Codes + +| Code | Meaning | Pays fees? | +|------|---------|-----------| +| `A` | Adult | Yes — calculated from this sheet | +| `J` | Junior | No — managed via a separate sheet | +| `X` | Exempt | No | + +#### Sentinel Row + +The system stops parsing member rows when it encounters a row whose first column contains `# last line` (case-insensitive). Rows starting with `#` are also skipped as comments. + +### 2. Payments Google Sheet + +| Property | Value | +|----------|-------| +| **Sheet ID** | `1Om0YPoDVCH5cV8BrNz5LG5eR5MMU05ypQC7UMN1xn_Y` | +| **Access** | Google Sheets API (service account authentication) | +| **Purpose** | Intermediary ledger for bank transactions + manual annotations | +| **Managed by** | `sync_fio_to_sheets.py` (append), `infer_payments.py` (update) | + +#### Main Sheet Schema (Columns A–K) + +| Column | Label | Populated by | Description | +|--------|-------|-------------|-------------| +| A | Date | `sync` | Transaction date (`YYYY-MM-DD`) | +| B | Amount | `sync` | Bank transaction amount in CZK | +| C | manual fix | Human | If non-empty, `infer` will skip this row | +| D | Person | `infer` or human | Member name(s), comma-separated for multi-person payments | +| E | Purpose | `infer` or human | Month(s) covered, e.g. `2026-01` or `2026-01, 2026-02` | +| F | Inferred Amount | `infer` or human | Amount to use for reconciliation (may differ from bank amount) | +| G | Sender | `sync` | Bank sender name/account | +| H | VS | `sync` | Variable symbol | +| I | Message | `sync` | Payment message for recipient | +| J | Bank ID | `sync` | Fio transaction ID (API only) | +| K | Sync ID | `sync` | SHA-256 deduplication hash | + +#### Exceptions Sheet Tab + +A separate tab named `exceptions` in the same spreadsheet, used for manual fee overrides: + +| Column | Label | Content | +|--------|-------|---------| +| A | Name | Member name (plain text) | +| B | Period | Month (`YYYY-MM`) | +| C | Amount | Overridden fee in CZK | +| D | Note | Reason for override (optional) | + +The first row is assumed to be a header and is skipped. Name and period values are normalized (diacritics stripped, lowercased) for matching. + +### 3. Fio Bank Account + +| Property | Value | +|----------|-------| +| **Account number** | `2800359168/2010` | +| **IBAN** | `CZ8520100000002800359168` | +| **Type** | Transparent account | +| **Owner** | Nathan Heilmann | +| **Public URL** | `https://ib.fio.cz/ib/transparent?a=2800359168` | + +#### Access Methods + +| Method | Trigger | Data richness | +|--------|---------|--------------| +| REST API | `FIO_API_TOKEN` env var set | Full data: sender account, bank ID, user identification, currency | +| HTML scraping | `FIO_API_TOKEN` not set | Partial: date, amount, sender name, message, VS/KS/SS | + +#### API Rate Limit + +The Fio REST API allows 1 request per 30 seconds per token. + +## Fee Calculation Rules + +Fees apply only to **tier A (Adult)** members. They are calculated per calendar month based on Tuesday practice attendance: + +| Practices attended | Monthly fee | +|-------------------|-------------| +| 0 | 0 CZK | +| 1 | 200 CZK | +| 2+ | 750 CZK | + +### Exception Overrides + +The fee can be manually overridden per member per month via the `exceptions` tab. When an exception exists: +- The `expected` amount in reconciliation uses the exception amount +- The `original_expected` amount preserves the attendance-based calculation +- The override is displayed in amber/orange in the web UI + +### Advance Payments + +If a payment references a month not yet covered by attendance data: +- It is tracked as **credit** on the member's account +- Credits are added to the total balance +- When attendance data becomes available for that month, the credit effectively offsets the expected fee + +## Reconciliation Data Model + +The `reconcile()` function returns this structure: + +```python +{ + "members": { + "Jan Novák": { + "tier": "A", + "months": { + "2026-01": { + "expected": 750, # Fee after exception application + "original_expected": 750, # Attendance-based fee + "attendance_count": 4, # How many times they came + "exception": None, # or {"amount": 400, "note": "..."} + "paid": 750.0, # Total matched payments + "transactions": [ # Individual payment records + { + "amount": 750.0, + "date": "2026-01-15", + "sender": "Jan Novák", + "message": "leden", + "confidence": "auto" + } + ] + } + }, + "total_balance": 0 # sum(paid - expected) across all months + off-window credits + } + }, + "unmatched": [ # Transactions that couldn't be assigned + { + "date": "2026-01-20", + "amount": 500, + "sender": "Unknown", + "message": "dar" + } + ], + "credits": { # Alias for positive total_balance entries + "Jan Novák": 200 + } +} +``` + +## Sync ID Generation + +The deduplication key for bank transactions is a SHA-256 hash of: + +``` +sha256("date|amount|currency|sender|vs|message|bank_id") +``` + +All values are lowercased before hashing. This ensures: +- Same transaction fetched twice produces the same ID +- Two payments on the same day with different amounts/senders produce different IDs +- The hash is stable across API and HTML scraping modes (shared fields) + +## Date Handling + +| Source | Format | Normalization | +|--------|--------|--------------| +| Attendance Sheet header | `M/D/YYYY` (US format) | `datetime.strptime(raw, "%m/%d/%Y")` | +| Fio API | `YYYY-MM-DD+HHMM` | Take first 10 characters | +| Fio transparent page | `DD.MM.YYYY` | `datetime.strptime(raw, "%d.%m.%Y")` | +| Google Sheets (unformatted) | Serial number (days since 1899-12-30) | `datetime(1899, 12, 30) + timedelta(days=val)` | + +All internal date representation uses `YYYY-MM-DD` format. Month keys use `YYYY-MM`. + +--- + +*Data model documentation generated from comprehensive code analysis on 2026-03-03.* diff --git a/docs/by-claude-opus/deployment.md b/docs/by-claude-opus/deployment.md new file mode 100644 index 0000000..09b095a --- /dev/null +++ b/docs/by-claude-opus/deployment.md @@ -0,0 +1,198 @@ +# Deployment Guide + +## Local Development + +### Prerequisites + +- **Python 3.13+** (required by `pyproject.toml`) +- **[uv](https://docs.astral.sh/uv/)** — Fast Python package manager +- Google Sheets API credentials (service account JSON) + +### Setup + +```bash +git clone +cd fuj-management + +# Install dependencies +uv sync + +# Configure credentials +mkdir -p .secret +cp /path/to/credentials.json .secret/fuj-management-bot-credentials.json + +# Optional: Set Fio API token for richer bank data +export FIO_API_TOKEN=your_token_here + +# Start the web dashboard +make web +# → Flask server at http://localhost:5001 +``` + +### Makefile Targets + +| Target | Command | Description | +|--------|---------|-------------| +| `help` | `make help` | List all available targets | +| `venv` | `make venv` | Sync virtual environment with pyproject.toml | +| `fees` | `make fees` | Print fee calculation table | +| `match` | `make match` | (Legacy) Direct bank matching | +| `web` | `make web` | Start Flask dashboard on port 5001 | +| `sync` | `make sync` | Sync last 30 days of bank transactions | +| `sync-2026` | `make sync-2026` | Sync full year 2026 transactions | +| `infer` | `make infer` | Auto-fill Person/Purpose in the sheet | +| `reconcile` | `make reconcile` | Print CLI balance report | +| `test` | `make test` | Run test suite | +| `test-v` | `make test-v` | Run tests with verbose output | +| `image` | `make image` | Build Docker image | +| `run` | `make run` | Run Docker container locally | + +The Makefile includes **automatic venv management**: targets that need Python depend on `.venv/.last_sync`, which triggers `uv sync` when `pyproject.toml` changes. + +--- + +## Docker Container + +### Building + +```bash +make image +# → docker build -t fuj-management:latest -f build/Dockerfile . +``` + +### Dockerfile Details + +**Base image**: `python:3.13-alpine` + +**Build stages**: +1. Install system packages (`bash`, `tzdata`) +2. Set timezone to `Europe/Prague` +3. Install Python dependencies via pip +4. Copy application files (`app.py`, `scripts/`, `templates/`, `Makefile`) +5. Copy entrypoint script + +**Exposed port**: 5001 + +**Health check**: `wget -q -O /dev/null http://localhost:5001/` every 60s + +### Running Locally via Docker + +```bash +make run +# → docker run -it --rm -p 5001:5001 fuj-management:latest + +# With credentials and environment: +docker run -it --rm \ + -p 5001:5001 \ + -v $(pwd)/.secret:/app/.secret:ro \ + -e FIO_API_TOKEN=your_token \ + -e BANK_ACCOUNT=CZ8520100000002800359168 \ + fuj-management:latest +``` + +### Entrypoint + +The `build/entrypoint.sh` script simply runs: +```bash +exec python3 /app/app.py +``` + +This uses Flask's built-in server directly. For a production deployment, consider adding gunicorn or waitress (noted as a TODO in the entrypoint). + +### Environment Variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `BANK_ACCOUNT` | `CZ8520100000002800359168` | IBAN for QR code generation | +| `FIO_API_TOKEN` | *(none)* | Fio REST API token | +| `PYTHONUNBUFFERED` | `1` (set in Dockerfile) | Ensures real-time log output | + +--- + +## CI/CD Pipeline + +### Gitea Actions + +The project uses two Gitea Actions workflows: + +#### 1. Build and Push (`build.yaml`) + +**Triggers**: +- Push of any tag +- Manual dispatch (with custom tag input) + +**Steps**: +1. Checkout code +2. Login to Gitea container registry (`gitea.home.hrajfrisbee.cz`) +3. Build Docker image using `build/Dockerfile` +4. Push to `gitea.home.hrajfrisbee.cz//:` + +**Tag resolution**: Uses the git tag name. For manual dispatch, uses the provided input. + +#### 2. Deploy to Kubernetes (`kubernetes-deploy.yaml`) + +**Triggers**: +- Push to any branch +- Manual dispatch + +**Steps**: +1. Checkout code +2. Install kubectl +3. Retrieve Kanidm token from HashiCorp Vault: + - Authenticate to Vault via AppRole (`VAULT_ROLE_ID` / `VAULT_SECRET_ID`) + - Fetch API token from `secret/data/gitea/gitea-ci` +4. Exchange API token for K8s OIDC token via Kanidm: + - POST to `https://idm.home.hrajfrisbee.cz/oauth2/token` + - Token exchange using `urn:ietf:params:oauth:grant-type:token-exchange` +5. Configure kubectl with the OIDC token +6. Run `kubectl auth whoami` and `kubectl get ns` (deploy commands are commented out — WIP) + +**Required secrets**: + +| Secret | Purpose | +|--------|---------| +| `REGISTRY_TOKEN` | Docker registry authentication | +| `VAULT_ROLE_ID` | HashiCorp Vault AppRole role ID | +| `VAULT_SECRET_ID` | HashiCorp Vault AppRole secret ID | +| `K8S_CA_CERT` | Kubernetes cluster CA certificate | + +### Infrastructure Topology + +``` +Gitea (git push / tag) + │ + ├── build.yaml → Docker Build → Gitea Container Registry + │ (gitea.home.hrajfrisbee.cz) + │ + └── kubernetes-deploy.yaml → Vault → Kanidm → K8s Cluster + (192.168.0.31:6443) +``` + +This is a self-hosted infrastructure stack: +- **Gitea** for git hosting and CI/CD +- **HashiCorp Vault** for secret management +- **Kanidm** for identity/OIDC +- **Kubernetes** for container orchestration + +--- + +## Credentials Management + +### Google Sheets API + +The system uses a **Google Cloud service account** for accessing the Payments Google Sheet. The credentials file must be: +- Stored at `.secret/fuj-management-bot-credentials.json` +- In Google Cloud service account JSON format +- The service account must be shared (as editor) on the target Google Sheet + +For local development with OAuth2 (personal Google account), the system also supports the OAuth2 installed app flow — it will generate a `token.pickle` file on first use. + +### Fio Bank API + +Optional. Set the `FIO_API_TOKEN` environment variable. The token is generated in Fio internetbanking under Settings → API. + +**Rate limit**: 1 request per 30 seconds per token. + +--- + +*Deployment documentation generated from comprehensive code analysis on 2026-03-03.* diff --git a/docs/by-claude-opus/development.md b/docs/by-claude-opus/development.md new file mode 100644 index 0000000..ea325de --- /dev/null +++ b/docs/by-claude-opus/development.md @@ -0,0 +1,228 @@ +# Development Guide + +## Development Environment + +### Required Tools + +| Tool | Version | Purpose | +|------|---------|---------| +| Python | 3.13+ | Runtime | +| uv | Latest | Dependency management | +| Docker | Latest | Container builds | +| Git | Any | Version control | +| Make | Any | Build automation | + +### Initial Setup + +```bash +# 1. Clone the repository +git clone +cd fuj-management + +# 2. Install dependencies (creates .venv automatically) +uv sync + +# 3. Activate the virtual environment +source .venv/bin/activate + +# 4. Set up credentials +mkdir -p .secret +# Copy your Google service account JSON here: +cp ~/Downloads/fuj-management-bot-credentials.json .secret/ + +# 5. (Optional) Set Fio API token +export FIO_API_TOKEN=your_token_here +``` + +### IDE Configuration + +The `.vscode/` directory contains workspace settings. If using VS Code, the Python interpreter should automatically detect the `.venv` directory. + +**PYTHONPATH note**: When running scripts from the project root, the Makefile sets `PYTHONPATH=scripts:$PYTHONPATH`. If your IDE doesn't do this, you may see import errors in `match_payments.py` and other scripts that import sibling modules. + +## Project Dependencies + +Defined in `pyproject.toml`: + +| Dependency | Version | Purpose | +|------------|---------|---------| +| `flask` | ≥3.1.3 | Web framework | +| `google-api-python-client` | ≥2.162.0 | Google Sheets API | +| `google-auth-httplib2` | ≥0.2.0 | Google auth transport | +| `google-auth-oauthlib` | ≥1.2.1 | OAuth2 support | +| `qrcode[pil]` | ≥8.0 | QR code generation (with PIL/Pillow backend) | + +The project uses `uv` with `package = false` in `[tool.uv]`, meaning it's not an installable package — dependencies are synced directly to the virtual environment. + +## Coding Conventions + +### Python Style + +- No linter or formatter is configured — the codebase uses a pragmatic, readable style +- Type hints are used for function signatures but not exhaustively +- Docstrings follow Google-style format on key functions +- Scripts use `if __name__ == "__main__": main()` pattern + +### Import Pattern + +Scripts in the `scripts/` directory import from each other as top-level modules: + +```python +# In match_payments.py: +from attendance import get_members_with_fees +from czech_utils import normalize, parse_month_references +from sync_fio_to_sheets import get_sheets_service, DEFAULT_SPREADSHEET_ID +``` + +This works because `scripts/` is added to `sys.path` at runtime (by `app.py` on startup, by Makefile via `PYTHONPATH`, or by scripts adding their own directory to `sys.path`). + +### Template Style + +- All CSS is inline (no external stylesheets) +- No CSS preprocessors or frameworks +- No JavaScript frameworks — plain DOM manipulation +- Terminal-inspired aesthetic: monospace fonts, green-on-black, dashed borders + +### Commit Conventions + +The project uses [Conventional Commits](https://www.conventionalcommits.org/): +``` +feat: add keyboard navigation to member popup +fix: correct diacritic-insensitive search filter +chore: update dependencies +``` + +AI commits include a co-author trailer: +``` +Co-authored-by: Antigravity +``` + +## Architecture Decisions + +### Why No Database? + +Google Sheets serves as the database because: +1. Club members can view and correct data without special tools +2. No database server to manage or back up +3. Built-in version history and collaborative editing +4. Good enough for ~40 members and ~hundreds of transactions + +### Why No Template Inheritance? + +Each HTML template is self-contained. While this means CSS duplication, it keeps each page fully independent and easy to understand. For a 3-page app, the duplication cost is minimal. + +### Why Flask Development Server in Production? + +The Docker container runs Flask's built-in server (`python3 app.py`) rather than gunicorn or waitress. This is intentional — the dashboard is an internal tool accessed by one person at a time. The simplicity outweighs the performance cost. + +### Why Scrape HTML When There's an API? + +The Fio transparent page scraping exists as a **zero-configuration fallback**. Not everyone has an API token, and the transparent page is always publicly accessible. The API is preferred when available (richer data, stable IDs). + +## Common Development Tasks + +### Adding a New Web Route + +1. Add the route function in `app.py`: +```python +@app.route("/new-page") +def new_page(): + # Fetch data + record_step("fetch_data") + # Process + record_step("process_data") + return render_template("new_page.html", ...) +``` + +2. Create `templates/new_page.html` (copy structure from `fees.html`) + +3. Add a link in the nav bar across all templates: +```html +[New Page] +``` + +4. Add a test in `tests/test_app.py` + +### Adding a New Script + +1. Create `scripts/new_script.py` +2. Add a Makefile target: +```makefile +new-target: $(PYTHON) + $(PYTHON) scripts/new_script.py +``` +3. Update `make help` output +4. Add the `.PHONY` declaration + +### Modifying Fee Rules + +Fee rules are defined as constants in `scripts/attendance.py`: +```python +FEE_FULL = 750 # 2+ practices +FEE_SINGLE = 200 # 1 practice +``` + +The calculation logic is in `calculate_fee()`: +```python +def calculate_fee(attendance_count: int) -> int: + if attendance_count == 0: return 0 + if attendance_count == 1: return FEE_SINGLE + return FEE_FULL +``` + +### Adding a New Czech Month Form + +If you encounter a Czech month declension not yet supported, add it to `CZECH_MONTHS` in `scripts/czech_utils.py`: + +```python +CZECH_MONTHS = { + "leden": 1, "ledna": 1, "lednu": 1, + "lednem": 1, # New instrumental case + ... +} +``` + +## Project History + +The project evolved through distinct phases: + +1. **Design phase** — Initial brainstorming captured in `docs/project-notes.md` +2. **CLI tools** — `calculate_fees.py` and `match_payments.py` for command-line workflows +3. **Bank integration** — `fio_utils.py` for transparent page scraping, later API support +4. **Google Sheets sync** — `sync_fio_to_sheets.py` + `infer_payments.py` for the ledger pipeline +5. **Web dashboard** — `app.py` with the `/fees`, `/reconcile`, and `/payments` pages +6. **Interactive features** — Modal popups, QR payments, keyboard navigation, search filter +7. **Fee exceptions** — Manual override system via the `exceptions` sheet tab +8. **CI/CD** — Gitea Actions for Docker builds and Kubernetes deployment + +## Troubleshooting + +### "No data." on the web dashboard + +The attendance Google Sheet couldn't be fetched, or it returned empty data. Check: +- Internet connectivity +- The sheet ID in `attendance.py` is still valid +- The sheet's public sharing settings haven't changed + +### Slow page loads + +Each page fetches data from Google Sheets on every request (no caching). Typical load times are 1-3 seconds. If significantly slower: +- Check the performance breakdown (click the render time in the footer) +- Google Sheets API rate limiting may be the cause + +### Import errors in scripts + +Ensure `PYTHONPATH` includes the `scripts/` directory: +```bash +export PYTHONPATH=scripts:$PYTHONPATH +``` + +Or use the Makefile, which sets this automatically. + +### "Could not fetch exceptions" warning + +The `exceptions` tab doesn't exist in the Payments Google Sheet. This is non-fatal — reconciliation proceeds without fee overrides. + +--- + +*Development guide generated from comprehensive code analysis on 2026-03-03.* diff --git a/docs/by-claude-opus/scripts.md b/docs/by-claude-opus/scripts.md new file mode 100644 index 0000000..3de1370 --- /dev/null +++ b/docs/by-claude-opus/scripts.md @@ -0,0 +1,325 @@ +# Scripts Reference + +All scripts live in the `scripts/` directory and are invoked via `make` targets or directly with Python. + +## Pipeline Scripts + +These scripts form the core data processing pipeline. They are typically run in sequence: + +### `sync_fio_to_sheets.py` — Bank → Google Sheet + +Syncs incoming Fio bank transactions to the Payments Google Sheet. Implements an append-only, deduplicated sync — re-running is always safe. + +**Usage**: +```bash +make sync # Last 30 days +make sync-2026 # Full year 2026 (Jan 1 – Dec 31, sorted) + +# Direct invocation with options: +python scripts/sync_fio_to_sheets.py \ + --credentials .secret/fuj-management-bot-credentials.json \ + --from 2026-01-01 --to 2026-03-01 \ + --sort-by-date +``` + +**Arguments**: + +| Argument | Default | Description | +|----------|---------|-------------| +| `--days` | `30` | Days to look back (ignored if `--from`/`--to` set) | +| `--sheet-id` | Built-in ID | Target Google Sheet | +| `--credentials` | `credentials.json` | Path to Google API credentials | +| `--from` | *(auto)* | Start date (YYYY-MM-DD) | +| `--to` | *(auto)* | End date (YYYY-MM-DD) | +| `--sort-by-date` | `false` | Sort the entire sheet by date after sync | + +**How it works**: + +1. Reads existing Sync IDs (column K) from the Google Sheet +2. Fetches transactions from Fio bank (API or transparent page scraping) +3. For each transaction, generates a SHA-256 hash: `sha256(date|amount|currency|sender|vs|message|bank_id)` +4. Appends only transactions whose hash doesn't exist in the sheet +5. Optionally sorts the sheet by date + +**Key functions**: + +| Function | Signature | Description | +|----------|-----------|-------------| +| `get_sheets_service` | `(credentials_path: str) → Resource` | Authenticates with Google Sheets API. Supports both service accounts and OAuth2 flows. | +| `generate_sync_id` | `(tx: dict) → str` | Creates the SHA-256 deduplication hash for a transaction. | +| `sort_sheet_by_date` | `(service, spreadsheet_id)` | Sorts all rows (excluding header) by the Date column. | +| `sync_to_sheets` | `(spreadsheet_id, credentials_path, ...)` | Main sync logic — read existing, fetch new, deduplicate, append. | + +**Output example**: +``` +Connecting to Google Sheets using .secret/fuj-management-bot-credentials.json... +Reading existing sync IDs from sheet... +Fetching Fio transactions from 2026-02-01 to 2026-03-03... +Found 15 transactions. +Appending 3 new transactions to the sheet... +Sync completed successfully. +Sheet sorted by date. +``` + +--- + +### `infer_payments.py` — Auto-Fill Person/Purpose + +Scans the Payments Google Sheet for rows with empty Person/Purpose columns and uses name matching and Czech month parsing to fill them automatically. + +**Usage**: +```bash +make infer + +# Dry run (preview without writing): +python scripts/infer_payments.py \ + --credentials .secret/fuj-management-bot-credentials.json \ + --dry-run +``` + +**Arguments**: + +| Argument | Default | Description | +|----------|---------|-------------| +| `--sheet-id` | Built-in ID | Target Google Sheet | +| `--credentials` | `credentials.json` | Path to Google API credentials | +| `--dry-run` | `false` | Print inferences without writing to the sheet | + +**How it works**: + +1. Reads all rows from the Payments Google Sheet +2. Fetches the member list from the Attendance Sheet +3. For each row where Person AND Purpose are empty AND there's no "manual fix": + - Combines sender name + message text + - Attempts to match against member names (using name variants and diacritics normalization) + - Parses Czech month references from the message + - Writes inferred Person, Purpose, and Amount back to the sheet +4. Low-confidence matches are prefixed with `[?]` for manual review + +**Skipping rules**: +- If `manual fix` column has any value → skip +- If `Person` column already has a value → skip +- If `Purpose` column already has a value → skip + +**Output example**: +``` +Connecting to Google Sheets... +Reading sheet data... +Fetching member list for matching... +Inffering details for empty rows... +Row 45: Inferred Jan Novák for 2026-02 (750 CZK) +Row 46: Inferred [?] František Vrbík for 2026-01, 2026-02 (1500 CZK) +Applying 2 updates to the sheet... +Update completed successfully. +``` + +--- + +### `match_payments.py` — Reconciliation Engine + CLI Report + +The core reconciliation engine. Matches payment transactions against expected fees and generates a detailed report. Also used as a library by `app.py` and `infer_payments.py`. + +**Usage**: +```bash +make reconcile + +# Direct invocation: +python scripts/match_payments.py \ + --credentials .secret/fuj-management-bot-credentials.json \ + --sheet-id YOUR_SHEET_ID +``` + +**Arguments**: + +| Argument | Default | Description | +|----------|---------|-------------| +| `--sheet-id` | Built-in ID | Payments Google Sheet | +| `--credentials` | `.secret/fuj-management-bot-credentials.json` | Google API credentials | +| `--bank` | `false` | Fetch directly from Fio bank instead of the Google Sheet | + +**Key functions**: + +| Function | Description | +|----------|-------------| +| `_build_name_variants(name)` | Generates searchable name variants from a member name. E.g., "František Vrbík (Štrúdl)" → `["frantisek vrbik", "strudl", "vrbik", "frantisek"]` | +| `match_members(text, member_names)` | Finds members mentioned in text. Returns `(name, confidence)` tuples where confidence is `auto` or `review`. | +| `infer_transaction_details(tx, member_names)` | Infers member(s) and month(s) for a single transaction. | +| `format_date(val)` | Normalizes dates from Google Sheets (handles serial numbers and strings). | +| `fetch_sheet_data(spreadsheet_id, credentials_path)` | Reads all rows from the Payments sheet as a list of dicts. | +| `fetch_exceptions(spreadsheet_id, credentials_path)` | Reads fee overrides from the `exceptions` sheet tab. | +| `reconcile(members, sorted_months, transactions, exceptions)` | **Core engine**: matches transactions to members/months, calculates balances. | +| `print_report(result, sorted_months)` | Prints the CLI reconciliation report. | + +**Name matching strategy**: + +The matching algorithm uses multiple tiers, in order of confidence: + +| Priority | What it checks | Confidence | +|----------|---------------|-----------| +| 1 | Full name (normalized) found in text | `auto` | +| 2 | Both first and last name present (any order) | `auto` | +| 3 | Nickname from parentheses matches | `auto` | +| 4 | Last name only (≥4 chars, not in common surname list) | `review` | +| 5 | First name only (≥3 chars) | `review` | + +**Common surnames excluded from last-name-only matching**: `novak`, `novakova`, `prach` + +If any `auto`-confidence match exists, all `review` matches are discarded. + +**Payment allocation**: + +When a transaction matches multiple members and/or multiple months, the amount is split **evenly** across all allocations: +``` +per_allocation = amount / (num_members × num_months) +``` + +**CLI report sections**: + +1. **Summary table** — Per-member, per-month grid: `OK`, `UNPAID {amount}`, `{paid}/{expected}`, balance +2. **Credits** — Members with positive total balance +3. **Debts** — Members with negative total balance +4. **Unmatched transactions** — Payments that couldn't be assigned +5. **Matched transaction details** — Full breakdown with `[REVIEW]` flags + +--- + +### `calculate_fees.py` — Fee Calculation + +Calculates and prints monthly fees in a simple table format. + +**Usage**: +```bash +make fees +``` + +**Output example**: +``` +Member | Jan 2026 | Feb 2026 +------------------------------------------------------- +Jan Novák | 750 CZK (4) | 200 CZK (1) +Alice Testová | - | 750 CZK (3) +------------------------------------------------------- +TOTAL | 750 CZK | 950 CZK +``` + +This is a simpler CLI version of the `/fees` web page. It only shows adults (tier A). + +--- + +## Shared Modules + +### `attendance.py` — Attendance Data & Fee Logic + +Shared module that fetches attendance data from the Google Sheet and computes fees. + +**Constants**: + +| Constant | Value | Description | +|----------|-------|-------------| +| `SHEET_ID` | `1E2e_gT_K5AwSRCDLDTa2UetZTkHmBOcz0kFbBUNUNBA` | Attendance Google Sheet ID | +| `FEE_FULL` | `750` | Monthly fee for 2+ practices | +| `FEE_SINGLE` | `200` | Monthly fee for exactly 1 practice | +| `COL_NAME` | `0` | Column index for member name | +| `COL_TIER` | `1` | Column index for member tier | +| `FIRST_DATE_COL` | `3` | First column with date headers | + +**Functions**: + +| Function | Signature | Description | +|----------|-----------|-------------| +| `fetch_csv` | `() → list[list[str]]` | Downloads the attendance sheet as CSV via its public export URL. No authentication needed. | +| `parse_dates` | `(header_row) → list[tuple[int, datetime]]` | Parses `M/D/YYYY` dates from the header row and returns `(column_index, date)` pairs. | +| `group_by_month` | `(dates) → dict[str, list[int]]` | Groups column indices by `YYYY-MM` month key. | +| `calculate_fee` | `(count: int) → int` | Applies fee rules: 0→0, 1→200, 2+→750 CZK. | +| `get_members` | `(rows) → list[tuple[str, str, list[str]]]` | Parses member rows. Stops at `# last line` sentinel. Skips comment rows (starting with `#`). | +| `get_members_with_fees` | `() → tuple[list, list[str]]` | Full pipeline: fetch → parse → compute. Returns `(members, sorted_months)` where each member is `(name, tier, {month: (fee, count)})`. | + +**Member tier codes**: + +| Tier | Meaning | Fees? | +|------|---------|-------| +| `A` | Adult | Yes (200 or 750 CZK) | +| `J` | Junior | No (separate sheet) | +| `X` | Exempt | No | + +--- + +### `fio_utils.py` — Fio Bank Integration + +Handles fetching transactions from Fio bank, supporting both API and HTML scraping modes. + +**Functions**: + +| Function | Description | +|----------|-------------| +| `fetch_transactions(date_from, date_to)` | Main entry point. Uses API if `FIO_API_TOKEN` is set, falls back to transparent page scraping. | +| `fetch_transactions_api(token, date_from, date_to)` | Fetches via Fio REST API (JSON). Returns richer data including sender account and stable bank IDs. | +| `fetch_transactions_transparent(date_from, date_to, account_id)` | Scrapes the public Fio transparent account HTML page. | +| `parse_czech_amount(s)` | Parses Czech currency strings like `"1 500,00 CZK"` to float. | +| `parse_czech_date(s)` | Parses `DD.MM.YYYY` or `DD/MM/YYYY` to `YYYY-MM-DD`. | + +**FioTableParser** — A custom `HTMLParser` subclass that extracts transaction rows from the second `` on the Fio transparent page. Column mapping: + +| Index | Column | +|-------|--------| +| 0 | Date (Datum) | +| 1 | Amount (Částka) | +| 2 | Type (Typ) | +| 3 | Sender name (Název protiúčtu) | +| 4 | Message (Zpráva pro příjemce) | +| 5 | KS (constant symbol) | +| 6 | VS (variable symbol) | +| 7 | SS (specific symbol) | +| 8 | Note (Poznámka) | + +**Transaction dict format** (returned by all fetch functions): + +```python +{ + "date": "2026-01-15", # YYYY-MM-DD + "amount": 750.0, # Float, always positive (outgoing filtered) + "sender": "Jan Novák", # Sender name + "message": "příspěvek", # Message for recipient + "vs": "12345", # Variable symbol + "ks": "", # Constant symbol + "ss": "", # Specific symbol + "bank_id": "abc123", # Bank operation ID (API only) + "user_id": "...", # User identification (API only) + "sender_account": "...", # Sender account number (API only) + "currency": "CZK" # Currency (API only) +} +``` + +--- + +### `czech_utils.py` — Czech Language Utilities + +Text processing utilities for Czech language content, critical for matching payment messages. + +**`normalize(text: str) → str`** + +Strips diacritics and lowercases text using Unicode NFKD normalization: +- `"Štrúdl"` → `"strudl"` +- `"František Vrbík"` → `"frantisek vrbik"` +- `"LEDEN 2026"` → `"leden 2026"` + +**`parse_month_references(text: str, default_year=2026) → list[str]`** + +Extracts YYYY-MM month references from Czech free text. Handles a remarkable variety of formats: + +| Input | Output | Pattern | +|-------|--------|---------| +| `"leden"` | `["2026-01"]` | Czech month name | +| `"ledna"` | `["2026-01"]` | Czech month declension | +| `"01/26"` | `["2026-01"]` | Numeric short year | +| `"1/2026"` | `["2026-01"]` | Numeric full year | +| `"11+12/2025"` | `["2025-11", "2025-12"]` | Multiple slash-separated | +| `"12.2025"` | `["2025-12"]` | Dot notation | +| `"listopad-leden"` | `["2025-11", "2025-12", "2026-01"]` | Range with year wrap | +| `"říjen"` | `["2025-10"]` | Months ≥ October assumed previous year | + +**`CZECH_MONTHS`** — Dictionary mapping all Czech month name forms (nominative, genitive, locative) to month numbers. 35 entries covering all 12 months in multiple declensions. + +--- + +*Scripts reference generated from comprehensive code analysis on 2026-03-03.* diff --git a/docs/by-claude-opus/testing.md b/docs/by-claude-opus/testing.md new file mode 100644 index 0000000..ecd8fd8 --- /dev/null +++ b/docs/by-claude-opus/testing.md @@ -0,0 +1,145 @@ +# Testing Guide + +## Overview + +The project uses Python's built-in `unittest` framework with `unittest.mock` for mocking external dependencies (Google Sheets API, attendance data). Tests live in the `tests/` directory. + +## Running Tests + +```bash +make test # Run all tests +make test-v # Run with verbose output +``` + +Under the hood: +```bash +PYTHONPATH=scripts:. python3 -m unittest discover tests +``` + +The `PYTHONPATH` includes both `scripts/` and the project root so that test files can import from both `app.py` and `scripts/*.py`. + +## Test Files + +### `test_app.py` — Flask Route Tests + +Tests the Flask web application routes using Flask's built-in test client. All external data fetching is mocked. + +| Test | What it verifies | +|------|-----------------| +| `test_index_page` | `GET /` returns 200 and contains a redirect to `/fees` | +| `test_fees_route` | `GET /fees` renders the fees dashboard with correct member names | +| `test_reconcile_route` | `GET /reconcile` renders the reconciliation page with payment matching | +| `test_payments_route` | `GET /payments` renders the ledger with grouped transactions | + +**Mocking strategy**: + +```python +@patch('app.get_members_with_fees') +def test_fees_route(self, mock_get_members): + mock_get_members.return_value = ( + [('Test Member', 'A', {'2026-01': (750, 4)})], + ['2026-01'] + ) + response = self.client.get('/fees') + self.assertEqual(response.status_code, 200) + self.assertIn(b'Test Member', response.data) +``` + +Each test patches the data-fetching functions (`get_members_with_fees`, `fetch_sheet_data`) to return controlled test data, avoiding any network calls. + +**Notable**: The reconcile route test also mocks `fetch_sheet_data` and verifies that the reconciliation engine correctly matches a payment against an expected fee (checking for "OK" in the response). + +### `test_reconcile_exceptions.py` — Reconciliation Logic Tests + +Tests the `reconcile()` function directly (unit tests for the core business logic): + +| Test | What it verifies | +|------|-----------------| +| `test_reconcile_applies_exceptions` | When a fee exception exists (400 CZK instead of 750), the expected amount is overridden and balance is calculated correctly | +| `test_reconcile_fallback_to_attendance` | When no exception exists, the attendance-based fee is used | + +**Why these tests matter**: The exception system is critical for correctness — an incorrect override could cause members to be shown incorrect amounts owed. These tests verify that: +- Exceptions properly override the attendance-based fee +- The absence of an exception correctly falls back to the standard calculation +- Balances are computed correctly against overridden amounts + +## Test Data Patterns + +The tests use minimal but representative data: + +```python +# A member with attendance-based fee +members = [('Alice', 'A', {'2026-01': (750, 4)})] + +# An exception reducing the fee +exceptions = {('alice', '2026-01'): {'amount': 400, 'note': 'Test exception'}} + +# A matching payment +transactions = [{ + 'date': '2026-01-05', + 'amount': 400, + 'person': 'Alice', + 'purpose': '2026-01', + 'inferred_amount': 400, + 'sender': 'Alice Sender', + 'message': 'fee' +}] +``` + +## What's Not Tested + +| Area | Status | Notes | +|------|--------|-------| +| Name matching logic | ❌ Not tested | `match_members()`, `_build_name_variants()` | +| Czech month parsing | ❌ Not tested | `parse_month_references()` | +| Fio bank data fetching | ❌ Not tested | Both API and HTML scraping | +| Sync deduplication | ❌ Not tested | `generate_sync_id()` | +| QR code generation | ❌ Not tested | `/qr` route | +| Payment inference | ❌ Not tested | `infer_payments.py` logic | +| Multi-person payment splitting | ❌ Not tested | Even split across members/months | +| Edge cases | ❌ Not tested | Empty sheets, malformed dates, etc. | + +## Writing New Tests + +### Adding a Flask route test + +```python +# In tests/test_app.py + +@patch('app.some_function') +def test_new_route(self, mock_fn): + mock_fn.return_value = expected_data + response = self.client.get('/new-route') + self.assertEqual(response.status_code, 200) + self.assertIn(b'expected content', response.data) +``` + +### Adding a reconciliation logic test + +```python +# In tests/test_reconcile_exceptions.py (or a new test file) + +def test_multi_month_payment(self): + members = [('Bob', 'A', { + '2026-01': (750, 3), + '2026-02': (750, 4) + })] + transactions = [{ + 'date': '2026-02-01', + 'amount': 1500, + 'person': 'Bob', + 'purpose': '2026-01, 2026-02', + 'inferred_amount': 1500, + 'sender': 'Bob', + 'message': 'leden+unor' + }] + result = reconcile(members, ['2026-01', '2026-02'], transactions) + bob = result['members']['Bob'] + self.assertEqual(bob['months']['2026-01']['paid'], 750) + self.assertEqual(bob['months']['2026-02']['paid'], 750) + self.assertEqual(bob['total_balance'], 0) +``` + +--- + +*Testing documentation generated from comprehensive code analysis on 2026-03-03.* diff --git a/docs/by-claude-opus/user-guide.md b/docs/by-claude-opus/user-guide.md new file mode 100644 index 0000000..7dd8647 --- /dev/null +++ b/docs/by-claude-opus/user-guide.md @@ -0,0 +1,166 @@ +# User Guide — FUJ Web Dashboard + +## Getting Started + +Start the dashboard with: + +```bash +make web +``` + +The dashboard is available at **http://localhost:5001** and provides three pages accessible via the green navigation bar at the top. + +## Page 1: Attendance & Fees (`/fees`) + +This page answers the question: **"How much does each member owe this month?"** + +### What You See + +A table with one row per adult member and one column per month. Each cell shows: + +| Cell | Meaning | +|------|---------| +| `750 CZK (4)` | Member owes 750 CZK (attended 4 practices that month) | +| `200 CZK (1)` | Member owes 200 CZK (attended 1 practice) | +| `-` | Member didn't attend — no fee | +| `400 (750) CZK (3)` | Fee **overridden** from 750 to 400 CZK (shown in orange) | + +The bottom row shows **monthly totals** — the total amount expected from all adult members. + +### Fee Rules + +| Practices in a month | Monthly fee | +|----------------------|-------------| +| 0 | 0 CZK (no charge) | +| 1 | 200 CZK | +| 2 or more | 750 CZK | + +### Source Links + +At the top, you'll find direct links to: +- **Attendance Sheet** — the Google Sheet with raw attendance data +- **Payments Ledger** — the Google Sheet with bank transactions + +--- + +## Page 2: Payment Reconciliation (`/reconcile`) + +This page answers: **"Who has paid, who hasn't, and who owes extra?"** + +### Main Table + +Each cell in the matrix shows the payment status for a member × month combination: + +| Cell | Color | Meaning | +|------|-------|---------| +| `OK` | 🟢 Green | Fully paid | +| `UNPAID 750` | 🔴 Red | Haven't paid at all | +| `300/750` | 🔴 Red | Partially paid (300 out of 750) | +| `-` | Gray | No fee expected | +| `PAID 200` | — | Payment received but no fee expected | + +The rightmost column shows each member's **total balance**: +- **Positive** (green): Member has overpaid / has credit +- **Negative** (red): Member still owes money +- **Zero**: Fully settled + +### Search Filter + +Type in the search box at the top to filter members by name. The search is **diacritic-insensitive** — typing "novak" will match "Novák". + +### Member Details + +Click the **`[i]`** icon next to any member's name to open a detailed popup: + +1. **Status Summary** — Month-by-month breakdown with attendance count, expected fee, paid amount, and status. Overridden fees are marked with an amber asterisk. + +2. **Fee Exceptions** — If any months have manual fee overrides, they're listed here with the override amount and reason. + +3. **Payment History** — Every bank transaction matched to this member, showing the date, amount, sender, and payment message. + +**Keyboard shortcuts** (when the popup is open): +- `↑` / `↓` — Navigate to the previous/next member +- `Escape` — Close the popup + +### QR Code Payments + +When you hover over an unpaid or partially paid cell, a red **"Pay"** button appears. Clicking it opens a QR code that can be scanned with any Czech banking app. The QR code is pre-filled with: + +- The club's bank account number +- The exact amount owed +- A payment message identifying the member and month + +This makes it trivial to send a payment link to a member who owes money. + +### Summary Sections + +Below the main table, three additional sections may appear: + +| Section | Shows | +|---------|-------| +| **Credits** | Members with positive balances (advance payments or overpayments) | +| **Debts** | Members with negative balances (outstanding fees) | +| **Unmatched Transactions** | Bank transactions that couldn't be automatically matched to any member | + +--- + +## Page 3: Payments Ledger (`/payments`) + +This page answers: **"What payments has each member made?"** + +### What You See + +Transactions grouped by member name, each showing: +- **Date** — When the payment was received +- **Amount** — How much was paid (in CZK) +- **Purpose** — Which month(s) the payment covers +- **Bank Message** — The original message from the bank transfer + +Transactions are sorted newest-first within each member's section. + +### Unmatched Payments + +Transactions that couldn't be assigned to a member appear under **"Unmatched / Unknown"** — these typically need manual review in the Google Sheet. + +--- + +## Performance Footer + +Every page shows a **render time** in the bottom-right corner (very small, gray text). This tells you how long the page took to generate. + +Click on it to reveal a detailed breakdown showing how much time was spent on each step (fetching members, fetching payments, reconciliation, template rendering, etc.). This is mostly useful for debugging slow page loads. + +--- + +## Common Workflows + +### "A member asks how much they owe" + +1. Open `/reconcile` +2. Search for the member's name +3. Their row shows the exact status per month +4. Click `[i]` for detailed payment history + +### "A member wants to pay" + +1. Open `/reconcile` +2. Find the unpaid cell +3. Hover and click the red **Pay** button +4. Share the QR code with the member (screenshot or show on screen) + +### "I want to see all payments from one person" + +1. Open `/payments` +2. Scroll to the member's section (alphabetically sorted) + +### "A transaction wasn't matched correctly" + +1. Open the **Payments Ledger** Google Sheet (link at the top of any page) +2. Find the row +3. Manually correct the **Person** and/or **Purpose** columns +4. Put any marker in the **manual fix** column to prevent the inference script from overwriting your edit +5. Refresh the web dashboard + +--- + +*User guide generated from comprehensive code analysis on 2026-03-03.* diff --git a/docs/by-claude-opus/web-app.md b/docs/by-claude-opus/web-app.md new file mode 100644 index 0000000..3df00fc --- /dev/null +++ b/docs/by-claude-opus/web-app.md @@ -0,0 +1,256 @@ +# Web Application Documentation + +## Overview + +The FUJ Management web application is a Flask-based dashboard that provides real-time visibility into club finances. It renders server-side HTML with embedded CSS and JavaScript — no build tools, no npm, no framework. The UI follows a distinctive **terminal-inspired aesthetic** with monospace fonts, green-on-black colors, and dashed borders. + +## Routes + +### `GET /` — Index (Redirect) + +Redirects to `/fees` via an HTML meta refresh tag. This exists so the root URL always leads somewhere useful. + +### `GET /fees` — Attendance & Fees Dashboard + +**Template**: `templates/fees.html` + +Displays a table of all adult members with their calculated monthly fees based on attendance. Each cell shows the fee amount (in CZK), the number of practices attended, or a dash for months with zero attendance. + +**Data pipeline**: +``` +attendance.py::get_members_with_fees() → Filter to tier "A" (adults) +match_payments.py::fetch_exceptions() → Check for fee overrides + → Format cells with override indicators + → Render fees.html with totals row +``` + +**Visual features**: +- Fee overrides shown in **orange** with the original amount in parentheses +- Empty months shown in muted gray +- Monthly totals row at the bottom +- Performance timing in the footer (click to expand breakdown) + +**Template variables**: + +| Variable | Type | Content | +|----------|------|---------| +| `months` | `list[str]` | Month labels like "Jan 2026" | +| `results` | `list[dict]` | `{name, months: [{cell, overridden}]}` | +| `totals` | `list[str]` | Monthly total strings like "3750 CZK" | +| `attendance_url` | `str` | Link to the attendance Google Sheet | +| `payments_url` | `str` | Link to the payments Google Sheet | + +### `GET /reconcile` — Payment Reconciliation + +**Template**: `templates/reconcile.html` (802 lines — the most complex template) + +The centerpiece of the application. Shows a matrix of members × months with payment status, plus summary sections for credits, debts, and unmatched transactions. + +**Data pipeline**: +``` +attendance.py::get_members_with_fees() → All members + fees +match_payments.py::fetch_sheet_data() → All payment transactions +match_payments.py::fetch_exceptions() → Fee overrides +match_payments.py::reconcile() → Match payments ↔ fees + → Render reconcile.html +``` + +**Cell statuses**: + +| Status | CSS Class | Display | Meaning | +|--------|-----------|---------|---------| +| `empty` | `cell-empty` | `-` | No fee expected, no payment | +| `ok` | `cell-ok` | `OK` | Paid in full (green) | +| `partial` | `cell-unpaid` | `300/750` | Partially paid (red) | +| `unpaid` | `cell-unpaid` | `UNPAID 750` | Nothing paid (red) | +| `surplus` | — | `PAID 200` | Payment received but no fee expected | + +**Interactive features**: + +1. **Member detail modal** — Click the `[i]` icon next to any member name to see: + - Status summary table (month, attendance count, expected, paid, status) + - Fee exceptions (if any, shown in amber) + - Full payment history with dates, amounts, senders, and messages + +2. **Keyboard navigation** — When a member modal is open: + - `↑` / `↓` arrows navigate between members (respecting search filter) + - `Escape` closes the modal + +3. **Name search filter** — Type in the search box to filter members. Uses diacritic-insensitive matching (e.g., typing "novak" matches "Novák"). + +4. **QR Payment** — Hover over an unpaid/partial cell to reveal a "Pay" button. Clicking it opens a QR code modal with: + - A Czech SPD-format QR code (scannable by Czech banking apps) + - Pre-filled account number, amount, and payment message + - The QR image is generated server-side via `GET /qr` + +**Client-side data**: + +The template receives a full JSON dump of member data (`member_data`) embedded in a ` + + +``` + +There is no shared base template (no Jinja2 template inheritance). CSS is duplicated across templates with small variations. + +--- + +*Web application documentation generated from comprehensive code analysis on 2026-03-03.* diff --git a/docs/by-gemini/README.md b/docs/by-gemini/README.md new file mode 100644 index 0000000..31d18c7 --- /dev/null +++ b/docs/by-gemini/README.md @@ -0,0 +1,36 @@ +# FUJ Management System + +Welcome to the **FUJ Management System**, a streamlined solution for managing Ultimate Frisbee club finances, attendance, and member payments. This system automates the tedious parts of club management, keeping your ledger clean and your reconciliation painless. + +## 🚀 Mission + +The project's goal is to minimize manual entry and potential human error in club management by: +1. **Automating Bank Synchronization**: Periodically fetching transactions from Fio bank. +2. **Smart Inference**: Using heuristics to match bank transactions to members and payment periods. +3. **Visual Reconciliation**: Providing a clear, real-time web dashboard for managers to track who has paid and who is in debt. + +## ✨ Key Features + +- **Seamless Bank Integration**: Synchronize transactions directly from the Fio bank API into a Google Spreadsheet. +- **Intelligent Matching**: Automatic detection of member names and payment periods from transaction messages using diacritic-insensitive Czech text processing. +- **Dynamic Dashboard**: A Flask-powered web interface displaying monthly fees, payment status (OK, Partial, Unpaid), and total balances. +- **Manual Overrides**: Support for fee exceptions and manual payment matching when automation needs a human touch. +- **QR Payment Generation**: Integrated QR code generation to make paying outstanding fees trivial for members. + +## 🛠 Tech Stack + +- **Backend**: Python 3.12+ (managed with `uv`) +- **Web Framework**: Flask with Jinja2 templates +- **Data Storage**: Google Sheets (used as a collaborative database) +- **APIs**: Fio Bank API, Google Sheets API v4 +- **Containerization**: Docker / OCI Images +- **Automation**: `Makefile` based workflow + +--- + +## 📂 Documentation Guide + +- [Architecture](architecture.md): High-level system design and data flow. +- [User Guide](user-guide.md): How to operate the system as a club manager. +- [Support Scripts](scripts.md): Detailed reference for CLI tools. +- [Deployment](deployment.md): Technical setup and infrastructure instructions. diff --git a/docs/by-gemini/architecture.md b/docs/by-gemini/architecture.md new file mode 100644 index 0000000..3f76a83 --- /dev/null +++ b/docs/by-gemini/architecture.md @@ -0,0 +1,48 @@ +# System Architecture + +The FUJ Management system is designed around a **"Sheet-as-a-Database"** architecture. This allows for easy manual editing and transparency while enabling powerful automation. + +## 🔄 High-Level Data Flow + +```mermaid +graph TD + Fio[Fio Bank API] -->|Sync Script| GS(Google Spreadsheet) + Att[Attendance Sheet] -->|CSV Export| App(Flask Web App) + GS -->|API Fetch| App + App -->|Display| UI[Manager Dashboard] + GS -.->|Manual Edits| GS + App -->|Generate| QR[QR Codes for Members] +``` + +### 1. Data Ingestion (Bank to Sheet) +The synchronization pipeline moves raw bank data into a structured format: +- `sync_fio_to_sheets.py` fetches transactions and appends them to the "Transactions" sheet. +- Each transaction is assigned a unique `Sync ID` to prevent duplicates. +- `infer_payments.py` processes new rows, attempting to fill the `Person`, `Purpose`, and `Inferred Amount` columns based on the message and sender. + +### 2. Logic & Reconciliation +The core logic resides in shared Python scripts: +- **Attendance**: `attendance.py` pulls the latest practice data from a separate attendance sheet and calculates expected fees (e.g., 0/200/750 CZK rules). +- **Matching**: `match_payments.py` performs the "heavy lifting" by correlating members, months, and payments. It handles partial payments, overpayments (credits), and manual exceptions. + +### 3. Presentation Layer +The Flask application (`app.py`) serves as the primary interface: +- **Fees View**: Shows attendance-based charges. +- **Reconciliation View**: The main "truth" dashboard showing balance per member. +- **Payments View**: Historical list of transactions grouped by member. + +## 🛡 Security & Authentication + +- **Fio Bank**: Authorized via a private API token (kept in `.secret/`). +- **Google Sheets**: Authenticated via a **Service Account** or **OAuth2** (using `.secret/fuj-management-bot-credentials.json`). +- **Environment**: Secrets are never committed; the `.secret/` directory is git-ignored. + +## 🧩 Key Components + +| Component | Responsibility | +| :--- | :--- | +| **Google Spreadsheet** | Unified source of truth for transactions and manual overrides. | +| **scripts/** | A suite of CLI utilities for batch processing and data maintenance. | +| **Flask App** | Read-only views for state visualization and QR code generation. | +| **czech_utils.py** | Diacritic-normalization and NLP for Czech month/name parsing. | +``` diff --git a/docs/by-gemini/deployment.md b/docs/by-gemini/deployment.md new file mode 100644 index 0000000..522f180 --- /dev/null +++ b/docs/by-gemini/deployment.md @@ -0,0 +1,72 @@ +# Deployment & Technical Setup + +This document provides instructions for developers and devops engineers to set up and deploy the FUJ Management system. + +## 🛠 Prerequisites + +- **Python 3.12+**: The project uses modern type hinting and syntax features. +- **uv**: High-performance Python package installer and resolver. + - Install via brew: `brew install uv` +- **Docker** (Optional): For containerized deployments. + +## ⚙️ Initial Setup + +1. **Clone the repository**: + ```bash + git clone + cd fuj-management + ``` + +2. **Install dependencies**: + Using `uv`, everything is handled automatically: + ```bash + make venv + ``` + +3. **Secrets Management**: + Create a `.secret/` directory. You will need two main credentials: + - `fuj-management-bot-credentials.json`: A Google Cloud Service Account key with access to the Sheets API. + - `fio-token.txt`: (Implicitly used by `fio_utils.py`) Your Fio bank API token. + + Ensure these are never committed! They are ignored by `.gitignore`. + +## 🐳 Containerization + +The project can be built and run as an OCI image. + +1. **Build the image**: + ```bash + make image + ``` + This uses the `build/Dockerfile`, which is optimized for small size and security. + +2. **Run the container**: + ```bash + make run + ``` + The app exposes port `5001`. + +## 🧪 Testing & Validation + +The project includes a suite of infrastructure and logic tests. + +- **Run all tests**: + ```bash + make test + ``` +- **Verbose output**: + ```bash + make test-v + ``` + +Tests are located in the `tests/` directory and use the standard Python `unittest` framework. They cover: +- CSV parsing logic. +- Fee calculation rules. +- Name matching and normalization. + +## 🚀 Future Roadmap + +- **Automated Backups**: Regular snapshots of the Google Sheet. +- **Authentication Layer**: Login for the web dashboard (currently assumes internal VPN or trusted environment). +- **Gitea Actions**: Continuous Integration for building and testing images. +``` diff --git a/docs/by-gemini/scripts.md b/docs/by-gemini/scripts.md new file mode 100644 index 0000000..e02c4c8 --- /dev/null +++ b/docs/by-gemini/scripts.md @@ -0,0 +1,66 @@ +# Support Scripts Reference + +The project includes several CLI utilities located in the `scripts/` directory. Most are accessible via `make` targets. + +## 🚀 Primary Scripts + +### `sync_fio_to_sheets.py` +**Target**: `make sync` | `make sync-2026` +- **Purpose**: Downloads transactions from Fio bank via API and appends new ones to the Google Sheet. +- **Key Logic**: Uses a `Sync ID` (SHA-256 hash of transaction details) to ensure that even if the sync is run multiple times, no duplicate rows are created. +- **Arguments**: + - `--days`: How many days back to look (default 30). + - `--from/--to`: Specific date range. + - `--sort-by-date`: Re-sorts the spreadsheet after appending. + +### `infer_payments.py` +**Target**: `make infer` +- **Purpose**: Processes the "Transactions" sheet to fill in `Person`, `Purpose`, and `Inferred Amount`. +- **Logic**: + - Analyzes the `Sender` and `Message` fields. + - Uses `match_payments.py` heuristics to find members. + - If confidence is low, prefixes the name with `[?]` to flag it for manual review. + - Won't overwrite cells that already have data (respecting your manual fixes). + +### `match_payments.py` +**Target**: `make match` | `make reconcile` +- **Purpose**: The core "Reconciliation Engine". +- **Logic**: + - Fetches attendance fees (from `attendance.py`). + - Fetches transaction data. + - Correlates them based on inferred `Person` and `Purpose`. + - Handles "rollover" balances—extra money from one month is tracked as a credit for the next. + +--- + +## 🛠 Utility Modules + +### `attendance.py` +- Handles the connection to the Google Attendance sheet (exported as CSV). +- Implements the club's fee rules: + - 0 practices = 0 CZK + - 1 practice = 200 CZK + - 2+ practices = 750 CZK +- *Note*: Fee calculation only applies to members in Tier "A" (Adults). + +### `czech_utils.py` +- **Normalization**: Strips diacritics and lowercases text (e.g., `František` -> `frantisek`). +- **Month Parsing**: Advanced regex to detect month references in Czech (e.g., "leden-brezen", "11+12/25", "na únor"). + +### `fio_utils.py` +- Low-level wrapper for the Fio Bank API. +- Handles HTTP requests and JSON parsing for transaction lists. + +--- + +## ⚙️ Makefile Summary + +| Command | Action | +| :--- | :--- | +| `make fees` | Preview calculated fees based on attendance. | +| `make sync` | Sync last 30 days of bank data. | +| `make infer` | Run smart tagging on the sheet. | +| `make reconcile` | Run a text-based reconciliation report in terminal. | +| `make web` | Start the Flask dashboard. | +| `make test` | Run the test suite. | +``` diff --git a/docs/by-gemini/user-guide.md b/docs/by-gemini/user-guide.md new file mode 100644 index 0000000..60930c9 --- /dev/null +++ b/docs/by-gemini/user-guide.md @@ -0,0 +1,61 @@ +# User Guide + +This guide is intended for club managers who use the FUJ Management system for day-to-day operations. + +## 🛠 Operational Workflow + +To keep the club finances up-to-date, follow these steps periodically (e.g., once a week): + +1. **Sync Bank Transactions**: + Run the sync script to pull the latest payments from Fio. + ```bash + make sync + ``` +2. **Infer Payments**: + Let the system automatically tag who paid for what. + ```bash + make infer + ``` +3. **Manual Review (Google Sheets)**: + Open the Google Spreadsheet. Check rows with the `[?]` prefix in the `Person` column—these require human confirmation. + - If correct: Remove the `[?]` prefix. + - If incorrect: Manually fix the `Person` and `Purpose`. + - If a payment covers a special case: Use the **exceptions** sheet to override expected fees. +4. **Check Reconciliation Dashboard**: + Start the web app to see the final balance report. + ```bash + make web + ``` + Navigate to `http://localhost:5001/reconcile`. + +--- + +## 📊 Understanding the Dashboard + +### Reconciliation Page +- **Green (OK)**: Member has paid exactly what was expected (or more). +- **Orange (Partial)**: Some payment was received, but there's still a debt. +- **Red (UNPAID)**: No payment recorded for this month. +- **Blue (SURPLUS)**: Payment received for a month where no fee was expected. + +### Handling Debts +If a member is in debt, you can click on the unpaid/partial cell to get a **QR Platba** link. You can send this link or screenshot to the member to facilitate quick payment. + +--- + +## ❓ FAQ & Troubleshooting + +### Why is a payment "Unmatched"? +A payment stays unmatched if neither the sender name nor the message contains recognizable member names or nicknames. +- **Fix**: Manually enter the member's name in the `Person` column in the Google Sheet. + +### How do I handle a "Family Discount" or "Prepaid Year"? +Use the `exceptions` sheet in the Google Spreadsheet. +1. Add the member's name (exactly as it appears in attendance). +2. Enter the month (e.g., `2026-03`). +3. Enter the new `Amount` (use `0` for prepaid). +4. Add a `Note` for clarity. + +### The web app is slow to load. +The app fetches data from Google Sheets API on every request. This ensures real-time data but can take a few seconds. The "Performance Breakdown" footer shows exactly where the time was spent. +``` diff --git a/docs/index.html b/docs/index.html new file mode 100644 index 0000000..cb4417e --- /dev/null +++ b/docs/index.html @@ -0,0 +1,43 @@ + + + + + + FUJ Management - Documentation + + + + + + + + +
Loading documentation...
+ + + + + + + + \ No newline at end of file