From 327688d9d2143c07949bf5f9d47b6f7e408cbb50 Mon Sep 17 00:00:00 2001 From: Jan Novak Date: Sat, 14 Feb 2026 23:20:54 +0100 Subject: [PATCH] Add comprehensive project documentation Cover the full pipeline (scrapers, merge, map generation), all 6 data sources with their parsing methods, filter criteria, CLI arguments, Docker setup, caching, rate limiting, and project structure. Co-Authored-By: Claude Opus 4.6 --- README.md | 244 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 244 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..cd7e52f --- /dev/null +++ b/README.md @@ -0,0 +1,244 @@ +# Maru hleda byt + +Apartment search aggregator for Prague. Scrapes listings from 6 Czech real estate portals, filters them by configurable criteria, deduplicates across sources, and generates a single interactive map with all matching apartments. + +Built for a specific use case: finding a 3+kk or larger apartment in Prague, excluding panel construction ("panelak") and housing estates ("sidliste"), with personal rating support. + +## How it works + +``` +┌─────────────────────────────────────────────────────────────┐ +│ run_all.sh │ +│ Orchestrates all scrapers, then merges results into map │ +├─────────┬──────────┬──────────┬────────┬────────┬───────────┤ +│Sreality │Realingo │Bezreal. │iDNES │PSN │CityHome │ +│ (API) │ (HTML) │ (HTML) │ (HTML) │ (HTML) │ (HTML) │ +├─────────┴──────────┴──────────┴────────┴────────┴───────────┤ +│ merge_and_map.py │ +│ Loads all byty_*.json, deduplicates, generates HTML map │ +├─────────────────────────────────────────────────────────────┤ +│ mapa_bytu.html │ +│ Interactive Leaflet.js map with filters & ratings │ +└─────────────────────────────────────────────────────────────┘ +``` + +### Pipeline + +1. **Scraping** -- Each scraper independently fetches listings from its portal, applies filters, and saves results to a JSON file (`byty_.json`). +2. **Merging** -- `merge_and_map.py` loads all 6 JSON files, deduplicates listings (by street name + price + area), and generates the final `mapa_bytu.html`. +3. **Serving** -- The HTML map can be opened locally as a file, or served via Docker with a built-in HTTP server. + +### Execution order in `run_all.sh` + +Scrapers run sequentially (to avoid overwhelming any single portal), except PSN and CityHome which run in parallel (different sites). If a scraper fails, it is logged but does not abort the pipeline -- remaining scrapers continue. + +``` +1. scrape_and_map.py (Sreality) +2. scrape_realingo.py (Realingo) +3. scrape_bezrealitky.py (Bezrealitky) +4. scrape_idnes.py (iDNES Reality) +5. scrape_psn.py + scrape_cityhome.py (parallel) +6. merge_and_map.py (merge + map generation) +``` + +## Scrapers + +All scrapers share the same CLI interface and a consistent two-phase approach: + +1. **Phase 1** -- Fetch listing pages (paginated) to get a list of all available apartments. +2. **Phase 2** -- Fetch detail pages for each listing to get floor, construction type, and other data needed for filtering. + +Each scraper uses a **JSON file cache**: if a listing's `hash_id` and `price` haven't changed since the last run, the cached data is reused and the detail page is not re-fetched. This significantly reduces runtime on subsequent runs. + +### Source details + +| Scraper | Portal | Data source | Output file | Notes | +|---------|--------|-------------|-------------|-------| +| `scrape_and_map.py` | [Sreality.cz](https://sreality.cz) | REST API (JSON) | `byty_sreality.json` | Main scraper. Also contains `generate_map()` used by all other scripts. | +| `scrape_realingo.py` | [Realingo.cz](https://realingo.cz) | `__NEXT_DATA__` JSON in HTML | `byty_realingo.json` | Next.js app, data extracted from server-side props. | +| `scrape_bezrealitky.py` | [Bezrealitky.cz](https://bezrealitky.cz) | `__NEXT_DATA__` Apollo cache in HTML | `byty_bezrealitky.json` | Next.js app with Apollo GraphQL cache in page source. | +| `scrape_idnes.py` | [Reality iDNES](https://reality.idnes.cz) | HTML parsing (regex) | `byty_idnes.json` | Traditional HTML site. GPS extracted from `dataLayer.push()`. Retry logic with 5 attempts and exponential backoff. | +| `scrape_psn.py` | [PSN.cz](https://psn.cz) | RSC (React Server Components) escaped JSON in HTML | `byty_psn.json` | Uses `curl` instead of `urllib` due to Cloudflare SSL issues. Hardcoded list of Prague projects with GPS coordinates. | +| `scrape_cityhome.py` | [city-home.cz](https://city-home.cz) | HTML table parsing (data attributes on ``) | `byty_cityhome.json` | CityHome/SATPO developer projects. GPS fetched from project locality pages. | + +### Scraper filter criteria + +All scrapers apply the same core filters (with minor per-source variations): + +| Filter | Value | Notes | +|--------|-------|-------| +| **Max price** | 13 500 000 CZK | PSN and CityHome use 14 000 000 CZK | +| **Min area** | 69 m^2 | | +| **Min floor** | 2. NP (2nd floor) | 2nd floor apartments are included but flagged on the map | +| **Dispositions** | 3+kk, 3+1, 4+kk, 4+1, 5+kk, 5+1, 6+ | | +| **Region** | Praha | | +| **Construction** | Excludes panel ("panelak") | | +| **Location** | Excludes housing estates ("sidliste") | | + +## Utility scripts + +### `merge_and_map.py` + +Merges all `byty_*.json` files into `byty_merged.json` and generates `mapa_bytu.html`. + +**Deduplication logic:** Two listings are considered duplicates if they share the same normalized street name + price + area. PSN and CityHome have priority during dedup (loaded first), so their listings are kept over duplicates from other portals. + +### `regen_map.py` + +Regenerates the map from existing `byty_sreality.json` data without re-scraping. Fetches missing area values from the Sreality API, fixes URLs, and re-applies the area filter. Useful for tweaking map output after data has already been collected. + +## Interactive map (`mapa_bytu.html`) + +The generated map is a standalone HTML file using Leaflet.js with CARTO basemap tiles. Features: + +- **Color-coded markers** by disposition (3+kk = blue, 3+1 = green, 4+kk = orange, etc.) +- **Heart-shaped markers** for PSN and CityHome listings (developer favorites) +- **Source badge** in each popup (Sreality, Realingo, Bezrealitky, iDNES, PSN, CityHome) +- **Client-side filters:** minimum floor, maximum price, hide rejected +- **Rating system** (persisted in `localStorage`): + - Star -- mark as favorite (enlarged marker with pulsing glow) + - Reject -- dim the marker, optionally hide it + - Notes -- free-text notes per listing +- **2nd floor warning** -- listings on 2. NP show an orange warning in the popup +- **Statistics panel** -- total count, price range, average price, disposition breakdown + +## CLI arguments + +All scrapers accept the same arguments. When run via `run_all.sh`, these arguments are forwarded to every scraper. + +``` +--max-pages N Maximum number of listing pages to scrape per source. + Limits the breadth of the initial listing fetch. + (For PSN: max pages per project) + +--max-properties N Maximum number of properties to fetch details for per source. + Limits the depth of the detail-fetching phase. + +--log-level LEVEL Logging verbosity. One of: DEBUG, INFO, WARNING, ERROR. + Default: INFO. + DEBUG shows HTTP request/response details, filter decisions + for every single listing, and cache hit/miss info. + +-h, --help Show help message (run_all.sh only). +``` + +### Examples + +```bash +# Full scrape (all pages, all properties) +./run_all.sh + +# Quick validation run (1 page per source, max 10 properties each) +./run_all.sh --max-pages 1 --max-properties 10 + +# Full scrape with debug logging +./run_all.sh --log-level DEBUG + +# Run a single scraper +python3 scrape_bezrealitky.py --max-pages 2 --max-properties 5 --log-level DEBUG +``` + +## Running with Docker + +The project includes a Docker setup for unattended operation with a cron-based schedule. + +### Container architecture + +``` +┌─────────────────────────────────────────┐ +│ Container (python:3.13-alpine) │ +│ │ +│ PID 1: python3 -m http.server :8080 │ +│ serves /app/data/ │ +│ │ +│ crond: runs run_all.sh at 06:00/18:00 │ +│ Europe/Prague timezone │ +│ │ +│ /app/ -- scripts (.py, .sh) │ +│ /app/data/ -- volume (JSON + HTML) │ +│ ^ symlinked from /app/byty_* │ +└─────────────────────────────────────────┘ +``` + +On startup, the HTTP server starts immediately. The initial scrape runs in the background. Subsequent cron runs update data in-place twice daily at 06:00 and 18:00 CET/CEST. + +### Quick start + +```bash +make run # Build image + start container on port 8080 +# Map available at http://localhost:8080/mapa_bytu.html +``` + +### Makefile targets + +| Target | Description | +|--------|-------------| +| `make help` | Show all available targets | +| `make build` | Build the Docker image | +| `make run` | Build and run the container (port 8080) | +| `make stop` | Stop and remove the container | +| `make logs` | Tail container logs | +| `make scrape` | Trigger a manual scrape inside the running container | +| `make restart` | Stop and re-run the container | +| `make clean` | Stop container and remove the Docker image | +| `make validation` | Run a limited scrape in a separate Docker container (port 8081) | +| `make validation-stop` | Stop the validation container | +| `make validation-local` | Run a limited scrape locally (1 page, 10 properties) | +| `make validation-local-debug` | Same as above with `--log-level DEBUG` | + +### Validation mode + +Validation targets run scrapers with `--max-pages 1 --max-properties 10` for a fast smoke test (~30 seconds instead of several minutes). The Docker validation target runs on port 8081 in a separate container so it doesn't interfere with production data. + +## Project structure + +``` +. +├── scrape_and_map.py # Sreality scraper + map generator (generate_map()) +├── scrape_realingo.py # Realingo scraper +├── scrape_bezrealitky.py # Bezrealitky scraper +├── scrape_idnes.py # iDNES Reality scraper +├── scrape_psn.py # PSN scraper +├── scrape_cityhome.py # CityHome scraper +├── merge_and_map.py # Merge all sources + generate final map +├── regen_map.py # Regenerate map from cached Sreality data +├── run_all.sh # Orchestrator script (runs all scrapers + merge) +├── mapa_bytu.html # Generated interactive map (output) +├── Makefile # Docker management + validation shortcuts +├── build/ +│ ├── Dockerfile # Container image definition (python:3.13-alpine) +│ ├── entrypoint.sh # Container entrypoint (HTTP server + cron + initial scrape) +│ ├── crontab # Cron schedule (06:00 and 18:00 CET) +│ └── CONTAINER.md # Container-specific documentation +└── .gitignore # Ignores byty_*.json, __pycache__, .vscode +``` + +## Dependencies + +**None.** All scrapers use only the Python standard library (`urllib`, `json`, `re`, `argparse`, `logging`, `html.parser`). The only external tool required is `curl` (used by `scrape_psn.py` for Cloudflare TLS compatibility). + +The Docker image is based on `python:3.13-alpine` (~70 MB) with `curl`, `bash`, and `tzdata` added. + +## Caching behavior + +Each scraper maintains a JSON file cache (`byty_.json`). On each run: + +1. The previous JSON file is loaded and indexed by `hash_id`. +2. For each listing found in the current run, if the `hash_id` exists in cache **and** the price is unchanged, the cached record is reused without fetching the detail page. +3. New or changed listings trigger a detail page fetch. +4. The JSON file is overwritten with the fresh results at the end. + +This means the first run is slow (fetches every detail page with rate-limiting delays), but subsequent runs are much faster as they only fetch details for new or changed listings. + +## Rate limiting + +Each scraper includes polite delays between requests: + +| Scraper | Delay between requests | +|---------|----------------------| +| Sreality | 0.3s (details), 0.5s (pages) | +| Realingo | 0.3s (details), 0.5s (pages) | +| Bezrealitky | 0.4s (details), 0.5s (pages) | +| iDNES | 0.4s (details), 1.0s (pages) + retry backoff (3/6/9/12s) | +| PSN | 0.5s (per project page) | +| CityHome | 0.5s (per project GPS fetch) |