Add status dashboard, server, scraper stats, and DATA_DIR support
All checks were successful
Build and Push / build (push) Successful in 7s
All checks were successful
Build and Push / build (push) Successful in 7s
Key changes:
- Replace ratings_server.py + status.html with a unified server.py that
serves the map, scraper status dashboard, and ratings API in one process
- Add scraper_stats.py utility: each scraper writes per-run stats (fetched,
accepted, excluded, duration) to stats_<source>.json for the status page
- generate_status.py: respect DATA_DIR env var so status.json lands in the
configured data directory instead of always the project root
- run_all.sh: replace the {"status":"running"} overwrite of status.json with
a dedicated scraper_running.json lock file; trap on EXIT ensures cleanup
even on kill/error, preventing the previous run's results from being wiped
- server.py: detect running state via scraper_running.json existence instead
of status["status"] field, eliminating the dual-use race condition
- Makefile: add serve (local dev), debug (Docker debug container) targets;
add SERVER_PORT variable
- build/Dockerfile + entrypoint.sh: switch to server.py, set DATA_DIR,
adjust volume mounts
- .gitignore: add *.json and *.log to keep runtime data files out of VCS
- mapa_bytu.html: price-per-m² colouring, status link, UX tweaks
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
30
Makefile
30
Makefile
@@ -3,9 +3,13 @@ CONTAINER_NAME := maru-hleda-byt
|
||||
VOLUME_NAME := maru-hleda-byt-data
|
||||
VALIDATION_CONTAINER := maru-hleda-byt-validation
|
||||
VALIDATION_VOLUME := maru-hleda-byt-validation-data
|
||||
DEBUG_CONTAINER := maru-hleda-byt-debug
|
||||
DEBUG_VOLUME := maru-hleda-byt-debug-data
|
||||
DEBUG_PORT ?= 8082
|
||||
PORT := 8080
|
||||
SERVER_PORT ?= 8080
|
||||
|
||||
.PHONY: build run stop logs scrape restart clean help validation validation-local validation-stop validation-local-debug
|
||||
.PHONY: build run stop logs scrape restart clean help serve validation validation-local validation-stop validation-local-debug debug debug-stop
|
||||
|
||||
help:
|
||||
@echo "Available targets:"
|
||||
@@ -20,6 +24,9 @@ help:
|
||||
@echo " validation-local-debug - Run validation locally with DEBUG logging"
|
||||
@echo " restart - Restart the container (stop and run again)"
|
||||
@echo " clean - Stop container and remove the Docker image"
|
||||
@echo " serve - Start server.py locally on port 8080"
|
||||
@echo " debug - Build and run debug Docker container with limited scrape (port $(DEBUG_PORT))"
|
||||
@echo " debug-stop - Stop and remove the debug Docker container"
|
||||
@echo " help - Show this help message"
|
||||
|
||||
build:
|
||||
@@ -59,6 +66,27 @@ validation-stop:
|
||||
@docker rm $(VALIDATION_CONTAINER) 2>/dev/null || true
|
||||
@echo "Validation container stopped and removed"
|
||||
|
||||
debug: build
|
||||
@docker stop $(DEBUG_CONTAINER) 2>/dev/null || true
|
||||
@docker rm $(DEBUG_CONTAINER) 2>/dev/null || true
|
||||
docker run -d --name $(DEBUG_CONTAINER) \
|
||||
-p $(DEBUG_PORT):8080 \
|
||||
-v $(DEBUG_VOLUME):/app/data \
|
||||
-e LOG_LEVEL=DEBUG \
|
||||
$(IMAGE_NAME)
|
||||
@sleep 2
|
||||
docker exec $(DEBUG_CONTAINER) bash /app/run_all.sh --max-pages 1 --max-properties 10
|
||||
@echo "Debug app at http://localhost:$(DEBUG_PORT)/mapa_bytu.html"
|
||||
@echo "Debug status at http://localhost:$(DEBUG_PORT)/scrapers-status"
|
||||
|
||||
debug-stop:
|
||||
@docker stop $(DEBUG_CONTAINER) 2>/dev/null || true
|
||||
@docker rm $(DEBUG_CONTAINER) 2>/dev/null || true
|
||||
@echo "Debug container stopped and removed"
|
||||
|
||||
serve:
|
||||
DATA_DIR=. SERVER_PORT=$(SERVER_PORT) python3 server.py
|
||||
|
||||
validation-local:
|
||||
./run_all.sh --max-pages 1 --max-properties 10
|
||||
|
||||
|
||||
Reference in New Issue
Block a user