6 Commits

Author SHA1 Message Date
CanbiZ (MickLesk)
0231b72d78 Add telemetry data service and dashboard revamp
Some checks failed
Update GitHub Versions (New) / update-github-versions (push) Has been cancelled
Update Versions from GitHub / update-versions (push) Has been cancelled
Introduce a telemetry data microservice under misc/data: add Dockerfile, entrypoint, migration tools, README, LICENSE and a .gitignore. Increase Docker CACHE_TTL_SECONDS to 300s. Implement extensive dashboard and analytics updates in dashboard.go: add total_all_time and sample_size, return total item counts from fetchRecords (with page/limit handling and a maxRecords guard), raise top-N limits, add a minimum-installs threshold for failed-apps, and numerous UI/style/layout improvements in the embedded DashboardHTML. Minor formatting tweak to misc/api.func.
2026-02-12 13:10:06 +01:00
CanbiZ (MickLesk)
d32b00ff31 Add weekly reports, cleanup, and dashboard UI
Introduce weekly summary reports and a cleanup job, enhance dashboard UI, and adjust telemetry/build settings.

- Add REPO_SOURCE to misc/api.func and include repo_source in telemetry payloads.
- Implement weekly report generation/scheduling in alerts.go: new data types, HTML/plain templates, scheduler, SendWeeklyReport/TestWeeklyReport, and email/HTML helpers.
- Add Cleaner (misc/data/cleanup.go) to detect and mark stuck installations as 'unknown' with scheduling and manual trigger APIs.
- Enhance dashboard backend/frontend (misc/data/dashboard.go): optional days filter (allow 'All'), increase fetch page size, simplify fetchRecords, add quick filter buttons, detail & health modals, improved styles and chart options, and client-side record detail view.
- Update Dockerfile (misc/data/Dockerfile): rename binaries to telemetry-service and build migrate from ./migration/migrate.go; copy adjusted in final image.
- Add migration tooling (misc/data/migration/migrate.sh and migration.go) and other small service changes.

These changes add operational reporting and cleanup capabilities, improve observability and UX of the dashboard, and align build and telemetry identifiers for the service.
2026-02-11 12:19:30 +01:00
MickLesk
0226a043b8 feat(telemetry): add caching, alerts, migration & dashboard improvements
- Add Redis/in-memory caching layer (cache.go)
- Add SMTP alerting for high failure rates (alerts.go)
- Add data migration script from old API (migrate.go)
- Add docker-compose.yml for easy deployment
- Move dashboard to / with redirect from /dashboard
- Add dark/light mode toggle
- Add error analysis and failed apps statistics
- Add PVE version and LXC/VM type stats
- Add /metrics Prometheus endpoint
- Add /api/records pagination endpoint
- Add CSV export functionality
- Enhanced healthcheck with PB connection status

New ENV vars:
- Cache: ENABLE_CACHE, CACHE_TTL_SECONDS, ENABLE_REDIS, REDIS_URL
- Alerts: ALERT_ENABLED, SMTP_*, ALERT_FAILURE_THRESHOLD, etc.
- Migration: RUN_MIGRATION, MIGRATION_REQUIRED, MIGRATION_SOURCE_URL
2026-02-09 18:33:33 +01:00
CanbiZ (MickLesk)
7759b53297 Update Dockerfile 2026-02-09 15:53:43 +01:00
CanbiZ (MickLesk)
7bd2ba7b54 Add telemetry-ingest service and Docker updates
Introduce a new telemetry ingestion service (misc/data/service.go) that implements an HTTP server to accept telemetry payloads, validate and sanitize inputs, apply rate limiting, compute dedupe hashes, and forward records to PocketBase with token-based auth. Add module file (misc/data/go.mod) setting module telemetry-ingest and Go version 1.25.5. Update Dockerfile to use golang:1.25-alpine and remove baked-in environment defaults (so runtime envs are required), keeping the build stage and final CMD. These changes add the core ingestion logic, dependency module, and align the build image/version.
2026-02-09 15:53:11 +01:00
CanbiZ (MickLesk)
7f0ca0f9d0 Add Dockerfile and move ingest service
Containerize the telemetry ingest service and reorganize source layout. Added misc/data/Dockerfile with a multi-stage build (golang:1.23-alpine -> alpine:3.23) to produce /app/telemetry-ingest, run as a non-root user, expose :8080, and provide default env vars for configuration (LISTEN_ADDR, MAX_BODY_BYTES, RATE_LIMIT_RPM, RATE_BURST, RATE_KEY_MODE, ENABLE_REQUEST_LOGGING, UPSTREAM_TIMEOUT_MS). Renamed misc/ingest.go to misc/data/service.go to reflect the new directory structure.
2026-02-09 15:38:29 +01:00