Introduce a telemetry data microservice under misc/data: add Dockerfile, entrypoint, migration tools, README, LICENSE and a .gitignore. Increase Docker CACHE_TTL_SECONDS to 300s. Implement extensive dashboard and analytics updates in dashboard.go: add total_all_time and sample_size, return total item counts from fetchRecords (with page/limit handling and a maxRecords guard), raise top-N limits, add a minimum-installs threshold for failed-apps, and numerous UI/style/layout improvements in the embedded DashboardHTML. Minor formatting tweak to misc/api.func.
Refactor telemetry backend to store all telemetry in a single collection and add repo_source-based filtering.
Key changes:
- Added detect_repo_source() in misc/api.func to auto-detect/export REPO_SOURCE (ProxmoxVE/ProxmoxVED/external) when scripts are sourced.
- Consolidated PocketBase collections into a single default collection (_telemetry_data) across service, migration, and scripts; updated defaults in migrate.go, migration.go, migrate.sh and migration shell scripts.
- Simplified PBClient to use one targetColl and removed collection resolution logic; updated create/update/find/fetch functions to use targetColl.
- Introduced repo_source field (values: "ProxmoxVE", "ProxmoxVED", "external") on telemetry records and telemetry payloads; updated validation and logging.
- Added repo filtering to dashboard endpoints, FetchDashboardData and FetchRecordsPaginated, plus a repo selector in the dashboard UI; default filter is ProxmoxVE (production), with an "all" option.
- Adjusted API handlers and callers to pass repo filters and include repo_source when upserting telemetry.
- Misc: updated comments, error messages, and logging to reflect the new model; added telemetry-service.exe binary.
Purpose: simplify data model (single collection), make telemetry attributable to repository sources, and enable dashboard filtering by repo/source.
Introduce weekly summary reports and a cleanup job, enhance dashboard UI, and adjust telemetry/build settings.
- Add REPO_SOURCE to misc/api.func and include repo_source in telemetry payloads.
- Implement weekly report generation/scheduling in alerts.go: new data types, HTML/plain templates, scheduler, SendWeeklyReport/TestWeeklyReport, and email/HTML helpers.
- Add Cleaner (misc/data/cleanup.go) to detect and mark stuck installations as 'unknown' with scheduling and manual trigger APIs.
- Enhance dashboard backend/frontend (misc/data/dashboard.go): optional days filter (allow 'All'), increase fetch page size, simplify fetchRecords, add quick filter buttons, detail & health modals, improved styles and chart options, and client-side record detail view.
- Update Dockerfile (misc/data/Dockerfile): rename binaries to telemetry-service and build migrate from ./migration/migrate.go; copy adjusted in final image.
- Add migration tooling (misc/data/migration/migrate.sh and migration.go) and other small service changes.
These changes add operational reporting and cleanup capabilities, improve observability and UX of the dashboard, and align build and telemetry identifiers for the service.
Replace unconditional debug echoes in misc/api.func with checks on DEV_MODE so telemetry debug output is only shown when DEV_MODE=true. Make curl fire-and-forget: verbose HTTP code and payload logging only in DEV_MODE, while production uses a silent curl invocation that never blocks or fails. Remove a stray trailing whitespace in detect_ram. Add misc/data/telemetry-ingest.exe binary to repo.
Extend telemetry collection to include GPU model, CPU vendor/model, and RAM speed. misc/api.func: enhance detect_gpu to capture GPU_MODEL and default unknown values; add detect_cpu and detect_ram (dmidecode used for RAM speed), export new globals, and include cpu_*/gpu_model/ram_speed in post_to_api and post_update_to_api payloads. misc/data/service.go: add GPUModel/CPUVendor/CPUModel/RAMSpeed fields to TelemetryIn/Out/StatusUpdate, update PBClient mapping, expand allowed enum values to accept "unknown", sanitize and default empty vendor/passthrough fields to "unknown", and validate new cpu_vendor values. Changes maintain backward compatibility by using "unknown" where data is unavailable.
Dashboard: add sortable table UI and client-side sorting support — CSS for sortable headers, data-sort attributes, default sort on Created (desc), timestamp formatting, header click handling, and inclusion of sort param in paginated fetches. Records now show a formatted Created column with full timestamp in the title. Initialize sortable headers on load.
Telemetry/client: switch to sending a full JSON payload (allows create if initial PATCH failed) and include extra fields (ct_type, disk_size, core_count, ram_size, os_type, os_version, pve_version, method). pve_version is detected when available.
Server: extend FetchRecordsPaginated to accept a sort field, validate allowed sort fields to prevent injection, use the sort when building the PB API request (default -created), and propagate the sort query param from the HTTP handler to the fetch call.
Overall this enables server-side sorted pagination from the dashboard and richer telemetry records.
Clean up formatting in misc/api.func for readability. Normalized spacing in the categorize_error case patterns, removed trailing blank-space lines, and standardized detect_gpu blank-line spacing. Converted heredoc assignments to use multiline $() style for JSON_PAYLOAD in post_tool_to_api, post_addon_to_api, and post_update_to_api_extended, and added a final newline at end of file. No functional changes intended; purely whitespace/formatting cleanup.
Add extended telemetry functions and server-side support: misc/api.func gains helpers (categorize_error, install timer, detect_gpu) and new reporters for tools, addons, and an extended post_update_to_api with duration, GPU and error_category. misc/data/service.go updated to accept and validate new fields (type: tool/addon, tool_name, parent_ct, gpu_vendor, gpu_passthrough, install_duration, error_category), expand allowed enums, include new fields in UpsertTelemetry and mapping, and add input sanitization. Also add telemetry-ingest.exe binary. These changes enable richer telemetry (tool/addon events, GPU info, durations and categorized errors) and server ingestion/validation for them.
Correct telemetry status spelling ("sucess" → "success") and map status values accordingly. Normalize EOF/newline handling in several misc/data files. Major refactor of tools/pve/microcode.sh: update copyright/author, remove the verbose CPU detection/display helpers, simplify current microcode detection, streamline package discovery and selection logic, switch wget to curl, clean up download/install flow (use basename and remove /tmp usage), simplify UI/messages and exit handling, and remove live microcode reload/version post-install checks.
Send only changing fields for status updates and add server-side update flow.
- Trimmed telemetry JSON payload in misc/api.func to include only status, error, and exit_code (removed static fields and pve_version) so updates are minimal.
- Added TelemetryStatusUpdate type and new PBClient methods: FindRecordByRandomID, UpdateTelemetryStatus, and UpsertTelemetry in misc/data/service.go. UpsertTelemetry creates a record for status="installing", otherwise finds the record by random_id and PATCHes only status/error/exit_code (fallbacks to create if not found).
- Relaxed validation logic in validate(): detect updates (status != "installing") and skip certain strict numeric checks for update requests while keeping required fields and other validations.
- Main handler now calls UpsertTelemetry instead of CreateTelemetry and logs generic errors.
These changes allow idempotent, minimal updates to existing telemetry records and avoid repeatedly sending/storing unchanged metadata.
Enhance post_update_to_api robustness and observability: add debug traces for entry, missing curl, duplicate submissions, DIAGNOSTICS/RANDOM_UUID checks, payload/URL output, and HTTP response code capture; make curl non-blocking and tolerant of failures. Also invoke post_update_to_api on installation failure so build/install errors are reported to telemetry. Includes a small comment glyph fix in build.func. Changes keep telemetry as a silent, best-effort path that won't break script execution.
Reformat telemetry pre-checks in misc/api.func for clearer multi-line condition handling and remove stray whitespace around the curl response logging. Tweak misc/build.func comment line (minor encoding/visual cleanup). In misc/data/service.go remove the strict allowedMethod map and its validation so the 'method' field is treated as optional/flexible; keep sanitization and numeric validations unchanged. These changes improve readability and allow custom method values without breaking behavior.
Replace direct PocketBase integration with a fire-and-forget telemetry ingest endpoint and tighten validation. misc/api.func: point to telemetry.community-scripts.org, add TELEMETRY_TIMEOUT, use DIAGNOSTICS=no opt-out, include random_id/NSAPP/status in payloads, unify LXC/VM POSTs, avoid blocking or failing scripts, remove PocketBase record lookup/patch logic. misc/data/service.go: update TelemetryIn/TelemetryOut schemas to match new payload, add stricter sanitization and enum/range validation, adjust hashing/deduplication usage, and update request logging to reflect nsapp/status. Overall: safer, non-blocking telemetry with improved schema validation and GDPR-friendly behavior.
Remove the old Go/Mongo API (api/main.go, go.mod, go.sum, .env.example) and switch telemetry backend to PocketBase (http://db.community-scripts.org). Update documentation and flowcharts to reflect the PocketBase collection (_dev_telemetry_data), new REST endpoints (POST/PATCH/GET), field schema, and revised api.func integration (LXC/VM reporting and status updates). Misc scripts and helpers were adjusted (misc/api.func, misc/build.func, misc/error_handler.func) and a new misc/ingest.go was added. This consolidates telemetry to a hosted PocketBase instance and updates docs and integration points accordingly.
Moved MOTD/SSH setup for dev mode in LXC containers to run after installation and when containers are kept for debugging, improving reliability and clarity. Updated journalctl cleanup to avoid using $STD and suppress errors if systemd-journald is not initialized. Also refactored indentation in misc/api.func for consistency.
Moved the explain_exit_code function directly into api.func, removing external dependency loading. Also cleaned up merge conflict markers and unified documentation for diagnostics_check in build.func.
Expanded inline documentation and section headers across misc/api.func, misc/build.func, misc/core.func, misc/error_handler.func, and misc/install.func for improved maintainability and clarity. Refactored error handling to use centralized explain_exit_code and updated API telemetry/reporting logic. Enhanced modularity and structure of utility, validation, and setup functions, and improved comments for user guidance and developer onboarding.
