CanbiZ dab67f7980 Updates

2025-12-01 10:44:40 +01:00

18 KiB

Raw Blame History

API.func Wiki

A telemetry and diagnostics module providing anonymous statistics collection and API integration with the Community-Scripts infrastructure for tracking container/VM creation metrics and installation success/failure data.

Overview

The API.func module provides anonymous telemetry reporting to Community-Scripts infrastructure, enabling:

✅ Container/VM creation statistics collection
✅ Installation success/failure tracking
✅ Comprehensive exit code mapping and explanation
✅ Anonymous session-based tracking (UUID)
✅ Privacy-respecting data collection (no personal data)
✅ Opt-out capability via DIAGNOSTICS setting
✅ Consistent error reporting across all scripts

Integration Points

# In container build scripts (on Proxmox host):
source <(curl -fsSL .../api.func)
post_to_api          # Report container creation
post_update_to_api   # Report installation completion

# Error handling (in all scripts):
source <(curl -fsSL .../error_handler.func)
# explain_exit_code shared for consistent mappings

Data Flow

Container/VM Creation
         ↓
    post_to_api()
         ↓
Community-Scripts API
         ↓
Anonymous Statistics
(No personal data)

Exit Code Reference

Category 1: Generic / Shell Errors

Code	Meaning	Recovery
1	General error / Operation not permitted	Check permissions, re-run command
2	Misuse of shell builtins (syntax error)	Fix shell syntax, validate script
126	Command invoked cannot execute	Fix file permissions (chmod +x)
127	Command not found	Install missing package or tool
128	Invalid argument to exit	Check exit code parameter (0-255)
130	Terminated by Ctrl+C (SIGINT)	User interrupted - retry manually
137	Killed (SIGKILL / Out of memory)	Insufficient RAM - increase allocation
139	Segmentation fault (core dumped)	Serious application bug - contact support
143	Terminated (SIGTERM)	System shutdown or manual termination

Category 2: Package Manager Errors

Code	Meaning	Recovery
100	APT: Package manager error (broken packages)	Run `apt --fix-broken install`
101	APT: Configuration error (bad sources.list)	Fix /etc/apt/sources.list, re-run apt update
255	DPKG: Fatal internal error	Run `dpkg --configure -a`

Category 3: Node.js / npm Errors

Code	Meaning	Recovery
243	Node.js: Out of memory (heap out of memory)	Increase container RAM, reduce workload
245	Node.js: Invalid command-line option	Check node/npm arguments
246	Node.js: Internal JavaScript Parse Error	Update Node.js version
247	Node.js: Fatal internal error	Check Node.js installation integrity
248	Node.js: Invalid C++ addon / N-API failure	Rebuild native modules
249	Node.js: Inspector error	Disable debugger, retry
254	npm/pnpm/yarn: Unknown fatal error	Check package.json, clear cache

Category 4: Python Errors

Code	Meaning	Recovery
210	Python: Virtualenv / uv environment missing	Recreate virtual environment
211	Python: Dependency resolution failed	Check package versions, fix conflicts
212	Python: Installation aborted (EXTERNALLY-MANAGED)	Use venv or remove marker file

Category 5: Database Errors

PostgreSQL

Code	Meaning	Recovery
231	Connection failed (server not running)	Start PostgreSQL service
232	Authentication failed (bad user/password)	Verify credentials
233	Database does not exist	Create database: `createdb dbname`
234	Fatal error in query / syntax error	Fix SQL syntax

MySQL / MariaDB

Code	Meaning	Recovery
241	Connection failed (server not running)	Start MySQL/MariaDB service
242	Authentication failed (bad user/password)	Reset password, verify credentials
243	Database does not exist	Create database: `CREATE DATABASE dbname;`
244	Fatal error in query / syntax error	Fix SQL syntax

MongoDB

Code	Meaning	Recovery
251	Connection failed (server not running)	Start MongoDB daemon
252	Authentication failed (bad user/password)	Verify credentials, reset if needed
253	Database not found	Create database in MongoDB shell
254	Fatal query error	Check query syntax

Category 6: Proxmox Custom Codes

Code	Meaning	Recovery
200	Failed to create lock file	Check /tmp permissions
203	Missing CTID variable	CTID must be provided to script
204	Missing PCT_OSTYPE variable	OS type not detected
205	Invalid CTID (<100)	Container ID must be >= 100
206	CTID already in use	Check `pct list`, remove conflicting container
207	Password contains special characters	Use alphanumeric only for passwords
208	Invalid configuration format	Check DNS/MAC/Network format
209	Container creation failed	Check pct create output for details
210	Cluster not quorate	Ensure cluster nodes are online
211	Timeout waiting for template lock	Wait for concurrent downloads to finish
214	Not enough storage space	Free up disk space or expand storage
215	Container created but not listed	Check /etc/pve/lxc/ for config files
216	RootFS entry missing in config	Incomplete container creation
217	Storage does not support rootdir	Use compatible storage backend
218	Template corrupted or incomplete	Re-download template
220	Unable to resolve template path	Verify template availability
221	Template not readable	Fix file permissions
222	Template download failed (3 attempts)	Check network/storage
223	Template not available after download	Storage sync issue
225	No template for OS/Version	Run `pveam available` to see options
231	LXC stack upgrade/retry failed	Update pve-container package

Telemetry Functions

`explain_exit_code()`

Purpose: Maps numeric exit codes to human-readable error descriptions. Shared between api.func and error_handler.func for consistency.

Signature:

explain_exit_code()

Parameters:

$1 - Numeric exit code (0-255)

Returns: Human-readable description string

Supported Codes:

1-2, 126-128, 130, 137, 139, 143 (Shell)
100-101, 255 (Package managers)
210-212 (Python)
231-234 (PostgreSQL)
241-244 (MySQL/MariaDB)
243-249, 254 (Node.js/npm)
251-254 (MongoDB)
200-231 (Proxmox custom)

Default: Returns "Unknown error" for unmapped codes

Usage Examples:

# Example 1: Common error
explain_exit_code 127
# Output: "Command not found"

# Example 2: Database error
explain_exit_code 241
# Output: "MySQL/MariaDB: Connection failed (server not running / wrong socket)"

# Example 3: Custom Proxmox error
explain_exit_code 206
# Output: "Custom: CTID already in use (check 'pct list' and /etc/pve/lxc/)"

# Example 4: Unknown code
explain_exit_code 999
# Output: "Unknown error"

`post_to_api()`

Purpose: Sends LXC container creation statistics to Community-Scripts telemetry API.

Signature:

post_to_api()

Parameters: None (uses global environment variables)

Returns: No explicit return value (curl result stored in RESPONSE if diagnostics enabled)

Requirements (Silent fail if not met):

curl command available
DIAGNOSTICS="yes"
RANDOM_UUID is set
Executed on Proxmox host (has access to pveversion)

Environment Variables Used:

CT_TYPE - Container type (privileged=1, unprivileged=0)
DISK_SIZE - Allocated disk in GB
CORE_COUNT - CPU core count
RAM_SIZE - RAM allocated in MB
var_os - Operating system name
var_version - OS version
NSAPP - Normalized application name
METHOD - Installation method (default, template, etc.)
DIAGNOSTICS - Enable telemetry (yes/no)
RANDOM_UUID - Session UUID for tracking

API Endpoint: http://api.community-scripts.org/dev/upload

Payload Structure:

{
    "ct_type": 1,                    // Privileged (1) or Unprivileged (0)
    "type": "lxc",                   // Always "lxc" for containers
    "disk_size": 8,                  // GB
    "core_count": 2,                 // CPU cores
    "ram_size": 2048,                // MB
    "os_type": "debian",             // OS name
    "os_version": "12",              // OS version
    "nsapp": "myapp",                // Application name
    "method": "default",             // Setup method
    "pve_version": "8.2.2",         // Proxmox VE version
    "status": "installing",          // Current status
    "random_id": "550e8400-e29b"    // Session UUID (anonymous)
}

Usage Examples:

# Example 1: Successful API post
CT_TYPE=1
DISK_SIZE=20
CORE_COUNT=4
RAM_SIZE=4096
var_os="ubuntu"
var_version="22.04"
NSAPP="jellyfin"
METHOD="default"
DIAGNOSTICS="yes"
RANDOM_UUID="550e8400-e29b-41d4-a716-446655440000"

post_to_api
# Result: Statistics sent to API (silently, no output)

# Example 2: Diagnostics disabled (opt-out)
DIAGNOSTICS="no"
post_to_api
# Result: Function returns immediately, no API call

# Example 3: Missing curl
DIAGNOSTICS="yes"
# curl not available in PATH
post_to_api
# Result: Function returns silently (curl requirement not met)

`post_to_api_vm()`

Purpose: Sends VM creation statistics to Community-Scripts API (similar to post_to_api but for virtual machines).

Signature:

post_to_api_vm()

Parameters: None (uses global environment variables)

Returns: No explicit return value

Requirements: Same as post_to_api()

Environment Variables Used:

VMID - Virtual machine ID
VM_TYPE - VM type (kvm, etc.)
VM_CORES - CPU core count
VM_RAM - RAM in MB
VM_DISK - Disk in GB
VM_OS - Operating system
VM_VERSION - OS version
VM_APP - Application name
DIAGNOSTICS - Enable telemetry
RANDOM_UUID - Session UUID

Payload Structure (similar to containers but for VMs):

{
    "vm_id": 100,
    "type": "qemu",
    "vm_cores": 4,
    "vm_ram": 4096,
    "vm_disk": 20,
    "vm_os": "ubuntu",
    "vm_version": "22.04",
    "vm_app": "jellyfin",
    "pve_version": "8.2.2",
    "status": "installing",
    "random_id": "550e8400-e29b"
}

`post_update_to_api()`

Purpose: Reports installation completion status (success/failure) for container or VM.

Signature:

post_update_to_api()

Parameters: None (uses global environment variables)

Returns: No explicit return value

Requirements: Same as post_to_api()

Environment Variables Used:

RANDOM_UUID - Session UUID (must match initial post_to_api call)
DIAGNOSTICS - Enable telemetry
Installation status parameters

Payload Structure:

{
    "status": "completed",           // "completed" or "failed"
    "random_id": "550e8400-e29b",   // Session UUID
    "exit_code": 0,                  // 0 for success, error code for failure
    "error_explanation": ""          // Error description if failed
}

API Payload Structure

Container Creation Payload

{
    "ct_type": 1,              // 1=Privileged, 0=Unprivileged
    "type": "lxc",             // Always "lxc"
    "disk_size": 20,           // GB
    "core_count": 4,           // CPU cores
    "ram_size": 4096,          // MB
    "os_type": "debian",       // Distribution name
    "os_version": "12",        // Version number
    "nsapp": "jellyfin",       // Application name
    "method": "default",       // Setup method
    "pve_version": "8.2.2",   // Proxmox VE version
    "status": "installing",    // Current phase
    "random_id": "550e8400"   // Unique session ID
}

VM Creation Payload

{
    "vm_id": 100,
    "type": "qemu",
    "vm_cores": 4,
    "vm_ram": 4096,
    "vm_disk": 20,
    "vm_os": "ubuntu",
    "vm_version": "22.04",
    "vm_app": "jellyfin",
    "pve_version": "8.2.2",
    "status": "installing",
    "random_id": "550e8400"
}

Update/Completion Payload

{
    "status": "completed",
    "random_id": "550e8400",
    "exit_code": 0,
    "error_explanation": ""
}

Privacy & Opt-Out

Privacy Policy

Community-Scripts telemetry is designed to be privacy-respecting:

✅ Anonymous: No personal data collected
✅ Session-based: UUID allows correlation without identification
✅ Aggregated: Only statistics are stored, never raw logs
✅ Opt-out capable: Single environment variable disables all telemetry
✅ No tracking: UUID cannot be linked to user identity
✅ No credentials: Passwords, SSH keys never transmitted

Opt-Out Methods

Method 1: Environment Variable (Single Script)

DIAGNOSTICS="no" bash ct/myapp.sh

Method 2: Script Header (Persistent)

#!/bin/bash
export DIAGNOSTICS="no"
# Rest of script continues without telemetry

Method 3: System-wide Configuration

# In /etc/environment or ~/.bashrc
export DIAGNOSTICS="no"

What Data Is Collected

Data	Why	Shared?
Container/VM specs (cores, RAM, disk)	Understand deployment patterns	Yes, aggregated
OS type/version	Track popular distributions	Yes, aggregated
Application name	Understand popular apps	Yes, aggregated
Method (standard vs. custom)	Measure feature usage	Yes, aggregated
Success/failure status	Identify issues	Yes, aggregated
Exit codes	Debug failures	Yes, anonymized

What Data Is NOT Collected

❌ Container/VM hostnames
❌ IP addresses
❌ User credentials
❌ SSH keys or secrets
❌ Application data
❌ System logs
❌ Any personal information

Error Mapping

Mapping Strategy

Exit codes are categorized by source:

Exit Code Range | Source | Handling
0              | Success | Not reported to API
1-2            | Shell/Script | Generic error
100-101, 255   | Package managers | APT/DPKG specific
126-128        | Command execution | Permission/not found
130, 143       | Signals | User interrupt/termination
137, 139       | Kernel | OOM/segfault
200-231        | Proxmox custom | Container creation issues
210-212        | Python | Python environment issues
231-234        | PostgreSQL | Database connection issues
241-244        | MySQL/MariaDB | Database connection issues
243-249, 254   | Node.js/npm | Runtime errors
251-254        | MongoDB | Database connection issues

Custom Exit Code Usage

Scripts can define custom exit codes:

# Example: Custom validation failure
if [[ "$CTID" -lt 100 ]]; then
  echo "Container ID must be >= 100"
  exit 205  # Custom Proxmox code
fi

Best Practices

1. Always Initialize RANDOM_UUID

# Generate unique session ID for tracking
RANDOM_UUID="$(cat /proc/sys/kernel/random/uuid)"

# Use first 8 chars for short session ID (logs)
SESSION_ID="${RANDOM_UUID:0:8}"
BUILD_LOG="/tmp/create-lxc-${SESSION_ID}.log"

2. Call post_to_api Early

# Call post_to_api right after container creation starts
# This tracks attempt, even if installation fails

variables() {
  RANDOM_UUID="$(cat /proc/sys/kernel/random/uuid)"
  # ... other variables ...
}

# Later, in main script:
post_to_api  # Report container creation started
# ... perform installation ...
post_update_to_api  # Report completion

3. Handle Graceful Failures

# Wrap API calls to handle network issues
if command -v curl &>/dev/null; then
  post_to_api || true  # Don't fail if API unavailable
else
  msg_warn "curl not available, telemetry skipped"
fi

4. Respect User Opt-Out

# Check DIAGNOSTICS early and skip all API calls if disabled
if [[ "${DIAGNOSTICS}" != "yes" ]]; then
  msg_info "Anonymous diagnostics disabled"
  return 0  # Skip telemetry
fi

5. Maintain Session Consistency

# Use same RANDOM_UUID throughout lifecycle
RANDOM_UUID="$(cat /proc/sys/kernel/random/uuid)"

post_to_api         # Initial report
# ... installation ...
post_update_to_api  # Final report (same UUID links them)

API Integration

Connecting to API

The API endpoint is:

http://api.community-scripts.org/dev/upload

API Response Handling

# Capture HTTP response code
RESPONSE=$(curl -s -w "%{http_code}" -L -X POST "$API_URL" \
  -H "Content-Type: application/json" \
  -d "$JSON_PAYLOAD") || true

# Extract status code (last 3 digits)
HTTP_CODE="${RESPONSE: -3}"

if [[ "$HTTP_CODE" == "200" ]]; then
  msg_ok "Telemetry submitted successfully"
elif [[ "$HTTP_CODE" == "429" ]]; then
  msg_warn "API rate limited, skipping telemetry"
else
  msg_info "Telemetry API unreachable (this is OK)"
fi

Network Resilience

API calls are best-effort and never block installation:

# Telemetry should never cause container creation to fail
if post_to_api 2>/dev/null; then
  msg_info "Diagnostics transmitted"
fi
# If API unavailable, continue anyway

Contributing

Adding New Exit Codes

Document in the appropriate category section
Update explain_exit_code() in both api.func and error_handler.func
Add recovery suggestions
Test mapping with scripts that use the new code

Testing API Integration

# Test with mock curl (local testing)
DIAGNOSTICS="yes"
RANDOM_UUID="test-uuid-12345678"
curl -X POST http://localhost:8000/dev/upload \
  -H "Content-Type: application/json" \
  -d '{"test": "payload"}'

# Verify payload structure
post_to_api 2>&1 | head -20

Telemetry Reporting Improvements

Suggestions for improvement:

Installation duration tracking
Package version compatibility data
Feature usage analytics
Performance metrics
Custom error codes

Notes

API calls are silent by default and never display sensitive information
Telemetry can be completely disabled via DIAGNOSTICS="no"
RANDOM_UUID must be generated before calling any post functions
Exit code mappings are shared between api.func and error_handler.func for consistency
API is optional - containers work perfectly without telemetry

18 KiB Raw Blame History

API.func Wiki

📋 Table of Contents

Overview

Integration Points

Data Flow

Exit Code Reference

Category 1: Generic / Shell Errors

Category 2: Package Manager Errors

Category 3: Node.js / npm Errors

Category 4: Python Errors

Category 5: Database Errors

PostgreSQL

MySQL / MariaDB

MongoDB

Category 6: Proxmox Custom Codes

Telemetry Functions

explain_exit_code()

post_to_api()

post_to_api_vm()

post_update_to_api()

API Payload Structure

Container Creation Payload

VM Creation Payload

Update/Completion Payload

Privacy & Opt-Out

Privacy Policy

Opt-Out Methods

What Data Is Collected

What Data Is NOT Collected

Error Mapping

Mapping Strategy

Custom Exit Code Usage

Best Practices

1. Always Initialize RANDOM_UUID

2. Call post_to_api Early

3. Handle Graceful Failures

4. Respect User Opt-Out

5. Maintain Session Consistency

API Integration

Connecting to API

API Response Handling

Network Resilience

Contributing

Adding New Exit Codes

Testing API Integration

Telemetry Reporting Improvements

Notes

18 KiB

Raw Blame History

`explain_exit_code()`

`post_to_api()`

`post_to_api_vm()`

`post_update_to_api()`