Telemetry

True Life includes a full observability stack with OpenTelemetry integration, providing distributed tracing, structured logging, and metrics.

Architecture

Flow: Web UI logs/spans → NUI fetch → FiveM Client → RPC → Server → OTLP HTTP → OTEL Collector → Tempo (traces) / Loki (logs) → Grafana

Server Configuration

Configure via convars in server.cfg:

set otel_enabled "true"
set otel_endpoint "http://localhost:4318"
set otel_service_name "true_life"

Pipeline Initialization

Server

// src/runtime/server/main.ts
import { initServerTelemetry, shutdownServerTelemetry } from "@core/telemetry/server";
 
// During bootstrap
initServerTelemetry({
	serviceName: "true_life",
	flushIntervalMs: 5000,
});
 
// During shutdown
await shutdownServerTelemetry();

Client

// src/runtime/client/main.ts
import { initClientTelemetry, shutdownClientTelemetry } from "@core/telemetry/client";
 
// During bootstrap
initClientTelemetry({
	flushIntervalMs: 2000,
});
 
// During shutdown
await shutdownClientTelemetry();

Web UI

// ui/src/main.tsx
import { initTelemetryForwarder, shutdownTelemetryForwarder } from "@ui/lib/telemetry";
 
// During app initialization
initTelemetryForwarder({
	flushIntervalMs: 1000,
});
 
// On app unmount
await shutdownTelemetryForwarder();

Observability Stack

The infra/ directory contains the full-stack configuration:

infra/
├── otel/                    # OpenTelemetry Collector
│   └── otel-collector-config.yaml
├── tempo/                   # Tempo (distributed tracing)
├── loki/                    # Loki (log aggregation)
├── grafana/                 # Grafana (visualization)
│   └── provisioning/
│       └── dashboards/      # Pre-configured dashboards
└── prometheus/              # Prometheus (metrics)

Starting the Stack

# Start full observability stack
docker compose up -d otel-collector tempo loki grafana prometheus
 
# Or just the essentials
docker compose up -d otel-collector tempo loki grafana

Accessing Services

Service	URL	Description
Grafana	http://localhost:3001	Dashboards and visualization
Prometheus	http://localhost:9090	Metrics queries
Tempo	http://localhost:3200	Trace queries
Loki	http://localhost:3100	Log queries
OTEL Collector	http://localhost:4318	OTLP HTTP receiver

Grafana Dashboards

Pre-configured dashboards are available:

• Service Overview - Request rates, error rates, latencies
• Trace Explorer - Search and visualize distributed traces
• Log Explorer - Search structured logs with trace correlation
• Player Metrics - Online players, resource usage

Exploring Traces

1. Open Grafana at http://localhost:3001
2. Go to Explore → Select Tempo data source
3. Search by trace ID, service, or operation name
4. Click on a trace to see the full span hierarchy

Log Correlation

Logs include trace context for correlation:

1. Find a trace in Tempo
2. Click "Logs for this trace" to see related logs in Loki
3. Or search Loki with {traceId="..."} filter

OTEL Collector Configuration

The collector receives telemetry and routes it to backends:

# infra/otel/otel-collector-config.yaml
receivers:
    otlp:
        protocols:
            grpc:
                endpoint: 0.0.0.0:4317
            http:
                endpoint: 0.0.0.0:4318
 
processors:
    batch:
        timeout: 1s
        send_batch_size: 1024
 
exporters:
    otlp/tempo:
        endpoint: tempo:4317
        tls:
            insecure: true
    loki:
        endpoint: http://loki:3100/loki/api/v1/push
 
service:
    pipelines:
        traces:
            receivers: [otlp]
            processors: [batch]
            exporters: [otlp/tempo]
        logs:
            receivers: [otlp]
            processors: [batch]
            exporters: [loki]

Custom Metrics

Expose custom metrics via Prometheus:

// Custom counter example
import { incrementCounter, setGauge } from "@core/metrics";
 
// Increment a counter
incrementCounter("banking_transfers_total", { status: "success" });
 
// Set a gauge
setGauge("players_online", playerCount);

Best Practices

1. Use spans for operations - Wrap significant operations
2. Include relevant attributes - Player IDs, amounts, etc.
3. Set appropriate log levels - Avoid noise
4. Monitor error rates - Set up alerts in Grafana
5. Sample high-volume traces - Reduce storage costs
6. Correlate logs with traces - Use trace context