Monitoring

Yokai combines a fast in-terminal operational view with an optional Prometheus and Grafana stack on each device.

Dashboard

The Dashboard polls the local daemon and groups AI services separately from monitoring services. It shows device and service health plus rolling utilization history.

Fleet and device detail can include:

• CPU, RAM, swap, and disk usage;
• GPU utilization, VRAM, temperature, power, and fan speed;
• container CPU, memory, GPU memory, state, and uptime;
• vLLM or llama.cpp throughput, queue depth, token totals, latency, and KV-cache utilization when available.

Select a service with J / K and press Enter for the inspector. Press Shift+L to open its SSE log stream. In Logs, F toggles follow mode and PgUp / PgDn scroll.

Agent metric endpoints

The daemon polls authenticated JSON metrics from each agent:

GET /metrics
Authorization: Bearer <agent-token>

For Prometheus, each agent also exposes:

GET /metrics/prometheus
Authorization: Bearer <agent-token>

The Prometheus endpoint normalizes service-level LLM metrics under the yokai_ prefix, including:

• yokai_service_up and yokai_service_info;
• prefill and decode tokens per second;
• in-flight and queued requests;
• prompt and generated token counters;
• request result counters;
• TTFT and end-to-end duration histograms;
• KV-cache utilization when the backend supports it.

High-cardinality labels such as container IDs, request IDs, prompts, or client IPs are intentionally excluded.

Provisioned stack

When monitoring is selected during device setup, Yokai provisions:

• Prometheus;
• Grafana;
• node_exporter;
• NVIDIA dcgm-exporter;
• a preconfigured Yokai GPU dashboard.

Prometheus receives the agent token through a credentials file and sends it as a bearer token. Grafana uses the provisioned Prometheus datasource.

JSON automation

yokai status
yokai metrics
yokai metrics --device gpu-box-01
yokai services logs --follow gpu-box-01 <container-id>

These commands are suitable for scripts because normal output is JSON and errors are JSON on stderr.

Monitoring unavailable

The Yokai Dashboard still works when the optional Grafana stack is absent. It reads the daemon’s live cache, which is populated from the agent JSON API. Install or repair the optional stack from the device setup workflow when historical dashboards are required.