Add Prometheus metrics endpoint for browser VM observability (#309)

## Summary

Adds a `GET /metrics` endpoint (Prometheus text format, dedicated
listener on `METRICS_PORT`, default `10002`) so an external collector
can scrape per-VM browser and system metrics and aggregate them across
the fleet. Metrics carry no per-session or per-user labels — instance
identity is attached (and aggregated away) by the scraper.

### Chrome metrics — `lib/metrics/chrome.go`

Read browser-level over CDP (`Browser.getHistograms`,
`Browser.getVersion`, `Target.getTargets`): no page attach, no script
injection, no navigation — invisible to automation running in the
browser.

| Metric | Meaning |
| --- | --- |
| `kernel_chromium_uma{histogram=...}` | Prometheus histogram
(`_bucket`/`_sum`/`_count`) per curated UMA histogram: FCP, LCP,
DOMContentLoaded, load event, parse start (~TTFB), INP, FID,
interactions per page, CLS, per-page CPU ms and
browser-start-to-first-paint. Cumulative since browser start, so
`rate()`/`increase()` work naturally. |
| `kernel_chromium_up` | CDP endpoint reachable and responsive |
| `kernel_chromium_info{product}` | Browser version (fleet version
distribution) |
| `kernel_chromium_pages` | Open page targets (tabs) |

Chrome's sparse `[low, high)` UMA buckets map to cumulative `le` bounds;
the overflow bucket folds into `+Inf`. The curated list lives in
`DefaultUMAHistograms`; histograms with no samples yet are simply
absent.

### VM / process metrics — `lib/metrics/system.go`

`kernel_vm_cpu_seconds_total{mode}`,
`kernel_vm_memory_{total,available}_bytes`, `kernel_vm_load1`,
`kernel_vm_uptime_seconds` (monotonic — excludes suspended time),
`kernel_vm_network_{receive,transmit}_bytes_total`,
`kernel_vm_disk_{total,free}_bytes{mount="/"}`,
`kernel_chromium_processes`, `kernel_chromium_memory_rss_bytes`.

## Design notes

- **Separate listener, no scale-to-zero middleware**: periodic scrapes
must not count as session activity or hold VMs awake. Also skips
per-request logging so scrapes don't drown the logs.
- **Not in the OpenAPI spec**: this is an internal scrape surface,
following the `/internal/fork-identity` precedent.
- **Browser-process histograms only**: `Browser.getHistograms` cannot
see renderer/child-process histograms (`EventLatency.*`,
`Graphics.Smoothness.*`, `Blink.*`) — those only merge into the
browser's recorder when a UMA log is staged or `chrome://histograms` is
opened, neither of which happens on these images. Verified empirically
against a live browser; the curated set is restricted to histograms that
populate without a sync.
- **No new dependencies**: the exposition format is rendered directly;
scrape cost is a few small CDP round-trips + `/proc` reads (~50ms).
- New `cdpclient` methods `GetHistograms` and `CountPageTargets` follow
the existing minimal-client patterns.

## Test plan

- [x] Unit tests for exposition rendering/escaping, UMA→Prometheus
bucket conversion (incl. exact-name filtering and overflow fold), /proc
parsers, and the chrome collector against a fake CDP server
- [x] Full unit suite (`go vet` + `go test -race`, non-e2e) passes
- [x] Validated the real `cmd/api` binary inside two live browser VMs
(both image variants) on alternate ports: all metric families present
with sane values; page-load histogram sums matched values read
independently via raw CDP
- [ ] e2e suite not run (requires pre-built Docker images); the endpoint
has no interaction with existing routes

## Sample output (from a live VM)

```
kernel_chromium_up 1
kernel_chromium_info{product="Chrome/145.0.7632.75"} 1
kernel_chromium_pages 1
kernel_chromium_uma_bucket{histogram="PageLoad.PaintTiming.NavigationToFirstContentfulPaint",le="885"} 1
kernel_chromium_uma_bucket{histogram="PageLoad.PaintTiming.NavigationToFirstContentfulPaint",le="1240"} 2
kernel_chromium_uma_bucket{histogram="PageLoad.PaintTiming.NavigationToFirstContentfulPaint",le="+Inf"} 2
kernel_chromium_uma_sum{histogram="PageLoad.PaintTiming.NavigationToFirstContentfulPaint"} 1977
kernel_chromium_uma_count{histogram="PageLoad.PaintTiming.NavigationToFirstContentfulPaint"} 2
kernel_vm_cpu_seconds_total{mode="user"} 75.1
kernel_chromium_processes 8
kernel_chromium_memory_rss_bytes 1.277575168e+09
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> New exposed listener and CDP/nvidia-smi work on each scrape add
operational surface and load; metrics path is intentionally isolated
from scale-to-zero and the public API.
> 
> **Overview**
> Adds a **dedicated Prometheus scrape surface** on `METRICS_PORT`
(default `10002`) with `GET /metrics`, wired in `cmd/api` on its own
HTTP listener **without** scale-to-zero middleware or request logging so
periodic scrapes do not keep VMs awake or flood logs.
> 
> Introduces **`lib/metrics`** with a custom text exposition handler
(serialized scrapes, per-collector timeouts, failed collectors dropped
without partial families) and three collectors: **Chrome**
(browser-level CDP: `kernel_chromium_up`, version, page count, curated
UMA histograms re-bucketed for fleet aggregation), **system** (`/proc`
CPU/memory/load/PSI/network/disk plus Chromium process count/RSS), and
**GPU** (`nvidia-smi` when present, else `kernel_gpu_present 0`).
> 
> Extends **`cdpclient`** with `GetHistograms` and `CountPageTargets`.
Config and README document `METRICS_PORT`; config tests cover the new
default and override.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
f993190. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: sjmiller609 <7516283+sjmiller609@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

Author: sjmiller609

server/README.md

@@ -51,6 +51,7 @@ Configure the server using environment variables:
 | Variable       | Default   | Description                                 |
 | -------------- | --------- | ------------------------------------------- |
 | `PORT`         | `10001`   | HTTP server port                            |
+| `METRICS_PORT` | `10002`   | Prometheus metrics port (`GET /metrics`)    |
 | `FRAME_RATE`   | `10`      | Default recording framerate (fps)           |
 | `DISPLAY_NUM`  | `1`       | Display/screen number to capture            |
 | `MAX_SIZE_MB`  | `500`     | Default maximum file size (MB)              |

server/cmd/api/main.go

@@ -27,6 +27,7 @@ import (
 	"github.com/kernel/kernel-images/server/lib/devtoolsproxy"
 	"github.com/kernel/kernel-images/server/lib/events"
 	"github.com/kernel/kernel-images/server/lib/logger"
+	"github.com/kernel/kernel-images/server/lib/metrics"
 	"github.com/kernel/kernel-images/server/lib/nekoclient"
 	oapi "github.com/kernel/kernel-images/server/lib/oapi"
 	"github.com/kernel/kernel-images/server/lib/recorder"
@@ -257,6 +258,22 @@ func main() {
 		Handler: rChromeDriver,
 	}
 
+	// Prometheus metrics for external collection. Served on its own
+	// listener, deliberately without the scale-to-zero middleware (periodic
+	// scrapes must not count as session activity) and without per-request
+	// logging (scrapes would drown the logs).
+	rMetrics := chi.NewRouter()
+	rMetrics.Use(chiMiddleware.Recoverer)
+	rMetrics.Method(http.MethodGet, "/metrics", metrics.Handler(slogger,
+		metrics.NewChromeCollector(upstreamMgr),
+		metrics.NewGPUCollector(),
+		metrics.NewSystemCollector(),
+	))
+	srvMetrics := &http.Server{
+		Addr:    fmt.Sprintf("0.0.0.0:%d", config.MetricsPort),
+		Handler: rMetrics,
+	}
+
 	go func() {
 		slogger.Info("http server starting", "addr", srv.Addr)
 		if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
@@ -281,6 +298,14 @@ func main() {
 		}
 	}()
 
+	go func() {
+		slogger.Info("metrics server starting", "addr", srvMetrics.Addr)
+		if err := srvMetrics.ListenAndServe(); err != nil && err != http.ErrServerClosed {
+			slogger.Error("metrics server failed", "err", err)
+			stop()
+		}
+	}()
+
 	// graceful shutdown
 	<-ctx.Done()
 	slogger.Info("shutdown signal received")
@@ -308,6 +333,9 @@ func main() {
 	g.Go(func() error {
 		return srvChromeDriver.Shutdown(shutdownCtx)
 	})
+	g.Go(func() error {
+		return srvMetrics.Shutdown(shutdownCtx)
+	})
 
 	if err := g.Wait(); err != nil {
 		slogger.Error("server failed to shutdown", "err", err)

server/cmd/config/config.go

@@ -13,6 +13,11 @@ type Config struct {
 	// Server configuration
 	Port int `envconfig:"PORT" default:"10001"`
 
+	// Port for the Prometheus metrics endpoint. Served on a separate
+	// listener so scrapes bypass the scale-to-zero middleware and the
+	// external API surface.
+	MetricsPort int `envconfig:"METRICS_PORT" default:"10002"`
+
 	// Recording configuration
 	FrameRate   int    `envconfig:"FRAME_RATE" default:"10"`
 	DisplayNum  int    `envconfig:"DISPLAY_NUM" default:"1"`
@@ -57,6 +62,7 @@ func (c *Config) LogValue() slog.Value {
 	}
 	return slog.GroupValue(
 		slog.Int("port", c.Port),
+		slog.Int("metrics_port", c.MetricsPort),
 		slog.Int("frame_rate", c.FrameRate),
 		slog.Int("display_num", c.DisplayNum),
 		slog.Int("max_size_mb", c.MaxSizeInMB),

server/cmd/config/config_test.go

@@ -19,6 +19,7 @@ func TestLoad(t *testing.T) {
 			env:  map[string]string{},
 			wantCfg: &Config{
 				Port:                     10001,
+				MetricsPort:              10002,
 				FrameRate:                10,
 				DisplayNum:               1,
 				MaxSizeInMB:              500,
@@ -37,6 +38,7 @@ func TestLoad(t *testing.T) {
 			name: "custom valid env",
 			env: map[string]string{
 				"PORT":                       "12345",
+				"METRICS_PORT":               "12346",
 				"FRAME_RATE":                 "20",
 				"DISPLAY_NUM":                "2",
 				"MAX_SIZE_MB":                "250",
@@ -51,6 +53,7 @@ func TestLoad(t *testing.T) {
 			},
 			wantCfg: &Config{
 				Port:                     12345,
+				MetricsPort:              12346,
 				FrameRate:                20,
 				DisplayNum:               2,
 				MaxSizeInMB:              250,
@@ -73,6 +76,7 @@ func TestLoad(t *testing.T) {
 			},
 			wantCfg: &Config{
 				Port:                     10001,
+				MetricsPort:              10002,
 				FrameRate:                10,
 				DisplayNum:               1,
 				MaxSizeInMB:              500,

server/lib/cdpclient/cdpclient.go

@@ -133,6 +133,64 @@ func (c *Client) GetBrowserVersion(ctx context.Context) (*BrowserVersion, error)
 	return &v, nil
 }
 
+// Histogram is a snapshot of a Chrome UMA histogram as returned by
+// Browser.getHistograms. Values are cumulative since browser start and the
+// units follow the UMA definition of the histogram (PageLoad timings are
+// milliseconds). Only buckets with at least one sample are present.
+type Histogram struct {
+	Name    string            `json:"name"`
+	Sum     int64             `json:"sum"`
+	Count   int64             `json:"count"`
+	Buckets []HistogramBucket `json:"buckets"`
+}
+
+// HistogramBucket is a [Low, High) bucket of a Histogram.
+type HistogramBucket struct {
+	Low   int64 `json:"low"`
+	High  int64 `json:"high"`
+	Count int64 `json:"count"`
+}
+
+// GetHistograms sends Browser.getHistograms, a browser-level command that
+// reads Chrome's in-memory UMA histograms without attaching to any page.
+// query is a substring filter on the histogram name; empty returns all.
+func (c *Client) GetHistograms(ctx context.Context, query string) ([]Histogram, error) {
+	raw, err := c.send(ctx, "Browser.getHistograms", map[string]any{"query": query}, "")
+	if err != nil {
+		return nil, fmt.Errorf("Browser.getHistograms: %w", err)
+	}
+	var result struct {
+		Histograms []Histogram `json:"histograms"`
+	}
+	if err := json.Unmarshal(raw, &result); err != nil {
+		return nil, fmt.Errorf("unmarshal Browser.getHistograms: %w", err)
+	}
+	return result.Histograms, nil
+}
+
+// CountPageTargets returns the number of open page targets.
+func (c *Client) CountPageTargets(ctx context.Context) (int, error) {
+	targetsResult, err := c.send(ctx, "Target.getTargets", nil, "")
+	if err != nil {
+		return 0, fmt.Errorf("Target.getTargets: %w", err)
+	}
+	var targets struct {
+		TargetInfos []struct {
+			Type string `json:"type"`
+		} `json:"targetInfos"`
+	}
+	if err := json.Unmarshal(targetsResult, &targets); err != nil {
+		return 0, fmt.Errorf("unmarshal targets: %w", err)
+	}
+	n := 0
+	for _, t := range targets.TargetInfos {
+		if t.Type == "page" {
+			n++
+		}
+	}
+	return n, nil
+}
+
 // DispatchStartURL closes extra page targets and dispatches a navigation on the
 // first page target. It does not wait for lifecycle events; Chrome owns the
 // eventual navigation result.