brettsophus.com
← all posts

Why a GBDT is the right brain for a Pi Zero 2 W

#edge-ai#gbdt#raspberry-pi#liten

I’ve been building liten — a tiny API gateway with six runtime deps — and the next thing I want it to do is pick which upstream a request goes to, not just whether it’s allowed. The constraint that made this interesting: the whole thing still needs to run on a Raspberry Pi Zero 2 W. 512 MB of RAM, four Cortex-A53 cores at 1 GHz, no swap headroom worth using.

When I sat down to pick a model, a gradient-boosted decision tree wasn’t a compromise. It was the only thing that fit.

What liten actually needs to decide

Every request that hits liten already produces a row of tabular features: method, path, hour of day, rolling p95 latency for that route, request size, user-agent family, API key tier. The label is the outcome — did that request come back ok, slow, error, or throttled?

About fifteen columns and a class. No sequence, no image, nothing that needs context from five minutes ago. That’s not a neural net problem. That’s a tree problem.

Three things about GBDTs that matter here:

  • Scale-invariant. No normalization pass, no embedding layer for one-hot columns.
  • Mixed types out of the box. route_key (categorical), req_bytes (int), has_query (bool) all sit in the same table.
  • Retrains in a minute. Not an hour, not on a GPU. On my laptop, on a week of logs.

That last one matters because the plan is for the user to retrain, on their own traffic, on their own machine. Nobody is spinning up CUDA for a router refresh.

The Pi Zero 2 W budget

Here’s what I gave myself for on-device inference:

  • Model artifact ≤ 200 KB
  • Resident memory ≤ 8 MB including runtime
  • Inference ≤ 1 ms p95 per request
  • Zero allocation on the hot path

Try to hit that with a small MLP. The model itself would fit — a quantized 2-layer network with one-hot features is small. But the runtime to score it — ONNX Runtime, TFLite, whatever — costs more resident memory than the entire liten process does today. On a Pi Zero 2 W, “the runtime is bigger than the model” is a real failure mode.

A LightGBM ensemble at, say, num_trees=64, max_depth=6 compiles down to a few hundred if / else branches. Treelite emits it as portable C or WASM. The runtime is a few kilobytes of glue. One traversal per tree, sum, done. On an A53 at 1 GHz that’s comfortably sub-millisecond and it never touches the heap.

The Pi Zero 2 W thermal-throttles the second you stop paying attention, too. Branch-heavy integer code is exactly what a Cortex-A53 is good at. Dense matmul is exactly what it isn’t.

Why not a neural net at all

I reach for neural nets on other projects — elderly-guardian runs YOLOv8 Pose on a Metis. So this isn’t reflex. It’s specific to this problem.

The features already are the model. Half the routing signal comes from recent_route_qps and recent_route_p95_latency — two numbers liten already computes in a ring buffer. A tree can split on “is p95 above 800 ms” and act on it that request. A network has to learn that threshold, with dropout and a schedule, and end up in the same place.

I need to read the decision. When the gateway routes a request the wrong way I want to look at one path down one tree and see why. dump_model() gives me that. Attention weights don’t.

Retraining has to be cheap. Not “cheap on a rented A100.” Cheap on a laptop, in a coffee break. LightGBM does that. A network doesn’t, not once you add the tuning.

The stack, concretely

  • Training: LightGBM in Python, multi-class over outcome.
  • Export: Treelite → WASM in phase one. Native .so in phase two, once I have real Pi latency numbers to argue with.
  • Runtime in liten: an optional peer dep, liten-brain-runtime, that loads the artifact and exposes predict(features) → { label, confidence }. Users who don’t want ML pay nothing — no dep, no bundle, no config surface.
  • Feature parity: one JSON feature-spec (order, one-hot vocab, bucket edges) is emitted by training and consumed by JS. The Python and JS extractors are never allowed to derive vocab independently. That’s the class of bug I don’t want to debug on a Pi.

Why this fits liten

Liten’s whole pitch is “small, one npm install, runs anywhere Node runs.” Adding a model shouldn’t break that pitch. A 150 KB tree ensemble that decides in under a millisecond, on a $15 board, with no GPU and no cloud round-trip — that’s the honest version of “AI at the edge” for a project like this. No thousand-line runtime. No download at boot. Just a compiled decision tree doing what compiled decision trees have always been good at.

More on the actual training pipeline — and whatever surprises come out of the Pi benchmarks — in the next post. Repo is github.com/moorebrett0/liten-brain.