Skip to content

A model picker that hides what is dead, and brings it back

For weeks the model picker on mit.nonlocally.org showed six OpenAI-backed models that could not answer: the account had no credits, and every request fell through to a Claude fallback. The job meant to hide dead models ran every fifteen minutes and reported "picker already in sync". It was wrong four separate ways. This is what we measured, and what runs now.

What the job was supposed to do

scripts/model_health_reconciler.py reads LiteLLM's /health, decides which upstreams are hard dead (quota, auth, not-found — never a cold-start timeout), and flips the matching OpenWebUI model records off, then back on when the upstream recovers.

Four independent reasons it did nothing

  1. It read a database that no longer existed. The job opened a SQLite file left over from before the platform moved to Postgres. Copilot's PR #471 replaced that with OpenWebUI's admin API, which is where the fix started.
  2. The marker list did not know OpenAI's current wording. "You have no credits remaining" matched none of quota, billing, insufficient. One marker fixed it.
  3. Records created directly on a LiteLLM model id have no base_model_id. The planner keyed on that field, so gpt-5, [cloud] o3 and friends were silently skipped as "unmanaged". A record's own id is now the route.
  4. LiteLLM appends a 1.2 kB stack trace to every health error, separated by a newline before stack trace:. The trace mentions connection and timeout symbols for every provider error, so a "transient word means transient" rule fired on the trace and classified the dead upstreams as healthy. We now judge only the message head.

The fourth one is the humbling one: a precedence change we made while fixing the third silently re-broke detection, and only re-running the head in the pod against the live payload caught it. Classifier changes get re-run against the live payload now, every time.

The endpoint that was active-only

The review sweep then falsified the central claim with a repro: the list endpoint the job read is OpenWebUI's merged, active-only picker. A hidden model vanishes from it, so nothing could ever be restored. We proved it live by creating a private inactive record: absent from that list, present in /api/v1/models/export. No single admin endpoint holds every record, so the fetch is now the union of /export (presets) and /base (records on a LiteLLM id). The evidence file is committed and pinned by a test, and a source-scan test forbids the old literal.

What runs now

Deployed on 2026-09-02. The first run:

[reconciler] litellm: 24 healthy / 8 unhealthy model_names
ACTIVATE dirk-assistant · [or] o3 · [or] o4-mini · [or] GPT-5
HIDE     [cloud] GPT-5 · [cloud] GPT-5 mini · [cloud] o3 · [cloud] o4-mini · gpt-5 · gpt-5-mini
[reconciler] changed 10 of 10 preset(s)

OpenWebUI's own access log shows exactly ten toggle calls, one per change; the next run reported "already in sync". Every toggle is re-read before and after acting, because the toggle endpoint flips rather than sets. When the OpenAI account is topped up, the six models come back on the next cycle with nobody in the loop.

The four restores were models switched off by an unattributed incident the day before, which the same run repaired. The rule is now written down: declared in models.yaml and healthy means active; a model you want hidden on purpose is removed from models.yaml.

PRs #471, #469, #472; issues #470 (closed), #479 (credential follow-up).