Skip to content

GLM-5.3-Flash (glm) through LiteLLM: how the path works and how to use it

Written 2026-09-02 for #500 / #468; serving recovery updated 2026-09-06. Earlier measured tool probes below were run on September 2.

TL;DR

question answer
Callable id glm (display name GLM-5.3-Flash)
Where it runs vLLM, 2x B200 Slurm slice on the AICR cluster (llm up glm-batch --keep on the Mac mini)
How the cluster reaches it LiteLLM alias glm → https://glm.kratuvak.com/v1 (named Cloudflare tunnel vvuq-bridge-mini on the always-on Mac mini) → the mini's 127.0.0.1:8001 (ssh -L, kept open by the llm.keeper.glm-batch launchd agent) → the compute node
Tool calling native. vLLM runs --enable-auto-tool-choice --tool-call-parser glm47; LiteLLM declares supports_function_calling: true so tool_policy.py keeps the tools
Key vLLM's --api-key (same file on the laptop and the mini: ~/.config/llm/keys/glm), in GCP-SM openweb-marimo-glm-api-key → Secret openweb-marimo-glm → env GLM_API_KEY

The path, hop by hop

user / freelancer
  └─ POST https://mit.nonlocally.org/api/chat/completions  {"model":"glm","tools":[…]}
       └─ OpenWebUI (connection #0 = LiteLLM; a duplicate id on a later connection is ignored)
            └─ LiteLLM  alias glm → openai/glm ; tool_policy: capable (explicit flag) ; timeout 300 ; budget billed
                 └─ https://glm.kratuvak.com/v1      (named Cloudflare tunnel vvuq-bridge-mini, TLS by Cloudflare)
                      └─ Mac mini  127.0.0.1:8001     (ssh -L to the Slurm node; llm.keeper.glm re-opens it when the node changes)
                           └─ AICR compute node :8001  vLLM  zai-org/GLM-5.3-Flash

What is not in the path any more for glm: the vvuq-bounties bridge (bridge-mini.kratuvak.com, OpenAI connection vvuq-specialists@mac-mini-2). That bridge forwards messages only, which is why tools "were stripped" before (#500). It still serves the other specialists. Until the LiteLLM route is verified live it also still lists glm; then _GLM_MODEL is dropped from its _TUNNELLED_UPSTREAMS so there is one server of glm.

Operate (on the Mac mini, ssh aadarshs-mac-mini-2)

All three must be true for glm to answer:

llm status glm-batch                 # job RUNNING, tunnel open, api answering at localhost:8001
launchctl list | grep llm.keeper.glm-batch # the keeper (polls every 30 s; re-opens the ssh -L on node change / tunnel death)
cloudflared tunnel info vvuq-bridge-mini   # the named tunnel is connected
curl -s -o /dev/null -w '%{http_code}\n' https://glm.kratuvak.com/v1/models   # 401 = alive, key required
  • The named tunnel's ingress lives in /Users/aadarwal/.cloudflared/config.yml on the mini (not the dashboard): bridge-mini.kratuvak.com → 127.0.0.1:8791, glm.kratuvak.com → 127.0.0.1:8001, then 404. Zone kratuvak.com. cloudflared is started by vvuq-bounties scripts/openwebui_specialist_chat.sh hosted (VVUQ_SPECIALIST_TUNNEL=named, vvuq-bounties #71) and restarted by its watch launchd agent if it dies. The hostname never changes.
  • The current glm-batch profile chains eight-hour b200-batch slices. Startup and scheduler delays can interrupt service; the September 5 recovery used an initial four-hour b200-devel slice with its successor on batch. During a gap, 127.0.0.1:8001 is unavailable and the public endpoint returns 502/530. Check the job and its log: do not assume a prolonged outage is normal weight loading. There is no configured cross-model fallback for glm.
  • Do not also start the legacy glm profile: both profiles use port 8001 and the same served model/key. The active keeper follows glm-batch.
  • Thinking budget: vLLM's chat template defaults to a very long reasoning_effort. The LiteLLM alias passes chat_template_kwargs: {reasoning_effort: high, clear_thinking: true} by default; API callers can override in extra_body.

September 5 outage: require both GPUs on one node

The failing llm-glm-batch slice was allocated across b0004 and b0028. The launcher requested -G 2 without a node constraint. The single vLLM process saw one local GPU and rejected tensor_parallel_size=2 with “World size (2) is larger than the number of available GPUs (1) in this node.” Its early-failure handler then cancelled the queued successor and removed the keep flag. An alive keeper cannot restore an API when no serving job remains.

The cluster launcher /scratch/aadarwal_mit/llm/up.sh now passes sbatch --parsable --nodes=1 ... -G "$GPUS". This matches serve.sh, which starts one host-local vLLM process. Both initial submissions and chained successors use that launcher. The exact patch is retained here; the prior cluster file is up.sh.before-single-node-20260905. The installed file has SHA-256 f61872ea129bed99b588045c2cf2238a3159074bceb6cb354f4a244632889154. The patch was syntax-checked and exercised with fixture sbatch/squeue commands for both initial and dependency-chained submissions before installation.

Recovery started llm up glm-batch --keep from the Mac mini. Slurm job 693797 requested exactly one node and two GPUs; its initial slice was moved to the existing development partition with a four-hour limit to restore service sooner. It started on b0031 with both B200 GPUs visible, and queued successor 693840 on batch. The API key, LiteLLM route and Cloudflare hostname were preserved.

The failed multi-node slice had also left an orphaned SSH listener on the mini: -L 8001:b[0004,0028]:8001 aicr. The keeper's pgrep pattern interpreted those brackets as a regular expression, failed to record the PID, and then could not replace the listener because port 8001 remained occupied. After checking the exact PID/command, only that stale GLM forward was stopped; llm tunnel glm-batch opened the correct b0031:8001 forward. The one-node constraint prevents Slurm node-range syntax from reaching this host-local tunnel path again.

At 20:31 UTC, authenticated public /v1/models, a GLM_OK completion and a forced lean_compile function call all passed with served_model=glm. The unauthenticated model endpoint returned the expected 401 instead of 502.

If a job is pending, use the scheduler directly; the current llm status helper can fail while parsing an empty node field:

ssh aicr 'squeue -u "$USER" -h -n llm-glm-batch -o "%i|%P|%T|%N|%M|%R"'

Verify authenticated /v1/models, a real completion, and forced tool calling before reporting the service recovered. A RUNNING job alone is insufficient.

September 6 recurrence: the successor was held before vLLM started

The four-hour recovery job 693797 timed out at 00:27 UTC. Its eight-hour successor 693840 was allocated on b0006, failed to launch before producing a serving log, and was requeued with priority zero and Reason=launch_failed_requeued_held. The one-node/two-GPU allocation fix was still present. Slurm accounting establishes a launch failure, but the exact node-side cause is not visible in the user job logs.

Recovery preserved the same job and resources: exclude the failed launch node, then release the automatic hold. Job 693840 started on b0025 at 01:53 UTC with an eight-hour limit; successor 700243 was queued on batch. The existing Mac mini keeper followed the new node without changes to the key, hostname, or LiteLLM route. Authenticated completions and forced tool calls then passed both directly and through Open WebUI, with served_model=glm.

The keeper repairs the SSH connection, but it does not release held Slurm jobs. scripts/glm_launch_guard.py adds a separate, single-pass recovery check, run on the cluster as the job owner. Default mode reports a proposed recovery; --apply enables scheduler writes. The Mac mini's llm.guard.glm-batch launch agent runs it every 60 seconds over the existing aicr SSH connection. Launchd does not start a second copy of an already running job, and the CLI takes a file lock to serialize cluster-side invocations that use the same state path. A caller importing recover() must hold that lock.

The guard acts only when the keep flag exists, no GLM allocation is active, and exactly one queued llm-glm-batch job has that automatic launch-failure reason. It verifies the owner, serving command, partition, one-node/two-GPU request, restart count and completed dependency. The latest REQUEUED accounting timestamp must identify one failed node; conflicting nodes at that timestamp fail closed. Existing exclusions are preserved, with at most three excluded nodes. Ordinary queue waits, operator/admin holds, legacy-profile jobs, missing keep flags, ambiguous jobs, and changed job state cause no release.

Each attempt is recorded atomically before any scheduler write. There is a five-minute cooldown, a maximum of three attempts per hour across jobs, and three attempts per job. The guard rechecks intent and job state before writing, verifies the exclusion and release, and counts timeouts as attempts. Commands have ten-second timeouts. Corrupt recovery state fails closed; errors and exhausted retry limits exit nonzero. Timestamped logs identify each check. It never submits or cancels jobs, changes resource requests, or changes cluster/node policy. Logs report scheduler state; a released or running job is not an API health claim. Concurrent operator changes are rechecked before release; Slurm does not provide an atomic compare-and-release operation.

The cluster script is installed at /scratch/aadarwal_mit/llm/ops/glm_launch_guard.py; its private attempt history is /scratch/aadarwal_mit/llm/state/glm-launch-guard.json. The Mac mini log is ~/.config/llm/tunnels/glm-batch.guard.log. For inspection:

ssh aicr 'python3 /scratch/aadarwal_mit/llm/ops/glm_launch_guard.py'
ssh aicr 'squeue -u "$USER" -h -n llm-glm-batch -o "%i|%P|%T|%N|%M|%R"'

To disable the guard without stopping a healthy model, on the Mac mini run launchctl bootout "gui/$(id -u)/llm.guard.glm-batch". llm down glm-batch removes the keep flag and stops serving; the guard respects that choice. Inspect repeated launch failures before resetting history. Batch scheduling and weight-loading gaps remain possible; this guard does not reserve GPUs or provide uninterrupted serving.

Use it

Chat. Pick GLM-5.3-Flash in the model picker. The preset carries marimo_tools, gdsfactory_photonic_tools, piel_photonic_tools, vvuq_tools; add more per chat with the integrations icon → Tools (GitHub, VVUQ MCP, …). Thinking arrives as reasoning_content and renders as a reasoning block.

Channel. In the glm channel (or any channel), @GLM-5.3-Flash <ask>. There is no tool picker in channels: the reply uses the preset's default tools, runs as you, and posts straight into the channel (prod is set to channels.model_response_mode = channel).

API (what the freelancer does).

curl -s https://mit.nonlocally.org/api/chat/completions \
  -H "Authorization: Bearer $OPENWEBUI_API_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"glm","messages":[{"role":"user","content":"List the projects."}],
       "tools":[{"type":"function","function":{"name":"list_projects","parameters":{"type":"object","properties":{}}}}],
       "tool_choice":{"type":"function","function":{"name":"list_projects"}}}' \
  | jq '.choices[0].message.tool_calls, .model'

A caller that sends its own tools gets them passed through unchanged (OpenWebUI skips server-side tool resolution when the body has tools), so the harness's GF+ MCP functions reach GLM verbatim.

Verify (the only test that counts is FORCED tool_choice)

Straight at vLLM through the named tunnel, from any machine that has the key (measured 2026-09-02: TOOL CALL):

python3 - <<'PY'
import json, pathlib, urllib.request, sys
sys.path.insert(0, "scripts"); import probe_tool_calling as p
key = pathlib.Path.home().joinpath(".config/llm/keys/glm").read_text().strip()
body = p.forced_tool_request("glm", p.DEFAULT_TOOL); body["stream"] = False
r = urllib.request.Request("https://glm.kratuvak.com/v1/chat/completions", data=json.dumps(body).encode(),
    headers={"Authorization": f"Bearer {key}", "Content-Type": "application/json"})
print(p.classify(json.load(urllib.request.urlopen(r, timeout=180))))
PY

Through the platform, in-pod (the record tests/data/tool_calling_probe.json comes from this):

P=$(kubectl get pods -n openweb-marimo --no-headers | awk '/^litellm-proxy/ && /Running/ {print $1; exit}')
kubectl exec -i -n openweb-marimo "$P" -- env PROBE_MODELS='["glm"]' python - --run -v < scripts/probe_tool_calling.py

Pass = "verdict": "TOOL CALL" and "served_model": "glm" (a LiteLLM fallback can answer for a dead model; assert on the served model, not just on tool_calls).

Gotchas

  • glm matches none of tool_policy.py's name prefixes. Only model_info.supports_function_calling: true keeps its tools; do not remove it.
  • Deploy order is load-bearing, and the failure signature is misleading. The GCP-SM secret version must exist before k8s/external-secrets/openweb-marimo-glm.yaml is applied and LiteLLM is rolled. Reversed, optional: true lets LiteLLM start with GLM_API_KEY unset, and the symptom is glm answering 401 while every other alias is healthy — it reads like a bad route or a dead tunnel, not a missing secret. Check kubectl get secret openweb-marimo-glm -n openweb-marimo -o jsonpath='{.data.GLM_API_KEY}' | wc -c (must be > 0), then scripts/deploy_litellm.sh again.
  • Two connections advertising glm is harmless (first wins, LiteLLM is #0) but confusing; the bridge's copy is dropped once this route is live.
  • Cloudflare's "Block AI Bots" managed rule must not cover the API hostnames on kratuvak.com. Scoped "Block on all pages", the edge answered 403 — before vLLM's auth — to the user agents OpenAI/Python … and AsyncOpenAI/Python … (measured 2026-09-02, both zone IPs, bridge host too), and LiteLLM's openai/ provider IS the OpenAI Python SDK. Fixed by turning the rule off (Security → Bots; "block only on hostnames with ads" would also spare API hosts). Bot Fight Mode was never on. The route also carries a User-Agent override so it survives the rule being switched back on. Both origins carry their own bearer auth, so the zone loses nothing. Check any time, no key needed:
curl -s -o /dev/null -w '%{http_code}\n' -A 'OpenAI/Python 1.0.0' https://glm.kratuvak.com/v1/models   # 401 = OK, 403 = the AI-bots rule is back on
  • A resolver that was asked for glm.kratuvak.com before the record existed may cache NXDOMAIN for the negative TTL (the MIT campus resolver did on 2026-09-02). GKE's resolvers were never asked early; from a laptop, dig @1.1.1.1 shows the truth.