AI Foundry lab, 2 October 2026: what changed

← AI Foundry lab dashboard › Worklog

9:45 to 16:15 PDT · the lab's maintenance session on aifoundry2 · every change below is a commit in the repository

The lab dashboard now shows every machine and card live, once a second, and keeps hour, day and week history. aifoundry2's card was watched through a whole idle thermal runaway to 138 °C, and the curve explained; its card is out of service until its cooling is fixed. aifoundry1's card 0 had a broken fan; it was replaced, tested, and the card is back in service. New users now start in three steps: create their own account as root once, start Claude on the lab machine as themselves, give it one line.

1. The live dashboard

• A watcher on each machine (a user service that comes back by itself after a reboot) streams one reading a second: CPU per thread, memory, disk, the host's temperatures, the busiest processes, and for each card its PCIe link, its lock, who holds it, and its die temperature. Tested: killed, it restarts within 2 s; it came back by itself after aifoundry2's two reboots.
• Card temperature once a second. A 4 ms read through the card's management node, made only while nobody holds any card on that machine. A program that opens the node in that same 4 ms, about 1 start in 100, gets "busy" and retries.
• Card state in real time. FREE, IN USE (login and program) and LINK DOWN now come from that stream while it is under 10 seconds old, marked "live". Before, they came from a check every 10 minutes that republishes at most every 10 minutes, which is why aifoundry3's card still showed as held at 15:35 after its user had finished.
• Card use, live. A "Live, last 15 minutes" lane per card shows who holds it, second by second, so a run appears the moment it starts; the 24-hour chart now runs to the current moment.
• "Unseen" usage. Every unseen open of the day was a management-node read under 10 ms by the lab's own tooling (the temperature monitor, the dashboard's sampler, tests); none ran work on a card. An open tracer (bpftrace) that names every open, however short, is written and staged for the lab admin to install; the dashboard lists its opens by person and program once it runs.
• Root. New people get the shared root login to create their own account. A card held by root is now told apart: "CI runner" when it runs under the CI runner, else "root (shared login)", which raises a warning, as does a terminal logged in as root; a notice for root logins is staged.
• History. Each machine also records a reading every 5 s on its own disk, so a reboot loses only the minutes it is down; the history page (hour, day, week) is rebuilt every 5 minutes and opens from a machine's temperature panel or a 48-hour chart.
• The page itself. A "Start here" row (new users, known problems, the chip diagram, the reports); a dark live console; much less explanatory text; no theme selector. Two fixes: the monitor's own 4 ms reads had shown up as an "unseen" person holding every card all day, and they had made every idle card look "used".

2. aifoundry2's card: an idle thermal runaway

A plain reboot left the card off the PCIe bus; the full reset (PCI, cold), which cuts the slot's power, brought it back. Then, with the machine idle and nobody using the card, it heated until it failed again:

The shape is a thermal runaway. The idle card's power grows with its temperature (13.6 W plus a leakage part that doubles every 23 °C), and its cooling only just fails to keep up: it sheds heat into air at about 51 °C, where aifoundry3's identical card, on the same board and slot, sees about 31 °C. Heating and cooling come within 0.1 W of each other near 97 °C, which is the slow crawl in the middle; past it nothing on the card stops the climb. Air 3 °C cooler would hold it near 81 °C, so the fix is the air reaching this card, on site. The card is out of service until then. Data and the fit are in docs/reports/data/2026-10-02-idle-runaway-aifoundry2/; the Requests for Roman page has the visit steps (the card's fan cable, a watched test, a desk-fan test, a card swap).

Update, 6 October 2026: fixed in the BIOS (every fan at full speed); the idle card now holds at about 64 °C — what was done on 6 October.

Update, 7 October 2026: the fix held for about a day. Idle, the card ran away again and fell off the PCIe bus at 16:46 on 7 October; it is out of service — what happened on 7 October.

Update, 9 October 2026: back in service. A two-fan 92 mm bracket was fitted at the card; it idles at about 51 °C at 23 W, and the heavy run settles at 75 °C — what was done on 9 October.

3. aifoundry1's card 0: fan replaced, back in service

The owner put it back in service. The new-user brief, the repository's guides and the dashboard were updated; an independent review of the repository found 31 places that still called card 0 unusable or aifoundry2 the main card, all changed (dated history kept).

4. New users: three steps

1. Your computer One ssh as the lab's shared root creates the person's own account (no password, no sudo) and logs out. Root is used for nothing else.
2. Your computer One ssh as the new account installs Claude Code, fetches the lab's brief, starts Claude in a tmux session on the lab machine, and attaches the terminal to it.
3. Claude, on the lab machine One line: "Read ~/lab-start.md and follow it." Claude picks a free card on that machine, runs the first-hour tests, and asks before it touches a card.

The brief went through nine editions today: aifoundry2's card out, card 0 back in, a check that looks only at your own card (two people can share aifoundry1, one per card), several people starting at once, and the root-first start. Two independent reviews (61 and 8 confirmed findings) were applied; the commands were tested against stand-ins and, for the user step, live on aifoundry3.

5. Open items

• Needs root: staged in the maintainer's home directory on each machine, one or two commands each: the updated onboarding script et-lab-start, the two machines' new login banners, the open tracer et-opens (check its probe first, for 5 s), and the root login notice.
• aifoundry2's card: the cooling fix on site (Requests for Roman, SH5). Done 6 October 2026: every fan set to full speed in the BIOS; the idle card holds at about 64 °C. Out of service again since 7 October: the cooling at the card got worse and the idle card ran away. Back in service since 9 October: a two-fan 92 mm bracket fitted at the card; it idles at about 51 °C at 23 W, and the heavy run settles at 75 °C.
• The monitor's once-a-second card check runs the lab's et-who, which uses a fixed sudo rule, so each machine logs a sudo line every second; a lighter check would be quieter.
• Signed-out viewers: in a signed-out test browser the dashboard's live streams did not appear; to confirm with a real signed-out visit.
• spacesheep.dev: a published link was slow to show the new version; reported to the spacesheep team, with timings (measured on this page).

Commits

Smaller commits in between: the dashboard's quick links, history links, theme and Reports changes, and the fifth to eighth editions of the brief. The full list is the repository's history for 2 October.