Build Logs from the Lab
Real engineering stories. Kernel panics at midnight, RAID failures on Christmas, and the occasional moment where everything just works.
The RAID That Almost Ate Christmas
Four HGST drives. One dying Synology. USB docks that kept disconnecting. The week between Christmas and New Year's became a crash course in mdadm, LVM, Btrfs, and why you should never trust USB for data recovery.
The Crash Recovery & The Lab Engine Goes Live
System crash at 00:30, crash recovery, fleet package drift discovered, lab engine deployed to production, UX fixed, and multi-container support planned. 4 parallel sessions, 7 critical infrastructure issues resolved.
The Kernel That Panicked Every Three Minutes
Server rebooting every 1-3 minutes. Couldn't stay up long enough to investigate. Turned out K3s pods were crash-looping so hard they destabilized the kernel, and Ubuntu's default panic setting auto-rebooted before I could catch it.
[object Object] — The Three-Headed Hydra That Ate My Admin Panel
Nine hours debugging why Cloudflare Pages rendered [object Object] instead of HTML. Three independent sources of process. One monkey-patched Response constructor. Zero regrets.
25 Bugs, One Night
12.5 hours, 25+ bugs across 4 components, deployed to every node before sunrise. The v2.5.1 overnight marathon.
The Tunnel With Two Heads
My websites were flipping between working and broken at random. Same URL, same moment, different results. Turns out I had two cloudflared instances fighting over the same tunnel — and Cloudflare was helpfully load-balancing between them.
The Thunderbolt That Killed DNS (and SSH, and My Confidence)
Plugged in a Thunderbolt ethernet adapter for faster NAS transfers. Lost DNS. Lost SSH. Three root causes, a CorpDNS detour, and a lesson about hot-pluggable networking on Unraid.
The Reboot Loop That Blamed the Wrong Code
When Alpha-Centauri started rebooting every 90 seconds, I was convinced my build swarm code had achieved sentience and was trying to escape. Spoiler: it was innocent.
The Race Condition That Ate My Binaries
Drones were deleting their packages before the orchestrator could validate them. Also: a Docker container crash-looping because it was looking for SSH keys that don't exist in the new architecture.
Three Rsync Bugs In One Day
A 7-hour debugging session uncovered three separate rsync bugs: missing timeouts, an invalid SSH flag, and uploading 3GB instead of one package. Also built a CLI tool because I was tired of SSHing everywhere.
The AudioBooks Folder That Ate Itself Three Times
When you lose your Claude context mid-cleanup and discover your Unraid server has 3.5TB of audiobooks nested three folders deep with 3,582 empty placeholder folders for good measure.
The GRUB That Forgot Everything
Deleted a corrupted GRUB. Now /etc/grub.d/ was empty. os-prober couldn't see Windows or CachyOS. NVIDIA parameters were wrong. Found the working config in a backup file I didn't know existed.
These are the highlights. The full journal has the rest of the trail.
Browse All Journal Entries