# Local Dev Server (Mac Mini, `guillaumes-mac-mini`) ## Current setup kView INVESTOR's local dev server no longer runs on `php artisan serve` (PHP's built-in development server). It is served by: - **Caddy** on `127.0.0.1:8000` — serves `public/build/*` and every other static file directly, and reverse-proxies dynamic requests to PHP-FPM. Config: `/opt/homebrew/etc/Caddyfile`. - **PHP-FPM** on `127.0.0.1:9000` — a dedicated `kview_investor` pool (`pm = dynamic`, up to 10 concurrent workers), replacing the single blocking `php artisan serve` process. Config: `/opt/homebrew/etc/php/8.4/php-fpm.d/kview_investor.conf`. The Homebrew default `www` pool is disabled (`php-fpm.d/www.conf.disabled`) to avoid a second pool trying to run as the unprivileged `_www` user, which cannot read this project. - **`tailscale serve`** forwards raw TCP from the tailnet (`guillaumes-mac-mini.tail668295.ts.net:8000` and `100.80.95.73:8000`) to `127.0.0.1:8000` — unchanged by this fix. Check with `tailscale serve status`. Its `(TLS over TCP, tailnet only)` label is misleading: it does **not** terminate application-layer TLS. It means the raw bytes are forwarded as-is, already protected by Tailscale's own WireGuard tunnel between peers. **Always use `http://`, never `https://`, against port 8000** — an `https://` request's TLS handshake gets forwarded straight to Caddy, which is plain HTTP and will reject it (see Incident 2 below, first diagnosed as a `tailscaled` problem before being correctly identified). Both Caddy and PHP-FPM run inside **detached `screen` sessions**, not as launchd services — see "Why not launchd" below. ``` screen -ls # kview_investor-caddy, kview_investor-php-fpm screen -r kview_investor-caddy # attach to watch Caddy's console (Ctrl-A D to detach) screen -r kview_investor-php-fpm # attach to watch PHP-FPM's console ``` ### Restarting after a reboot or a crash Both processes are self-healing while running (each session wraps its program in a `while true; do ...; sleep 1; done` loop, so a crash restarts it within a second), but a full reboot kills the `screen` sessions themselves. Restart both with: ```bash /opt/homebrew/var/kview_investor/start-dev-server.sh ``` This is idempotent — it only starts a session that isn't already running, safe to run any time (e.g. add it to your own login script if you want it automatic; a launchd LaunchAgent version of this exact idea was tried and did not reproduce the underlying bug in testing, but was not kept loaded by default — see `/opt/homebrew/var/kview_investor/disabled-launchd-plists/` and the incident notes below before re-enabling it). ### Logs - `/opt/homebrew/var/log/caddy.log` — Caddy startup/runtime log - `/opt/homebrew/var/log/caddy-kview_investor-access.log` — Caddy access log - `/opt/homebrew/var/log/php-fpm.log` — PHP-FPM master log - `/opt/homebrew/var/log/php-fpm-kview_investor.log` — `kview_investor` pool error log ## Incident: intermittent blank page, `#app` never mounts **Reported 21 July 2026** on `http://guillaumes-mac-mini.tail668295.ts.net:8000/investors`: 200 HTML response with valid Inertia data, but `#app` stayed empty because one or more `/build/assets/*.js` chunk requests failed with `net::ERR_CONNECTION_TIMED_OUT` (the file existed and answered fine when requested again individually). Reloading usually fixed it. The server always sent `Connection: close`. ### Root cause 1. **Two duplicate `php artisan serve` processes** were running simultaneously — one since 19 July 09:26 (`--host=127.0.0.1`, inside a `screen` session named `kview_investor-server`), one since 20 July 19:56 (`--host=0.0.0.0`). `storage/logs/artisan-serve.log` had a logged `Failed to listen on 0.0.0.0:8000 (reason: Address already in use)`, confirming repeated, uncoordinated server starts. 2. `php artisan serve` is PHP's **built-in development server**: it processes one request at a time (no real concurrency) and never sends `Connection: keep-alive` — confirmed by direct testing and matching the reported `Connection: close`. 3. An Inertia/Vue page load fires a burst of **~48 parallel requests** for Vite's code-split chunks (confirmed against `public/build/manifest.json`). Against a single-threaded, non-keep-alive server, each of those needs its own fresh TCP handshake and waits in line behind whichever request the server happens to be blocking on. 4. Traffic arrives over **Tailscale** — `tailscale serve status` shows `guillaumes-mac-mini.tail668295.ts.net:8000` (TLS) proxying to `127.0.0.1:8000` — adding a hop of latency on top of the already-serial queue. Under that combination, an occasional request (like the reported `AppLink-CayOLmZ8.js`) queued long enough to exceed the browser's connection timeout, even though the manifest/build were perfectly consistent and the file always answered fine on retry - this was purely a concurrency/queueing failure, not a missing- or stale-asset problem. ### Fix Replaced both `php artisan serve` processes with Caddy (real concurrency + keep-alive for static assets, which are now served without touching PHP at all) in front of a PHP-FPM pool (10 concurrent workers instead of 1) — see "Current setup" above. Two Caddy-specific pitfalls hit along the way, now documented as comments in `/opt/homebrew/etc/Caddyfile`: - A bare `host:port` site address makes Caddy try to provision its own local HTTPS certificate via `sudo`, which has no TTY under a background service and crash-loops; the fix is the explicit `http://` scheme. - A Caddyfile site address's host part is only a **request-matching** filter by default; without an explicit `bind 127.0.0.1` directive, Caddy still listens on the wildcard address and collides with `tailscaled`'s own listeners on the tailnet-interface addresses (invisible to non-root `lsof`, since `tailscaled` runs as root), which read as an unexplained intermittent "address already in use". ### Why not launchd The first implementation used `brew services`/dedicated LaunchAgents for both Caddy and PHP-FPM (the "normal", most robust way to run a persistent local service on macOS). On this machine, **both hung indefinitely processing every request** when spawned by launchd - confirmed with a minimal two-line `echo` PHP script that hung before printing anything, and with a bare `file_server`-only Caddy config serving one static file with no PHP involved at all. The identical binaries and configs, run manually in a foreground shell (or inside `screen`, which provides the same kind of session), answered every request in single-digit milliseconds, every time. Ruled out while isolating this: `events.mechanism` (kqueue vs. poll), `catch_workers_output`, all PHP extensions (`-n`), explicit `EnvironmentVariables` (`PATH`/`HOME`/`TMPDIR`), stale OPcache shared-memory segments (a real but separate artifact from repeatedly `kill -9`-ing FPM during this diagnosis - clear with `ipcrm -m ` if it recurs), `OBJC_DISABLE_INITIALIZE_FORK_SAFETY`, `pm = dynamic` vs `pm = ondemand`, `LimitLoadToSessionType`, and the specific port number (8000 vs. a spare port). A plain Python `http.server` launchd service was unaffected, so this is not a blanket "launchd-spawned processes are unreachable" sandbox artifact either. The exact mechanism was not fully isolated - macOS blocked `sample`/`spindump` stack capture in the diagnosis environment used - so `screen`-based supervision (proven reliable) is the deliberate choice here, not a placeholder. If revisiting this, capture a stack trace of a hung worker with `sample ` (needs a session where that is not sandboxed) before trying launchd again. ### Validation performed (21 July 2026) - 3 rounds x 48 concurrent requests against every `public/build/assets/*.js` file through Caddy: 0 failures, slowest 75 ms (previously the exact failure mode). - 12 fresh-login page loads across `/investors`, `/funds`, `/gps` (4 each, cache bypassed via a fresh browser context each time): `#app` mounted with real content every time, 0 failed/timed-out local requests, 0 console errors. (Unrelated: `fonts.bunny.net` requests were aborted in this sandboxed test environment, which has no outbound internet access - not a regression, not a local asset.) - `tests/e2e/phase7-production-indexes.spec.ts` (`all seven indexes render the Vue engine`, includes Investors) passed against the fixed server. - `php artisan test`: 228 passed, 5,524 assertions - unaffected, as expected, since this was an infrastructure fix with no application code changes. No kView INVESTOR application code, business logic, or data was changed; every file touched by this fix lives under `/opt/homebrew/etc`, `/opt/homebrew/var`, or `~/Library/LaunchAgents` (system/Homebrew configuration, outside this repository). ## Incident 2 (22 July 2026): tailnet clients got an empty response **Reported**: after Incident 1's fix, `http://127.0.0.1:8000` worked fine on the Mac Mini itself, but tailnet clients (a Windows laptop over Tailscale, then an iPhone) could not load the app at all. On iOS, Chrome downloaded an empty file named after the route slug (`investors`) instead of rendering the page. ### Root cause — a bug introduced by Incident 1's own fix The Caddyfile from Incident 1 used this site address: ```caddyfile http://127.0.0.1:8000 { bind 127.0.0.1 ... } ``` A Caddyfile site address's host part is not just documentation of where to listen — Caddy uses it as a **Host-header matching filter** for routing. `bind 127.0.0.1` correctly restricted the *socket* to loopback (avoiding the collision with `tailscaled`'s own listeners, per Incident 1), but the site address `127.0.0.1:8000` additionally meant Caddy only answered requests whose `Host` header was literally `127.0.0.1:8000`. Real tailnet traffic arrives with `Host: guillaumes-mac-mini.tail668295.ts.net:8000` or `Host: 100.80.95.73:8000` — neither matched, so Caddy had no route for it and returned an empty `200`, which iOS Chrome rendered as a file download. This bug was masked during Incident 1's own validation because every test was run directly against `http://127.0.0.1:8000` (matching Host, so it "worked"), and my own attempt to test the tailnet path used `https://` (see the corrected note above) — which fails at the TLS layer before ever reaching Caddy's Host-matching logic, so it looked like a `tailscaled` problem rather than a Caddy routing bug. **Lesson: validate a reverse proxy against every Host header a real client will actually send, not only the one used during local testing.** ### Fix (applied by a separate Codex session with local sudo access) Changed the site address to accept any Host on port 8000, while keeping the loopback bind that actually matters for the socket: ```caddyfile http://:8000 { bind 127.0.0.1 ... } ``` `bind 127.0.0.1` is unchanged and still required — do not remove it or widen it to `0.0.0.0`, or Caddy will again collide with `tailscaled`'s own listeners (Incident 1). Do not put a host back in the site address (e.g. `http://127.0.0.1:8000`), or tailnet Host headers will be rejected again (this incident). Validated: `curl` with `Host: 127.0.0.1:8000`, `Host: guillaumes-mac-mini.tail668295.ts.net:8000` and `Host: 100.80.95.73:8000` all return `200` against `/login`. Confirmed working from the iPhone. A temporary experiment enabling real TLS termination (`--tls-terminated-tcp`) produced `TLS internal error` because Tailscale HTTPS Certificates are not yet enabled for this node in the admin console (`CertDomains` is `null`) — reverted, node is back to plain TCP forwarding. To actually move to `https://` some day: enable **HTTPS Certificates** in the Tailscale admin console first, confirm `tailscale status --json` shows a non-null `CertDomains` for this machine, only then reconfigure `tailscale serve` for TLS termination and change Caddy's Caddyfile accordingly (it would need to stop being plain `http://` at that point). Full intervention log, including the backup file paths under `/tmp` used during the fix: `dev_guide/tailscale-intervention-2026-07-22.md`. ## Incident 3 (22 July 2026): PHP-FPM master died, orphaned workers blocked the restart **Reported**: real browser session (multiple tabs against `guillaumes-mac-mini.tail668295.ts.net:8000`) showed the Home page's metric cards rendering completely blank (title bar stuck on the previous page, suggesting a stalled Inertia navigation) and the Investors index stuck on its skeleton loading state indefinitely — the authenticated shell/sidebar rendered fine (client-side, JS-only), but nothing that required a fresh request to PHP-FPM ever resolved. ### Root cause The `php-fpm` **master** process (PID 65858's spawned child) had died at some point, but its three **worker** processes (pool `kview_investor`) did not exit with it and kept holding the listening socket on `127.0.0.1:9000`. The supervisor script's `while true; do ...; sleep 1; done` loop (see "Current setup" above) correctly kept trying to start a fresh `php-fpm` master every second, but every attempt failed immediately with `unable to bind listening socket for address '127.0.0.1:9000': Address already in use` — because the orphaned workers, not a real second master, still owned the port. `php-fpm.log` showed this failing every second for at least several minutes before it was noticed. Static assets (`public/build/assets/*.js`/`.css`) kept loading throughout, because Caddy serves those directly without touching PHP-FPM at all — which is exactly why the symptom was "shell renders, data never arrives" rather than a fully blank page: the orphaned workers could still `accept()` on the inherited socket for a while (traditional prefork model, no master needed for that part), so some requests kept succeeding intermittently, until enough of them recycled or timed out that the app effectively stopped answering. This is the same underlying failure class as Incident 1 (something else holding a port the supervisor loop expects to reclaim on restart) but on port 9000 (PHP-FPM) instead of port 8000, and the self-healing loop's usual assumption — that a crashed master's children exit with it, freeing the port within the 1-second retry window — did not hold here. The exact trigger for the master's death was not isolated (no OOM/signal evidence found in the surrounding system log); this note only covers the recovery. ### Fix ```bash lsof -nP -iTCP:9000 # find the PIDs still bound to the port kill -9 # e.g. the 2-4 `php-fpm: pool kview_investor` PIDs ``` Once the orphaned workers are gone, the supervisor loop's very next 1-second retry succeeds on its own — no need to touch the `screen` session or restart Caddy. Confirmed via `php-fpm.log` (`NOTICE: fpm is running, pid ` / `NOTICE: ready to handle connections`) and a full `curl` round trip (login, `/`, `/investors`, all `200` in under 200ms). ### If this recurs Check `lsof -nP -iTCP:9000` first whenever the app looks stuck loading with a working shell but no data — a crash-looping `php-fpm.log` (`unable to bind... Address already in use`, repeating every second) is the tell. Killing the stale PIDs holding the port is sufficient; the supervisor loop handles the rest.