serving-mode: headless / multi-user toggle to free unified memory for vLLM
Flip a DGX Spark (or any Ubuntu box) between its desktop state and a headless inference-serving state — multi-user.target with the desktop + maintenance services pared back — handing the unified 128 GB's desktop overhead (~10-15 GB) back to the GPU KV cache. Companion to Entrpi/qwen3.5-122B-A10B-on-spark. - serving-mode: on / serve / off / status (persists across reboots) - serving-mode.conf.example: optional site config (GPU sidecars / model unit) - examples/vllm-model.service: run the vLLM container under systemd
This commit is contained in:
Entrpi
@@ -0,0 +1,104 @@
|
||||
# dgx-spark-serving-mode
|
||||
|
||||
A small `serving-mode` script to flip a **DGX Spark** (or any Ubuntu box) between
|
||||
its full desktop state and a **headless inference-serving** state — handing the
|
||||
GPU as much memory as possible.
|
||||
|
||||
## Why this matters
|
||||
|
||||
GB10-based systems — the NVIDIA DGX Spark, ASUS Ascent GX10, and similar — are
|
||||
uniquely suited to *serving* large MoE LLMs: 128 GB of unified memory per node,
|
||||
and with ConnectX-7 you can cluster them (up to TP=8) to put even ~1T-parameter
|
||||
models within reach. Once that's the box's job, serving is effectively its
|
||||
primary purpose — and **every MB you free is more room for KV cache: more
|
||||
concurrency and better prefix-cache reuse.**
|
||||
|
||||
The catch is that the 128 GB is *unified*, shared between the OS/desktop and the
|
||||
GPU. Whatever the GNOME desktop, the display manager, snap, and assorted
|
||||
maintenance daemons/timers hold (commonly **~10–15 GB**) is memory the vLLM KV
|
||||
cache *doesn't* get. Dropping to `multi-user.target` (no GUI) and paring back
|
||||
those services hands it back to the model — directly raising the usable KV pool,
|
||||
context length, and concurrency.
|
||||
|
||||
If you're seeing a smaller-than-expected KV pool (e.g. `Maximum concurrency for
|
||||
N tokens` below ~1×, or `Available KV cache memory` a dozen GiB lower than
|
||||
someone else's on the same hardware), the desktop/extra services are the usual
|
||||
cause.
|
||||
|
||||
This is the companion to the serving recipe at
|
||||
**[Entrpi/qwen3.5-122B-A10B-on-spark](https://github.com/Entrpi/qwen3.5-122B-A10B-on-spark)**.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
mkdir -p ~/bin
|
||||
curl -fsSL https://raw.githubusercontent.com/Entrpi/dgx-spark-serving-mode/main/serving-mode -o ~/bin/serving-mode
|
||||
chmod +x ~/bin/serving-mode
|
||||
# optional: manage your own GPU sidecars / a model unit
|
||||
cp serving-mode.conf.example ~/.config/serving-mode.conf # then edit
|
||||
```
|
||||
|
||||
## Use
|
||||
|
||||
```bash
|
||||
~/bin/serving-mode status # current target, MemAvailable, docker, units (no sudo)
|
||||
sudo ~/bin/serving-mode on # pare desktop + maintenance (incl. docker); multi-user.target
|
||||
sudo ~/bin/serving-mode serve # headless serve: pare desktop, KEEP docker, bring up model + user units
|
||||
sudo ~/bin/serving-mode off # restore the full graphical desktop and all services
|
||||
```
|
||||
|
||||
- **`on`** — maximum memory freed for *manual* serving (you run the container
|
||||
yourself). Disables docker too; add `-u` to also stop your `--user` units.
|
||||
- **`serve`** — the production state: desktop pared, **docker kept up**, the
|
||||
model unit + your user units (sidecars/agent) brought up, **linger enabled** so
|
||||
the whole stack survives logout/reboot.
|
||||
- **`off`** — back to the normal graphical desktop.
|
||||
|
||||
Both `on` and `serve` `set-default multi-user.target`, so they **persist across
|
||||
reboots**. `status` needs no sudo.
|
||||
|
||||
### What stays running (and why)
|
||||
|
||||
`serving-mode` pares back the desktop and *maintenance* layer only — the box
|
||||
stays reachable and serving. These are **never touched** in any mode (all
|
||||
verified `active` on a serving DGX Spark):
|
||||
|
||||
| Unit | Purpose | Why it stays on |
|
||||
|---|---|---|
|
||||
| `ssh.service` | SSH server | The only way into a headless box — stopping it locks you out |
|
||||
| `NetworkManager` / `systemd-networkd` | Network & link management | One of them owns the SSH link; stopping it drops the network |
|
||||
| `systemd-resolved` | DNS resolution | Name resolution for model / registry / package fetches |
|
||||
| `systemd-timesyncd` | NTP clock sync | Correct time for TLS certs, logs, and scheduled jobs |
|
||||
| `systemd-udevd` | Device manager | Enumerates the GPU and storage; the GPU may not initialize without it |
|
||||
| `systemd-journald` | System logging | Captures vLLM / container logs for debugging |
|
||||
| `systemd-logind` | Login & session manager | Sessions + `enable-linger`, so `serve`'s user units survive logout |
|
||||
| `dbus.service` | IPC message bus | `systemctl --user`, logind, and NetworkManager all talk over it |
|
||||
| `polkit.service` | Privilege authorization | `systemctl` / service actions need it to authorize |
|
||||
| `nvidia-persistenced` | NVIDIA persistence daemon | Keeps the driver/GPU initialized between CUDA clients — avoids re-init latency/instability |
|
||||
| `rasdaemon` | ECC / RAS error logging | Witnesses memory (ECC) errors under the heavy memory pressure of large-model serving |
|
||||
| `getty@tty1` | Local console login | Recovery TTY if SSH / the network ever fails |
|
||||
| `user@<uid>.service` | systemd `--user` manager | Hosts your `--user` units (sidecars/agent) and ssh-session scopes |
|
||||
| `logrotate.timer` | Log rotation | Keeps journald / logs from filling the disk |
|
||||
| `fstrim.timer` | Weekly SSD TRIM | Maintains SSD performance and longevity |
|
||||
|
||||
The desktop/maintenance units it *does* stop are listed in the `SERVICES`,
|
||||
`DOCKER_SERVICES`, and `TIMERS` arrays at the top of the script.
|
||||
|
||||
## Optional: run vLLM as a managed service
|
||||
|
||||
[`examples/vllm-model.service`](examples/vllm-model.service) is a template that
|
||||
runs the [qwen3.5-122B-A10B-on-spark](https://github.com/Entrpi/qwen3.5-122B-A10B-on-spark)
|
||||
container under systemd. Install it, set `MODEL_UNIT="vllm-model.service"` in
|
||||
`~/.config/serving-mode.conf`, and `serving-mode serve` will start it (and it
|
||||
auto-restarts + survives reboot).
|
||||
|
||||
## Notes
|
||||
|
||||
- `serving-mode off` re-enables the **full** managed list — including anything
|
||||
you'd manually disabled before. Adjust the `SERVICES`/`TIMERS` arrays at the
|
||||
top of the script to taste.
|
||||
- The script is conservative: each unit is toggled with `disable --now` /
|
||||
`enable --now` and failures are ignored, so a unit you don't have is a no-op.
|
||||
- No secrets here. If you run GPU sidecars with their own tokens, keep those in
|
||||
your `systemd --user` unit files (referenced by name in `serving-mode.conf`),
|
||||
not in this repo.
|
||||
Reference in New Issue
Block a user