From 477bc987bde23352c6773dd774b03a0369544ebc Mon Sep 17 00:00:00 2001 From: Entrpi Date: Mon, 29 Jun 2026 11:27:41 +1000 Subject: [PATCH] serving-mode: headless / multi-user toggle to free unified memory for vLLM MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Flip a DGX Spark (or any Ubuntu box) between its desktop state and a headless inference-serving state — multi-user.target with the desktop + maintenance services pared back — handing the unified 128 GB's desktop overhead (~10-15 GB) back to the GPU KV cache. Companion to Entrpi/qwen3.5-122B-A10B-on-spark. - serving-mode: on / serve / off / status (persists across reboots) - serving-mode.conf.example: optional site config (GPU sidecars / model unit) - examples/vllm-model.service: run the vLLM container under systemd --- README.md | 104 +++++++++++++++++ examples/vllm-model.service | 43 ++++++++ serving-mode | 215 ++++++++++++++++++++++++++++++++++++ serving-mode.conf.example | 32 ++++++ 4 files changed, 394 insertions(+) create mode 100644 README.md create mode 100644 examples/vllm-model.service create mode 100755 serving-mode create mode 100644 serving-mode.conf.example diff --git a/README.md b/README.md new file mode 100644 index 0000000..affa541 --- /dev/null +++ b/README.md @@ -0,0 +1,104 @@ +# dgx-spark-serving-mode + +A small `serving-mode` script to flip a **DGX Spark** (or any Ubuntu box) between +its full desktop state and a **headless inference-serving** state — handing the +GPU as much memory as possible. + +## Why this matters + +GB10-based systems — the NVIDIA DGX Spark, ASUS Ascent GX10, and similar — are +uniquely suited to *serving* large MoE LLMs: 128 GB of unified memory per node, +and with ConnectX-7 you can cluster them (up to TP=8) to put even ~1T-parameter +models within reach. Once that's the box's job, serving is effectively its +primary purpose — and **every MB you free is more room for KV cache: more +concurrency and better prefix-cache reuse.** + +The catch is that the 128 GB is *unified*, shared between the OS/desktop and the +GPU. Whatever the GNOME desktop, the display manager, snap, and assorted +maintenance daemons/timers hold (commonly **~10–15 GB**) is memory the vLLM KV +cache *doesn't* get. Dropping to `multi-user.target` (no GUI) and paring back +those services hands it back to the model — directly raising the usable KV pool, +context length, and concurrency. + +If you're seeing a smaller-than-expected KV pool (e.g. `Maximum concurrency for +N tokens` below ~1×, or `Available KV cache memory` a dozen GiB lower than +someone else's on the same hardware), the desktop/extra services are the usual +cause. + +This is the companion to the serving recipe at +**[Entrpi/qwen3.5-122B-A10B-on-spark](https://github.com/Entrpi/qwen3.5-122B-A10B-on-spark)**. + +## Install + +```bash +mkdir -p ~/bin +curl -fsSL https://raw.githubusercontent.com/Entrpi/dgx-spark-serving-mode/main/serving-mode -o ~/bin/serving-mode +chmod +x ~/bin/serving-mode +# optional: manage your own GPU sidecars / a model unit +cp serving-mode.conf.example ~/.config/serving-mode.conf # then edit +``` + +## Use + +```bash +~/bin/serving-mode status # current target, MemAvailable, docker, units (no sudo) +sudo ~/bin/serving-mode on # pare desktop + maintenance (incl. docker); multi-user.target +sudo ~/bin/serving-mode serve # headless serve: pare desktop, KEEP docker, bring up model + user units +sudo ~/bin/serving-mode off # restore the full graphical desktop and all services +``` + +- **`on`** — maximum memory freed for *manual* serving (you run the container + yourself). Disables docker too; add `-u` to also stop your `--user` units. +- **`serve`** — the production state: desktop pared, **docker kept up**, the + model unit + your user units (sidecars/agent) brought up, **linger enabled** so + the whole stack survives logout/reboot. +- **`off`** — back to the normal graphical desktop. + +Both `on` and `serve` `set-default multi-user.target`, so they **persist across +reboots**. `status` needs no sudo. + +### What stays running (and why) + +`serving-mode` pares back the desktop and *maintenance* layer only — the box +stays reachable and serving. These are **never touched** in any mode (all +verified `active` on a serving DGX Spark): + +| Unit | Purpose | Why it stays on | +|---|---|---| +| `ssh.service` | SSH server | The only way into a headless box — stopping it locks you out | +| `NetworkManager` / `systemd-networkd` | Network & link management | One of them owns the SSH link; stopping it drops the network | +| `systemd-resolved` | DNS resolution | Name resolution for model / registry / package fetches | +| `systemd-timesyncd` | NTP clock sync | Correct time for TLS certs, logs, and scheduled jobs | +| `systemd-udevd` | Device manager | Enumerates the GPU and storage; the GPU may not initialize without it | +| `systemd-journald` | System logging | Captures vLLM / container logs for debugging | +| `systemd-logind` | Login & session manager | Sessions + `enable-linger`, so `serve`'s user units survive logout | +| `dbus.service` | IPC message bus | `systemctl --user`, logind, and NetworkManager all talk over it | +| `polkit.service` | Privilege authorization | `systemctl` / service actions need it to authorize | +| `nvidia-persistenced` | NVIDIA persistence daemon | Keeps the driver/GPU initialized between CUDA clients — avoids re-init latency/instability | +| `rasdaemon` | ECC / RAS error logging | Witnesses memory (ECC) errors under the heavy memory pressure of large-model serving | +| `getty@tty1` | Local console login | Recovery TTY if SSH / the network ever fails | +| `user@.service` | systemd `--user` manager | Hosts your `--user` units (sidecars/agent) and ssh-session scopes | +| `logrotate.timer` | Log rotation | Keeps journald / logs from filling the disk | +| `fstrim.timer` | Weekly SSD TRIM | Maintains SSD performance and longevity | + +The desktop/maintenance units it *does* stop are listed in the `SERVICES`, +`DOCKER_SERVICES`, and `TIMERS` arrays at the top of the script. + +## Optional: run vLLM as a managed service + +[`examples/vllm-model.service`](examples/vllm-model.service) is a template that +runs the [qwen3.5-122B-A10B-on-spark](https://github.com/Entrpi/qwen3.5-122B-A10B-on-spark) +container under systemd. Install it, set `MODEL_UNIT="vllm-model.service"` in +`~/.config/serving-mode.conf`, and `serving-mode serve` will start it (and it +auto-restarts + survives reboot). + +## Notes + +- `serving-mode off` re-enables the **full** managed list — including anything + you'd manually disabled before. Adjust the `SERVICES`/`TIMERS` arrays at the + top of the script to taste. +- The script is conservative: each unit is toggled with `disable --now` / + `enable --now` and failures are ignored, so a unit you don't have is a no-op. +- No secrets here. If you run GPU sidecars with their own tokens, keep those in + your `systemd --user` unit files (referenced by name in `serving-mode.conf`), + not in this repo. diff --git a/examples/vllm-model.service b/examples/vllm-model.service new file mode 100644 index 0000000..2ffbbc1 --- /dev/null +++ b/examples/vllm-model.service @@ -0,0 +1,43 @@ +# vllm-model.service — run the vLLM model container as a managed system service, +# so `serving-mode serve` auto-starts it and it survives reboot. +# +# Install: +# sudo cp examples/vllm-model.service /etc/systemd/system/vllm-model.service +# # edit the paths / image / model below to match your box, then: +# sudo systemctl daemon-reload +# sudo systemctl enable --now vllm-model.service # (or: serving-mode serve) +# +# This template launches the recipe from +# https://github.com/Entrpi/qwen3.5-122B-A10B-on-spark +# Adjust USER, paths, image tag, MODEL, and the serve.sh args for your setup. +# Runs the container in the FOREGROUND (no -d) so systemd supervises it; --rm so +# a crash/restart starts clean. Prefix caching is ON by default in serve.sh +# (set -e PREFIX_CACHE=0 to disable). + +[Unit] +Description=vLLM model server (Qwen3.5-122B-A10B, DFlash) in Docker +After=docker.service network-online.target +Requires=docker.service +Wants=network-online.target + +[Service] +Type=simple +# Clean up any stale container from a previous run. +ExecStartPre=-/usr/bin/docker rm -f qwen-spark +ExecStart=/usr/bin/docker run --rm --name qwen-spark \ + --gpus all --net=host --ipc=host --ulimit memlock=-1:-1 \ + -e HF_HOME=/hf -e VLLM_CACHE_ROOT=/hf/.vllm_cache \ + -e MODEL=bleysg/Qwen3.5-122B-A10B-int4-fp8-hybrid -e INC_HYBRID=1 -e INT8_LMHEAD_V3=1 \ + -v /home/YOUR_USER/.cache/huggingface:/hf \ + -v /home/YOUR_USER/qwen3.5-122B-A10B-on-spark/runtime:/host:ro \ + --entrypoint bash \ + ghcr.io/aeon-7/aeon-vllm-ultimate:2026-06-18-v0.23.0-dflashfix \ + /host/serve.sh 12 flash_attn +ExecStop=/usr/bin/docker stop qwen-spark +# Model load is ~3 min; give it room before systemd considers start failed. +TimeoutStartSec=900 +Restart=on-failure +RestartSec=15 + +[Install] +WantedBy=multi-user.target diff --git a/serving-mode b/serving-mode new file mode 100755 index 0000000..8aa58ac --- /dev/null +++ b/serving-mode @@ -0,0 +1,215 @@ +#!/usr/bin/env bash +# serving-mode — toggle a DGX Spark (or any Ubuntu box) between its full desktop +# state, a pared-down ssh-only state, and a headless inference-SERVING state. +# +# Why: the DGX Spark has 128 GB of UNIFIED memory shared between the OS/desktop +# and the GPU. The GNOME desktop, snap/maintenance daemons, and assorted timers +# can hold ~10-15 GB that would otherwise be available to the vLLM KV cache. +# Dropping to multi-user.target (no GUI) and paring back desktop/maintenance +# services hands that memory back to the model — directly raising the KV pool / +# usable context length. +# +# sudo ~/bin/serving-mode on [-u] pared down: multi-user target, desktop / +# maintenance services and timers off +# (INCLUDING docker); -u also stops+disables +# YOUR user units (see serving-mode.conf). +# sudo ~/bin/serving-mode serve headless SERVE: multi-user target, desktop / +# maintenance off, but DOCKER KEPT UP and the +# model unit + your user units brought up. +# Linger enabled so the stack survives logout/ +# reboot. Re-enables NO graphical services. +# sudo ~/bin/serving-mode off [-u] restore: graphical target, everything back; +# -u also re-enables+starts the user units. +# ~/bin/serving-mode status show current state incl. user units (no sudo) +# +# 'on' and 'serve' persist across reboots (set-default multi-user.target). +# NEVER touched in any mode: ssh, NetworkManager + systemd-networkd (one of them +# owns the SSH link), resolved, timesyncd, udevd, journald, logind, dbus, polkit, +# nvidia-persistenced, rasdaemon (ECC/RAS witness under memory pressure), +# logrotate, fstrim, getty consoles, user sessions. +# +# Site config (optional): define your own GPU sidecars / model unit in +# ~/.config/serving-mode.conf (see serving-mode.conf.example) +# Without it, serving-mode still does the valuable part — pare the desktop and +# free unified memory; it just won't manage any user units or a model unit. +# +# From: https://github.com/Entrpi/qwen3.5-122B-A10B-on-spark (serving recipe) +set -u + +# ---- Desktop / maintenance services turned OFF in both 'on' and 'serve'. ---- +SERVICES=( + cups.service cups.socket cups.path cups-browsed.service + bluetooth.service + avahi-daemon.service avahi-daemon.socket + ModemManager.service + fwupd.service colord.service upower.service + snapd.service snapd.socket snapd.seeded.service + nvidia-dcgm.service dgx-dashboard-admin.service + lldpd.service smartmontools.service + multipathd.service multipathd.socket + rsyslog.service cron.service +) +# Docker/containerd: EXTRANEOUS in 'on' (disabled), but LOAD-BEARING in 'serve' +# (the model is served from a container) — kept up there. Restored by 'off'. +DOCKER_SERVICES=(docker.service docker.socket containerd.service) +TIMERS=( + apt-daily.timer apt-daily-upgrade.timer + update-notifier-download.timer update-notifier-motd.timer motd-news.timer + man-db.timer ua-timer.timer fwupd-refresh.timer anacron.timer + sysstat-collect.timer sysstat-summary.timer dpkg-db-backup.timer + e2scrub_all.timer systemd-tmpfiles-clean.timer +) + +# ---- Site-specific units (overridden by ~/.config/serving-mode.conf) ---- +# USER_UNITS — your systemd --user units managed by 'on -u' / 'off -u'. +# SERVE_USER_UNITS — the user stack brought up by 'serve'. +# MODEL_UNIT — system unit that runs the vLLM container (see examples/). +# Defaults are EMPTY so the script is useful with no config (desktop paring only). +USER_UNITS=() +SERVE_USER_UNITS=() +MODEL_UNIT="" + +TARGET_USER="${SUDO_USER:-$USER}" +TARGET_UID="$(id -u "$TARGET_USER")" +TARGET_HOME="$(getent passwd "$TARGET_USER" | cut -d: -f6)" +CONF="${SERVING_MODE_CONF:-$TARGET_HOME/.config/serving-mode.conf}" +# shellcheck disable=SC1090 +[ -f "$CONF" ] && source "$CONF" + +mem() { awk '/MemAvailable/{printf "%.1f GB", $2/1048576}' /proc/meminfo; } +need_root() { [ "$(id -u)" = 0 ] || { echo "run: sudo $0 ${1:-}"; exit 1; }; } + +# Run systemctl --user as the target user (works from root via runuser, and +# directly when invoked unprivileged). +uctl() { + if [ "$(id -u)" = 0 ]; then + runuser -u "$TARGET_USER" -- env "XDG_RUNTIME_DIR=/run/user/$TARGET_UID" \ + "DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$TARGET_UID/bus" \ + systemctl --user "$@" + else + systemctl --user "$@" + fi +} + +USER_FLAG=0 +{ [ "${2:-}" = "-u" ] || [ "${2:-}" = "--user" ]; } && USER_FLAG=1 + +case "${1:-}" in + on) + need_root on + echo "MemAvailable before: $(mem)" + systemctl set-default multi-user.target >/dev/null + for u in "${SERVICES[@]}" "${DOCKER_SERVICES[@]}" "${TIMERS[@]}"; do + systemctl disable --now "$u" >/dev/null 2>&1 + done + systemctl mask unattended-upgrades.service >/dev/null 2>&1 + # Wi-Fi supplicant: only stop it if no wifi link is currently active. + if nmcli -t -f TYPE,STATE d 2>/dev/null | grep -q '^wifi:connected'; then + echo "NOTE: wifi link active -> wpa_supplicant left running" + else + systemctl disable --now wpa_supplicant.service >/dev/null 2>&1 + fi + if [ "$USER_FLAG" = 1 ] && [ "${#USER_UNITS[@]}" -gt 0 ]; then + for uu in "${USER_UNITS[@]}"; do + uctl disable --now "$uu" >/dev/null 2>&1 + done + echo "user units stopped+disabled: ${USER_UNITS[*]}" + fi + systemctl isolate multi-user.target + sleep 2 + echo "MemAvailable after: $(mem)" + echo "serving mode ON (persists across reboots; restore: sudo $0 off${USER_FLAG:+ -u})" + ;; + serve) + need_root serve + echo "MemAvailable before: $(mem)" + systemctl set-default multi-user.target >/dev/null + # Pare down desktop/maintenance, but DO NOT touch docker here. + for u in "${SERVICES[@]}" "${TIMERS[@]}"; do + systemctl disable --now "$u" >/dev/null 2>&1 + done + systemctl mask unattended-upgrades.service >/dev/null 2>&1 + if nmcli -t -f TYPE,STATE d 2>/dev/null | grep -q '^wifi:connected'; then + echo "NOTE: wifi link active -> wpa_supplicant left running" + else + systemctl disable --now wpa_supplicant.service >/dev/null 2>&1 + fi + # Docker is load-bearing for serving: ensure it is enabled and running. + for u in "${DOCKER_SERVICES[@]}"; do + systemctl enable --now "$u" >/dev/null 2>&1 + done + echo "docker kept up: $(systemctl is-active docker.service 2>/dev/null)" + # Let user units run headless without an active login session. + loginctl enable-linger "$TARGET_USER" >/dev/null 2>&1 + # Bring up the user stack (agent + GPU sidecars), if any configured. + if [ "${#SERVE_USER_UNITS[@]}" -gt 0 ]; then + for uu in "${SERVE_USER_UNITS[@]}"; do + uctl enable --now "$uu" >/dev/null 2>&1 + done + echo "user units up: ${SERVE_USER_UNITS[*]}" + fi + # Bring up the model server if its unit is installed. + if [ -n "$MODEL_UNIT" ] && systemctl cat "$MODEL_UNIT" >/dev/null 2>&1; then + systemctl enable --now "$MODEL_UNIT" >/dev/null 2>&1 + echo "model unit started: $MODEL_UNIT ($(systemctl is-active "$MODEL_UNIT" 2>/dev/null))" + else + echo "NOTE: model unit '${MODEL_UNIT:-}' not installed — model not auto-started" + echo " (install examples/vllm-model.service and set MODEL_UNIT in the conf)" + fi + systemctl isolate multi-user.target + sleep 2 + echo "MemAvailable after: $(mem)" + echo "SERVE mode ON (docker up; model+user units up; persists across reboots; restore: sudo $0 off)" + ;; + off) + need_root off + systemctl unmask unattended-upgrades.service >/dev/null 2>&1 + for u in "${SERVICES[@]}" "${DOCKER_SERVICES[@]}"; do + systemctl enable --now "$u" >/dev/null 2>&1 + done + for t in "${TIMERS[@]}"; do + systemctl enable "$t" >/dev/null 2>&1 + done + systemctl enable --now wpa_supplicant.service >/dev/null 2>&1 + if [ "$USER_FLAG" = 1 ] && [ "${#USER_UNITS[@]}" -gt 0 ]; then + for uu in "${USER_UNITS[@]}"; do + uctl enable --now "$uu" >/dev/null 2>&1 + done + echo "user units re-enabled+started: ${USER_UNITS[*]}" + fi + systemctl set-default graphical.target >/dev/null + systemctl isolate graphical.target + echo "default state restored (graphical target, services + timers re-enabled)" + echo "NOTE: 'off' enables the FULL managed list, including anything you had" + echo " manually disabled before serving-mode ever ran." + ;; + status) + echo "default target : $(systemctl get-default)" + echo "MemAvailable : $(mem)" + echo "docker : $(systemctl is-active docker.service 2>/dev/null)" + echo "linger ($TARGET_USER) : $(loginctl show-user "$TARGET_USER" -p Linger --value 2>/dev/null)" + run=0; stop=0 + for u in "${SERVICES[@]}"; do + if systemctl is-active --quiet "$u" 2>/dev/null; then run=$((run+1)); else stop=$((stop+1)); fi + done + echo "managed services: $run running / $stop stopped" + if systemctl is-active --quiet gdm3 2>/dev/null || systemctl is-active --quiet gdm 2>/dev/null; then + echo "display manager : running" + else + echo "display manager : stopped" + fi + if [ -n "$MODEL_UNIT" ] && systemctl cat "$MODEL_UNIT" >/dev/null 2>&1; then + echo "model unit : $MODEL_UNIT = $(systemctl is-active "$MODEL_UNIT" 2>/dev/null)" + else + echo "model unit : ${MODEL_UNIT:-} = not-installed" + fi + for uu in "${SERVE_USER_UNITS[@]}"; do + st=$(uctl is-active "$uu" 2>/dev/null); [ -n "$st" ] || st=unknown + echo "user unit : $uu = $st" + done + ;; + *) + echo "usage: sudo $0 on|serve|off (or: $0 status)" + exit 1 + ;; +esac diff --git a/serving-mode.conf.example b/serving-mode.conf.example new file mode 100644 index 0000000..bbda4f6 --- /dev/null +++ b/serving-mode.conf.example @@ -0,0 +1,32 @@ +# serving-mode.conf — optional site config for the `serving-mode` script. +# Copy to ~/.config/serving-mode.conf and edit. All three vars are optional; +# without this file, serving-mode still pares the desktop and frees unified +# memory — it just won't touch any user units or a model unit. +# +# These are bash array / scalar assignments sourced by serving-mode. + +# systemd --user units you want stopped+disabled by `serving-mode on -u` +# and restored by `serving-mode off -u`. (Your own GPU sidecars / agents.) +USER_UNITS=( + # my-gpu-sidecar.service + # my-agent.service +) + +# The user stack `serving-mode serve` brings up headless (linger-enabled). +# Usually the same as USER_UNITS, optionally plus extras. +SERVE_USER_UNITS=( + # my-gpu-sidecar.service + # my-agent.service +) + +# System unit that runs the vLLM model container (see examples/vllm-model.service). +# `serving-mode serve` starts it; leave empty to manage the model yourself. +MODEL_UNIT="vllm-model.service" + +# --------------------------------------------------------------------------- +# Example: the setup this script came from (Hermes agent on a DGX Spark) ran +# USER_UNITS=(hermes-gateway.service stt-sidecar.service diarize-sidecar.service) +# SERVE_USER_UNITS=(hermes-gateway.service stt-sidecar.service diarize-sidecar.service pantalaimon.service) +# i.e. a couple of GPU sidecars (STT / diarization) + the agent + a Matrix proxy, +# all as `systemd --user` units. `serve` brings the whole stack up headless and +# linger-enabled so it survives reboot. Put YOUR units above.