serving-mode: headless / multi-user toggle to free unified memory for vLLM

Flip a DGX Spark (or any Ubuntu box) between its desktop state and a headless
inference-serving state — multi-user.target with the desktop + maintenance
services pared back — handing the unified 128 GB's desktop overhead (~10-15 GB)
back to the GPU KV cache. Companion to Entrpi/qwen3.5-122B-A10B-on-spark.

- serving-mode: on / serve / off / status (persists across reboots)
- serving-mode.conf.example: optional site config (GPU sidecars / model unit)
- examples/vllm-model.service: run the vLLM container under systemd

README.md

@@ -0,0 +1,104 @@
+# dgx-spark-serving-mode
+
+A small `serving-mode` script to flip a **DGX Spark** (or any Ubuntu box) between
+its full desktop state and a **headless inference-serving** state — handing the
+GPU as much memory as possible.
+
+## Why this matters
+
+GB10-based systems — the NVIDIA DGX Spark, ASUS Ascent GX10, and similar — are
+uniquely suited to *serving* large MoE LLMs: 128 GB of unified memory per node,
+and with ConnectX-7 you can cluster them (up to TP=8) to put even ~1T-parameter
+models within reach. Once that's the box's job, serving is effectively its
+primary purpose — and **every MB you free is more room for KV cache: more
+concurrency and better prefix-cache reuse.**
+
+The catch is that the 128 GB is *unified*, shared between the OS/desktop and the
+GPU. Whatever the GNOME desktop, the display manager, snap, and assorted
+maintenance daemons/timers hold (commonly **~10–15 GB**) is memory the vLLM KV
+cache *doesn't* get. Dropping to `multi-user.target` (no GUI) and paring back
+those services hands it back to the model — directly raising the usable KV pool,
+context length, and concurrency.
+
+If you're seeing a smaller-than-expected KV pool (e.g. `Maximum concurrency for
+N tokens` below ~1×, or `Available KV cache memory` a dozen GiB lower than
+someone else's on the same hardware), the desktop/extra services are the usual
+cause.
+
+This is the companion to the serving recipe at
+**[Entrpi/qwen3.5-122B-A10B-on-spark](https://github.com/Entrpi/qwen3.5-122B-A10B-on-spark)**.
+
+## Install
+
+```bash
+mkdir -p ~/bin
+curl -fsSL https://raw.githubusercontent.com/Entrpi/dgx-spark-serving-mode/main/serving-mode -o ~/bin/serving-mode
+chmod +x ~/bin/serving-mode
+# optional: manage your own GPU sidecars / a model unit
+cp serving-mode.conf.example ~/.config/serving-mode.conf   # then edit
+```
+
+## Use
+
+```bash
+~/bin/serving-mode status        # current target, MemAvailable, docker, units (no sudo)
+sudo ~/bin/serving-mode on       # pare desktop + maintenance (incl. docker); multi-user.target
+sudo ~/bin/serving-mode serve    # headless serve: pare desktop, KEEP docker, bring up model + user units
+sudo ~/bin/serving-mode off      # restore the full graphical desktop and all services
+```
+
+- **`on`** — maximum memory freed for *manual* serving (you run the container
+  yourself). Disables docker too; add `-u` to also stop your `--user` units.
+- **`serve`** — the production state: desktop pared, **docker kept up**, the
+  model unit + your user units (sidecars/agent) brought up, **linger enabled** so
+  the whole stack survives logout/reboot.
+- **`off`** — back to the normal graphical desktop.
+
+Both `on` and `serve` `set-default multi-user.target`, so they **persist across
+reboots**. `status` needs no sudo.
+
+### What stays running (and why)
+
+`serving-mode` pares back the desktop and *maintenance* layer only — the box
+stays reachable and serving. These are **never touched** in any mode (all
+verified `active` on a serving DGX Spark):
+
+| Unit | Purpose | Why it stays on |
+|---|---|---|
+| `ssh.service` | SSH server | The only way into a headless box — stopping it locks you out |
+| `NetworkManager` / `systemd-networkd` | Network & link management | One of them owns the SSH link; stopping it drops the network |
+| `systemd-resolved` | DNS resolution | Name resolution for model / registry / package fetches |
+| `systemd-timesyncd` | NTP clock sync | Correct time for TLS certs, logs, and scheduled jobs |
+| `systemd-udevd` | Device manager | Enumerates the GPU and storage; the GPU may not initialize without it |
+| `systemd-journald` | System logging | Captures vLLM / container logs for debugging |
+| `systemd-logind` | Login & session manager | Sessions + `enable-linger`, so `serve`'s user units survive logout |
+| `dbus.service` | IPC message bus | `systemctl --user`, logind, and NetworkManager all talk over it |
+| `polkit.service` | Privilege authorization | `systemctl` / service actions need it to authorize |
+| `nvidia-persistenced` | NVIDIA persistence daemon | Keeps the driver/GPU initialized between CUDA clients — avoids re-init latency/instability |
+| `rasdaemon` | ECC / RAS error logging | Witnesses memory (ECC) errors under the heavy memory pressure of large-model serving |
+| `getty@tty1` | Local console login | Recovery TTY if SSH / the network ever fails |
+| `user@<uid>.service` | systemd `--user` manager | Hosts your `--user` units (sidecars/agent) and ssh-session scopes |
+| `logrotate.timer` | Log rotation | Keeps journald / logs from filling the disk |
+| `fstrim.timer` | Weekly SSD TRIM | Maintains SSD performance and longevity |
+
+The desktop/maintenance units it *does* stop are listed in the `SERVICES`,
+`DOCKER_SERVICES`, and `TIMERS` arrays at the top of the script.
+
+## Optional: run vLLM as a managed service
+
+[`examples/vllm-model.service`](examples/vllm-model.service) is a template that
+runs the [qwen3.5-122B-A10B-on-spark](https://github.com/Entrpi/qwen3.5-122B-A10B-on-spark)
+container under systemd. Install it, set `MODEL_UNIT="vllm-model.service"` in
+`~/.config/serving-mode.conf`, and `serving-mode serve` will start it (and it
+auto-restarts + survives reboot).
+
+## Notes
+
+- `serving-mode off` re-enables the **full** managed list — including anything
+  you'd manually disabled before. Adjust the `SERVICES`/`TIMERS` arrays at the
+  top of the script to taste.
+- The script is conservative: each unit is toggled with `disable --now` /
+  `enable --now` and failures are ignored, so a unit you don't have is a no-op.
+- No secrets here. If you run GPU sidecars with their own tokens, keep those in
+  your `systemd --user` unit files (referenced by name in `serving-mode.conf`),
+  not in this repo.

examples/vllm-model.service

@@ -0,0 +1,43 @@
+# vllm-model.service — run the vLLM model container as a managed system service,
+# so `serving-mode serve` auto-starts it and it survives reboot.
+#
+# Install:
+#   sudo cp examples/vllm-model.service /etc/systemd/system/vllm-model.service
+#   # edit the paths / image / model below to match your box, then:
+#   sudo systemctl daemon-reload
+#   sudo systemctl enable --now vllm-model.service     # (or: serving-mode serve)
+#
+# This template launches the recipe from
+#   https://github.com/Entrpi/qwen3.5-122B-A10B-on-spark
+# Adjust USER, paths, image tag, MODEL, and the serve.sh args for your setup.
+# Runs the container in the FOREGROUND (no -d) so systemd supervises it; --rm so
+# a crash/restart starts clean. Prefix caching is ON by default in serve.sh
+# (set -e PREFIX_CACHE=0 to disable).
+
+[Unit]
+Description=vLLM model server (Qwen3.5-122B-A10B, DFlash) in Docker
+After=docker.service network-online.target
+Requires=docker.service
+Wants=network-online.target
+
+[Service]
+Type=simple
+# Clean up any stale container from a previous run.
+ExecStartPre=-/usr/bin/docker rm -f qwen-spark
+ExecStart=/usr/bin/docker run --rm --name qwen-spark \
+  --gpus all --net=host --ipc=host --ulimit memlock=-1:-1 \
+  -e HF_HOME=/hf -e VLLM_CACHE_ROOT=/hf/.vllm_cache \
+  -e MODEL=bleysg/Qwen3.5-122B-A10B-int4-fp8-hybrid -e INC_HYBRID=1 -e INT8_LMHEAD_V3=1 \
+  -v /home/YOUR_USER/.cache/huggingface:/hf \
+  -v /home/YOUR_USER/qwen3.5-122B-A10B-on-spark/runtime:/host:ro \
+  --entrypoint bash \
+  ghcr.io/aeon-7/aeon-vllm-ultimate:2026-06-18-v0.23.0-dflashfix \
+  /host/serve.sh 12 flash_attn
+ExecStop=/usr/bin/docker stop qwen-spark
+# Model load is ~3 min; give it room before systemd considers start failed.
+TimeoutStartSec=900
+Restart=on-failure
+RestartSec=15
+
+[Install]
+WantedBy=multi-user.target

serving-mode

@@ -0,0 +1,215 @@
+#!/usr/bin/env bash
+# serving-mode — toggle a DGX Spark (or any Ubuntu box) between its full desktop
+# state, a pared-down ssh-only state, and a headless inference-SERVING state.
+#
+# Why: the DGX Spark has 128 GB of UNIFIED memory shared between the OS/desktop
+# and the GPU. The GNOME desktop, snap/maintenance daemons, and assorted timers
+# can hold ~10-15 GB that would otherwise be available to the vLLM KV cache.
+# Dropping to multi-user.target (no GUI) and paring back desktop/maintenance
+# services hands that memory back to the model — directly raising the KV pool /
+# usable context length.
+#
+#   sudo ~/bin/serving-mode on [-u]   pared down: multi-user target, desktop /
+#                                     maintenance services and timers off
+#                                     (INCLUDING docker); -u also stops+disables
+#                                     YOUR user units (see serving-mode.conf).
+#   sudo ~/bin/serving-mode serve     headless SERVE: multi-user target, desktop /
+#                                     maintenance off, but DOCKER KEPT UP and the
+#                                     model unit + your user units brought up.
+#                                     Linger enabled so the stack survives logout/
+#                                     reboot. Re-enables NO graphical services.
+#   sudo ~/bin/serving-mode off [-u]  restore: graphical target, everything back;
+#                                     -u also re-enables+starts the user units.
+#        ~/bin/serving-mode status    show current state incl. user units (no sudo)
+#
+# 'on' and 'serve' persist across reboots (set-default multi-user.target).
+# NEVER touched in any mode: ssh, NetworkManager + systemd-networkd (one of them
+# owns the SSH link), resolved, timesyncd, udevd, journald, logind, dbus, polkit,
+# nvidia-persistenced, rasdaemon (ECC/RAS witness under memory pressure),
+# logrotate, fstrim, getty consoles, user sessions.
+#
+# Site config (optional): define your own GPU sidecars / model unit in
+#   ~/.config/serving-mode.conf   (see serving-mode.conf.example)
+# Without it, serving-mode still does the valuable part — pare the desktop and
+# free unified memory; it just won't manage any user units or a model unit.
+#
+# From: https://github.com/Entrpi/qwen3.5-122B-A10B-on-spark (serving recipe)
+set -u
+
+# ---- Desktop / maintenance services turned OFF in both 'on' and 'serve'. ----
+SERVICES=(
+  cups.service cups.socket cups.path cups-browsed.service
+  bluetooth.service
+  avahi-daemon.service avahi-daemon.socket
+  ModemManager.service
+  fwupd.service colord.service upower.service
+  snapd.service snapd.socket snapd.seeded.service
+  nvidia-dcgm.service dgx-dashboard-admin.service
+  lldpd.service smartmontools.service
+  multipathd.service multipathd.socket
+  rsyslog.service cron.service
+)
+# Docker/containerd: EXTRANEOUS in 'on' (disabled), but LOAD-BEARING in 'serve'
+# (the model is served from a container) — kept up there. Restored by 'off'.
+DOCKER_SERVICES=(docker.service docker.socket containerd.service)
+TIMERS=(
+  apt-daily.timer apt-daily-upgrade.timer
+  update-notifier-download.timer update-notifier-motd.timer motd-news.timer
+  man-db.timer ua-timer.timer fwupd-refresh.timer anacron.timer
+  sysstat-collect.timer sysstat-summary.timer dpkg-db-backup.timer
+  e2scrub_all.timer systemd-tmpfiles-clean.timer
+)
+
+# ---- Site-specific units (overridden by ~/.config/serving-mode.conf) ----
+#   USER_UNITS       — your systemd --user units managed by 'on -u' / 'off -u'.
+#   SERVE_USER_UNITS — the user stack brought up by 'serve'.
+#   MODEL_UNIT       — system unit that runs the vLLM container (see examples/).
+# Defaults are EMPTY so the script is useful with no config (desktop paring only).
+USER_UNITS=()
+SERVE_USER_UNITS=()
+MODEL_UNIT=""
+
+TARGET_USER="${SUDO_USER:-$USER}"
+TARGET_UID="$(id -u "$TARGET_USER")"
+TARGET_HOME="$(getent passwd "$TARGET_USER" | cut -d: -f6)"
+CONF="${SERVING_MODE_CONF:-$TARGET_HOME/.config/serving-mode.conf}"
+# shellcheck disable=SC1090
+[ -f "$CONF" ] && source "$CONF"
+
+mem() { awk '/MemAvailable/{printf "%.1f GB", $2/1048576}' /proc/meminfo; }
+need_root() { [ "$(id -u)" = 0 ] || { echo "run: sudo $0 ${1:-}"; exit 1; }; }
+
+# Run systemctl --user as the target user (works from root via runuser, and
+# directly when invoked unprivileged).
+uctl() {
+  if [ "$(id -u)" = 0 ]; then
+    runuser -u "$TARGET_USER" -- env "XDG_RUNTIME_DIR=/run/user/$TARGET_UID" \
+      "DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$TARGET_UID/bus" \
+      systemctl --user "$@"
+  else
+    systemctl --user "$@"
+  fi
+}
+
+USER_FLAG=0
+{ [ "${2:-}" = "-u" ] || [ "${2:-}" = "--user" ]; } && USER_FLAG=1
+
+case "${1:-}" in
+  on)
+    need_root on
+    echo "MemAvailable before: $(mem)"
+    systemctl set-default multi-user.target >/dev/null
+    for u in "${SERVICES[@]}" "${DOCKER_SERVICES[@]}" "${TIMERS[@]}"; do
+      systemctl disable --now "$u" >/dev/null 2>&1
+    done
+    systemctl mask unattended-upgrades.service >/dev/null 2>&1
+    # Wi-Fi supplicant: only stop it if no wifi link is currently active.
+    if nmcli -t -f TYPE,STATE d 2>/dev/null | grep -q '^wifi:connected'; then
+      echo "NOTE: wifi link active -> wpa_supplicant left running"
+    else
+      systemctl disable --now wpa_supplicant.service >/dev/null 2>&1
+    fi
+    if [ "$USER_FLAG" = 1 ] && [ "${#USER_UNITS[@]}" -gt 0 ]; then
+      for uu in "${USER_UNITS[@]}"; do
+        uctl disable --now "$uu" >/dev/null 2>&1
+      done
+      echo "user units stopped+disabled: ${USER_UNITS[*]}"
+    fi
+    systemctl isolate multi-user.target
+    sleep 2
+    echo "MemAvailable after:  $(mem)"
+    echo "serving mode ON (persists across reboots; restore: sudo $0 off${USER_FLAG:+ -u})"
+    ;;
+  serve)
+    need_root serve
+    echo "MemAvailable before: $(mem)"
+    systemctl set-default multi-user.target >/dev/null
+    # Pare down desktop/maintenance, but DO NOT touch docker here.
+    for u in "${SERVICES[@]}" "${TIMERS[@]}"; do
+      systemctl disable --now "$u" >/dev/null 2>&1
+    done
+    systemctl mask unattended-upgrades.service >/dev/null 2>&1
+    if nmcli -t -f TYPE,STATE d 2>/dev/null | grep -q '^wifi:connected'; then
+      echo "NOTE: wifi link active -> wpa_supplicant left running"
+    else
+      systemctl disable --now wpa_supplicant.service >/dev/null 2>&1
+    fi
+    # Docker is load-bearing for serving: ensure it is enabled and running.
+    for u in "${DOCKER_SERVICES[@]}"; do
+      systemctl enable --now "$u" >/dev/null 2>&1
+    done
+    echo "docker kept up: $(systemctl is-active docker.service 2>/dev/null)"
+    # Let user units run headless without an active login session.
+    loginctl enable-linger "$TARGET_USER" >/dev/null 2>&1
+    # Bring up the user stack (agent + GPU sidecars), if any configured.
+    if [ "${#SERVE_USER_UNITS[@]}" -gt 0 ]; then
+      for uu in "${SERVE_USER_UNITS[@]}"; do
+        uctl enable --now "$uu" >/dev/null 2>&1
+      done
+      echo "user units up: ${SERVE_USER_UNITS[*]}"
+    fi
+    # Bring up the model server if its unit is installed.
+    if [ -n "$MODEL_UNIT" ] && systemctl cat "$MODEL_UNIT" >/dev/null 2>&1; then
+      systemctl enable --now "$MODEL_UNIT" >/dev/null 2>&1
+      echo "model unit started: $MODEL_UNIT ($(systemctl is-active "$MODEL_UNIT" 2>/dev/null))"
+    else
+      echo "NOTE: model unit '${MODEL_UNIT:-<none>}' not installed — model not auto-started"
+      echo "      (install examples/vllm-model.service and set MODEL_UNIT in the conf)"
+    fi
+    systemctl isolate multi-user.target
+    sleep 2
+    echo "MemAvailable after:  $(mem)"
+    echo "SERVE mode ON (docker up; model+user units up; persists across reboots; restore: sudo $0 off)"
+    ;;
+  off)
+    need_root off
+    systemctl unmask unattended-upgrades.service >/dev/null 2>&1
+    for u in "${SERVICES[@]}" "${DOCKER_SERVICES[@]}"; do
+      systemctl enable --now "$u" >/dev/null 2>&1
+    done
+    for t in "${TIMERS[@]}"; do
+      systemctl enable "$t" >/dev/null 2>&1
+    done
+    systemctl enable --now wpa_supplicant.service >/dev/null 2>&1
+    if [ "$USER_FLAG" = 1 ] && [ "${#USER_UNITS[@]}" -gt 0 ]; then
+      for uu in "${USER_UNITS[@]}"; do
+        uctl enable --now "$uu" >/dev/null 2>&1
+      done
+      echo "user units re-enabled+started: ${USER_UNITS[*]}"
+    fi
+    systemctl set-default graphical.target >/dev/null
+    systemctl isolate graphical.target
+    echo "default state restored (graphical target, services + timers re-enabled)"
+    echo "NOTE: 'off' enables the FULL managed list, including anything you had"
+    echo "      manually disabled before serving-mode ever ran."
+    ;;
+  status)
+    echo "default target : $(systemctl get-default)"
+    echo "MemAvailable   : $(mem)"
+    echo "docker         : $(systemctl is-active docker.service 2>/dev/null)"
+    echo "linger ($TARGET_USER) : $(loginctl show-user "$TARGET_USER" -p Linger --value 2>/dev/null)"
+    run=0; stop=0
+    for u in "${SERVICES[@]}"; do
+      if systemctl is-active --quiet "$u" 2>/dev/null; then run=$((run+1)); else stop=$((stop+1)); fi
+    done
+    echo "managed services: $run running / $stop stopped"
+    if systemctl is-active --quiet gdm3 2>/dev/null || systemctl is-active --quiet gdm 2>/dev/null; then
+      echo "display manager : running"
+    else
+      echo "display manager : stopped"
+    fi
+    if [ -n "$MODEL_UNIT" ] && systemctl cat "$MODEL_UNIT" >/dev/null 2>&1; then
+      echo "model unit      : $MODEL_UNIT = $(systemctl is-active "$MODEL_UNIT" 2>/dev/null)"
+    else
+      echo "model unit      : ${MODEL_UNIT:-<none>} = not-installed"
+    fi
+    for uu in "${SERVE_USER_UNITS[@]}"; do
+      st=$(uctl is-active "$uu" 2>/dev/null); [ -n "$st" ] || st=unknown
+      echo "user unit       : $uu = $st"
+    done
+    ;;
+  *)
+    echo "usage: sudo $0 on|serve|off   (or: $0 status)"
+    exit 1
+    ;;
+esac

serving-mode.conf.example

@@ -0,0 +1,32 @@
+# serving-mode.conf — optional site config for the `serving-mode` script.
+# Copy to ~/.config/serving-mode.conf and edit. All three vars are optional;
+# without this file, serving-mode still pares the desktop and frees unified
+# memory — it just won't touch any user units or a model unit.
+#
+# These are bash array / scalar assignments sourced by serving-mode.
+
+# systemd --user units you want stopped+disabled by `serving-mode on -u`
+# and restored by `serving-mode off -u`. (Your own GPU sidecars / agents.)
+USER_UNITS=(
+  # my-gpu-sidecar.service
+  # my-agent.service
+)
+
+# The user stack `serving-mode serve` brings up headless (linger-enabled).
+# Usually the same as USER_UNITS, optionally plus extras.
+SERVE_USER_UNITS=(
+  # my-gpu-sidecar.service
+  # my-agent.service
+)
+
+# System unit that runs the vLLM model container (see examples/vllm-model.service).
+# `serving-mode serve` starts it; leave empty to manage the model yourself.
+MODEL_UNIT="vllm-model.service"
+
+# ---------------------------------------------------------------------------
+# Example: the setup this script came from (Hermes agent on a DGX Spark) ran
+#   USER_UNITS=(hermes-gateway.service stt-sidecar.service diarize-sidecar.service)
+#   SERVE_USER_UNITS=(hermes-gateway.service stt-sidecar.service diarize-sidecar.service pantalaimon.service)
+# i.e. a couple of GPU sidecars (STT / diarization) + the agent + a Matrix proxy,
+# all as `systemd --user` units. `serve` brings the whole stack up headless and
+# linger-enabled so it survives reboot. Put YOUR units above.