Run models on a rented GB10

A rented DGX Spark is a whole NVIDIA GB10 for you alone, for the hours you booked. This page is how to run models on it: what is on it, serving a model with vLLM, reaching it from your computer, and the memory. Booking, paying and connecting are in How GPU rentals work. The whole machine and its own system, by the month, is bare metal.

1. What is on it

Your software has to run on ARM64: exec format error means a program built for x86 only. The machine's software is rebuilt now and then, so vllm --version shows yours:

vllm --version
python3 -c "import torch; print(torch.__version__, torch.version.cuda)"
free -h      # the memory: nvidia-smi shows it as Not Supported on a GB10

2. Your models go on /workspace

The Hugging Face cache is on your disk (HF_HOME=/workspace/hf), so what hf download and vllm serve fetch lands on /workspace, with fast downloads on. To keep a model's files in a folder of your own:

hf download Qwen/Qwen3-8B --local-dir /workspace/models/qwen3-8b

3. Serve a model with vLLM

vLLM gives your model an OpenAI-compatible API on port 8000. Start it in tmux, so it keeps running when you disconnect:

tmux new -s vllm
vllm serve Qwen/Qwen3-8B --max-model-len 32768
# leave it running with Ctrl-b, then d; back to it with: tmux attach -t vllm

On a GB10, vllm serve starts at --gpu-memory-utilization 0.75 unless you give your own value, and says so on its first line. That share is a share of the whole machine's memory, whatever the size of the model: vLLM takes 75% for the weights and its cache, and the rest stays for the system and your other programs. The flags that decide the memory:

The smaller the build, the more room for the cache: 4-bit NVFP4 and FP8 builds take a half to a quarter of BF16, and the GB10's Blackwell GPU computes both in hardware. vLLM's startup log says how much the weights took and how many tokens of cache fit.

4. Reach it from your computer

The server runs on the machine, not on the internet. Forward its port in a second terminal on your computer, with your machine's line from the console:

# port 8000 on your computer is port 8000 on the machine while this runs
ssh -o HostKeyAlias=axforge-BOOKING -o StrictHostKeyChecking=accept-new -L 8000:localhost:8000 dev@HOST -p PORT

Then, on your computer:

curl http://localhost:8000/v1/models
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Say hello"}]}'

Any OpenAI-compatible tool or SDK works with the base URL http://localhost:8000/v1. vLLM asks for no key unless you start it with --api-key.

5. The other tools

Reach each one through the tunnel of step 4, with its own port. llama-server takes the whole -c context's cache when it starts, so the memory rules below hold for it too. Every image, video and music model's page in the catalogue has its ComfyUI steps.

6. The memory: stay under about 110 GB

The GPU and the system share this machine's 128 GB of memory, so about 110 GB is usable for your models. Loading more than that can make the machine unstable or hang it. If the memory runs out, the system stops the program that is overloading it so the machine recovers. That program is usually your server, gone in the middle of its requests.

• Keep vLLM at 0.75 until you know your model's needs, and set --max-model-len and --max-num-seqs to what you use.
• Test with your heaviest load (the longest prompts, every user at once) while watch -n 1 free -h runs in a second terminal.
• What fits, and what we have seen happen when a GB10 is filled to the brim: What fits in a GB10's 110 GB.

7. When something does not work