# On a rented GB10 # Run models on a rented GB10 A rented DGX Spark is a whole NVIDIA GB10 for you alone, for the hours you booked. This page is how to run models on it: what is on it, serving a model with vLLM, reaching it from your computer, and the memory. Booking, paying and connecting are in How GPU rentals work. The whole machine and its own system, by the month, is bare metal. ## 1. What is on it | | A rented GB10 | | | The machine | An NVIDIA GB10: an ARM64 CPU and a Blackwell GPU. Its memory is 128 GB, shared by the GPU and the system. About 110 GB is usable for models. | | | Your environment | Ubuntu 24.04 for ARM64, in a container of its own on the machine. You are `dev`, with `sudo` in it: install anything with `apt` or `pip`. | | | Your disk | `/workspace`, 650 GB. It is deleted when the rental ends, with everything else. | | | Installed | vLLM, PyTorch with CUDA 13.0, the CUDA toolkit, llama.cpp, ComfyUI, JupyterLab, Ollama, the Hugging Face CLI (`hf`), `uv`, `git`, `git-lfs`, `tmux`, `nvtop`, `rsync` and `ffmpeg`. On 2026-11-05: vLLM 0.33.0, PyTorch 2.14.0, Ollama 0.40.14 and nvtop 3.0.2. | | | Not on it | Docker: your environment cannot start containers of its own. | | | The network | Your environment reaches the internet: `apt`, `pip` and downloads work. From outside, only your SSH reaches it. A server you start is reached through an SSH tunnel (step 4). | | Your software has to run on ARM64: `exec format error` means a program built for x86 only. The machine's software is rebuilt now and then, so `vllm --version` shows yours: ``` vllm --version python3 -c "import torch; print(torch.__version__, torch.version.cuda)" free -h # the memory: nvidia-smi shows it as Not Supported on a GB10 ``` ## 2. Your models go on /workspace The Hugging Face cache is on your disk (`HF_HOME=/workspace/hf`), so what `hf download` and `vllm serve` fetch lands on `/workspace`, with fast downloads on. To keep a model's files in a folder of your own: ``` hf download Qwen/Qwen3-8B --local-dir /workspace/models/qwen3-8b ``` ## 3. Serve a model with vLLM vLLM gives your model an OpenAI-compatible API on port 8000. Start it in `tmux`, so it keeps running when you disconnect: ``` tmux new -s vllm vllm serve Qwen/Qwen3-8B --max-model-len 32768 # leave it running with Ctrl-b, then d; back to it with: tmux attach -t vllm ``` On a GB10, `vllm serve` starts at `--gpu-memory-utilization 0.75` unless you give your own value, and says so on its first line. That share is a share of the whole machine's memory, whatever the size of the model: vLLM takes 75% for the weights and its cache, and the rest stays for the system and your other programs. The flags that decide the memory: | Flag | What it does | | | `--gpu-memory-utilization 0.75` | The share of the machine's memory vLLM takes. Higher leaves the system less: see What fits in 110 GB before you raise it. | | | `--max-model-len 32768` | The longest context of one request. Without it, vLLM plans for the model's maximum. | | | `--max-num-seqs 8` | How many requests run at once. | | | `--kv-cache-dtype fp8` | A cache in FP8: twice the tokens in the same memory, where the model's attention supports it. | | | `--served-model-name my-model` | The name your client sends as `model`. | | The smaller the build, the more room for the cache: 4-bit NVFP4 and FP8 builds take a half to a quarter of BF16, and the GB10's Blackwell GPU computes both in hardware. vLLM's startup log says how much the weights took and how many tokens of cache fit. ## 4. Reach it from your computer The server runs on the machine, not on the internet. Forward its port in a second terminal on your computer, with your machine's line from the console: ``` # port 8000 on your computer is port 8000 on the machine while this runs ssh -o HostKeyAlias=axforge-BOOKING -o StrictHostKeyChecking=accept-new -L 8000:localhost:8000 dev@HOST -p PORT ``` Then, on your computer: ``` curl http://localhost:8000/v1/models curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \ -d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Say hello"}]}' ``` Any OpenAI-compatible tool or SDK works with the base URL `http://localhost:8000/v1`. vLLM asks for no key unless you start it with `--api-key`. ## 5. The other tools | Tool | Start it, and its port | | | llama.cpp | For GGUF builds: `llama-server -m /workspace/models/model.gguf -c 32768 -ngl 99`, port 8080 | | | ComfyUI | For pictures and video: `start-comfyui`, port 8188, its models in `/workspace/models/` | | | Ollama | `start-ollama`, then `ollama run `, port 11434, its models in `/workspace/ollama` | | | JupyterLab | `start-jupyter`, port 8888, token `axforge` | | | PyTorch | `python3`, with `import torch` | | Reach each one through the tunnel of step 4, with its own port. `llama-server` takes the whole `-c` context's cache when it starts, so the memory rules below hold for it too. Every image, video and music model's page in the catalogue has its ComfyUI steps. ## 6. The memory: stay under about 110 GB The GPU and the system share this machine's 128 GB of memory, so about 110 GB is usable for your models. Loading more than that can make the machine unstable or hang it. If the memory runs out, the system stops the program that is overloading it so the machine recovers. That program is usually your server, gone in the middle of its requests. - Keep vLLM at 0.75 until you know your model's needs, and set `--max-model-len` and `--max-num-seqs` to what you use. - Test with your heaviest load (the longest prompts, every user at once) while `watch -n 1 free -h` runs in a second terminal. - What fits, and what we have seen happen when a GB10 is filled to the brim: What fits in a GB10's 110 GB. ## 7. When something does not work | What you see | What it means | What to do | | | `exec format error` | A program built for x86 only | Install its ARM64 (aarch64) build | | | `docker: command not found` | Your environment cannot run containers | Install what you need with `pip` or `apt`. For Docker, a bare metal GB10 has it | | | vLLM stops while starting: the cache does not fit | The context or the requests at once do not fit its share | Lower `--max-model-len` or `--max-num-seqs`, use an FP8 cache, or a smaller build | | | Your server ends with `Killed` | The memory ran out and the system stopped it | Stay under about 110 GB (step 6) | | | `Address already in use` on `ssh -L` | That port is taken on your computer | Pick another one on your side, for example `8001:localhost:8000` | | ← How GPU rentals work Run models on a bare metal GB10 → Source: https://dev.axforge.ai/docs/gb10-rental/