# On a bare metal GB10 # Run models on your bare metal GB10 Your bare metal machine is a whole NVIDIA GB10 with root. Nothing of ours runs your models or watches its memory: what runs on it, and how much memory it takes, is yours to decide. This page is how to serve a model on it the way we serve ours. Connecting, the network, Reboot and copying files are in Your bare metal machine. ## 1. What is on it - An NVIDIA GB10: an ARM64 CPU and a Blackwell GPU. Its memory is **128 GB, shared by the GPU and the system. About 110 GB is usable for models.** - NVIDIA's DGX OS (Ubuntu 24.04), installed fresh: the NVIDIA driver, CUDA, Docker and the NVIDIA container toolkit. You are `dev`, with `sudo`, and Docker runs without it. - Containers need an arm64 or multi-platform image: `exec format error` means one built for x86 only. ## 2. Serve a model with vLLM Our Qwen3.8 27B runs on vLLM's own container image, `vllm/vllm-openai:v0.32.0-aarch64`, since 2026-11-01. The same on yours, with a model's Hugging Face name in place of `MODEL`: ``` docker run -d --name vllm --gpus all --ipc=host -p 127.0.0.1:8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --entrypoint vllm vllm/vllm-openai:v0.32.0-aarch64 \ serve MODEL --gpu-memory-utilization 0.75 --max-model-len 32768 # the weights' size, then how many tokens of cache fit (Ctrl-C leaves it running) docker logs -f vllm # once it says it is ready curl http://localhost:8000/v1/models ``` - The model's files stay in `~/.cache/huggingface` on the machine, so the next start does not download them again. A gated model also needs `-e HF_TOKEN`, with your token in `HF_TOKEN`. - `-p 127.0.0.1:8000:8000` keeps the API on the machine. Reach it from your computer through an SSH tunnel (`-L 8000:localhost:8000`, in the bare metal guide), then use `http://localhost:8000/v1` as the base URL of any OpenAI-compatible tool. ## 3. The settings that decide the memory What each one does, and what we run Qwen3.8 27B (NVFP4) with on our own GB10: | Flag | What it does | Ours | | | `--gpu-memory-utilization` | The share of the whole machine's memory vLLM may take, whatever the size of the model. 0.75 is a safe start on a machine of its own. | 0.44: that machine runs other models too | | | `--kv-cache-memory-bytes` | A fixed size for the cache, in bytes, in place of whatever the share leaves after the weights: the server takes the same memory at every start and under any load. | 20 GiB | | | `--kv-cache-dtype fp8_e4m3` | The cache in FP8: twice the tokens in the same memory. | On | | | `--max-model-len` | The longest context of one request. | 262,144 | | | `--max-num-seqs` | How many requests run at once. | 16 | | | `--max-num-batched-tokens` | The tokens computed in one step: more reads long prompts faster and takes more memory. | 8,192 | | | `--enable-prefix-caching` | The same start of a prompt is read once, not every time. | On | | | `--speculative-config '{"method":"mtp","num_speculative_tokens":1}'` | For a model with a multi-token prediction head: it drafts the next token ahead, so one user gets tokens faster. | On | | With these, Qwen3.8 27B's 20 GiB cache holds 582,708 tokens, shared by the requests running at once: two at the whole 262,144, or 16 at about 36,000 each. ## 4. Free the file cache before a big start A new GPU program on a GB10 needs memory that is really free. The **available** column of `free -h` also counts the file cache, which a big download fills, so a start can fail with `CUDA error: out of memory` while `free -h` still shows room. On our own GB10 (2026-10-20), the really free memory went from 1.9 GB to 8.6 GB once the cache was dropped, and the server started at the first try. ``` sync; echo 3 | sudo tee /proc/sys/vm/drop_caches free -h # the free column is what a new GPU program can get ``` ## 5. Keep it running - `docker run -d` keeps the server running when you log out. `docker stop vllm` stops it, `docker start vllm` starts it again with the same settings. - To have it start again after a reboot: `docker update --restart unless-stopped vllm`. Do this only once your settings have run under your heaviest load: a server that fills the memory would start again after every Reboot and hang the machine again. ## 6. The memory: stay under about 110 GB The GPU and the system share this machine's 128 GB of memory, so about 110 GB is usable for your models. Loading more than that can make the machine unstable or hang it. Nothing on the machine stops a program that takes it all: if the machine stops answering, use Reboot in your console. - Reboot cuts the power for a moment, like pulling the plug: whatever was not saved to disk is lost, and the machine takes a few minutes to come back. - A container's memory limit (`docker run --memory`) does not protect the machine: it does not hold GPU memory on a GB10. On ours, a container limited to 12 GB took 24 GB for the GPU. - Test with your heaviest load (the longest prompts, every user at once) while `watch -n 1 free -h` runs in a second terminal. - What fits, and what we have seen happen when a GB10 is filled to the brim: What fits in a GB10's 110 GB. ## 7. When something does not work | What you see | What it means | What to do | | | `CUDA error: out of memory` as it starts, with room in `free -h` | The room is file cache | Free the cache (step 4), then start it again | | | vLLM stops while starting: the cache does not fit | The context or the requests at once do not fit | Lower `--max-model-len` or `--max-num-seqs`, use an FP8 cache, or a smaller build | | | `exec format error` | An image built for x86 only | Pull an arm64 or multi-platform image | | | The machine stops answering, even over SSH | Usually the memory | Reboot in the console, then stay under about 110 GB | | ← Run models on a rented GB10 What fits in 110 GB → Source: https://dev.axforge.ai/docs/gb10-bare-metal/