Run models on your bare metal GB10

Your bare metal machine is a whole NVIDIA GB10 with root. Nothing of ours runs your models or watches its memory: what runs on it, and how much memory it takes, is yours to decide. This page is how to serve a model on it the way we serve ours. Connecting, the network, Reboot and copying files are in Your bare metal machine.

1. What is on it

• An NVIDIA GB10: an ARM64 CPU and a Blackwell GPU. Its memory is 128 GB, shared by the GPU and the system. About 110 GB is usable for models.
• NVIDIA's DGX OS (Ubuntu 24.04), installed fresh: the NVIDIA driver, CUDA, Docker and the NVIDIA container toolkit. You are dev, with sudo, and Docker runs without it.
• Containers need an arm64 or multi-platform image: exec format error means one built for x86 only.

2. Serve a model with vLLM

Our Qwen3.8 27B runs on vLLM's own container image, vllm/vllm-openai:v0.32.0-aarch64, since 2026-11-01. The same on yours, with a model's Hugging Face name in place of MODEL:

docker run -d --name vllm --gpus all --ipc=host -p 127.0.0.1:8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --entrypoint vllm vllm/vllm-openai:v0.32.0-aarch64 \
  serve MODEL --gpu-memory-utilization 0.75 --max-model-len 32768
# the weights' size, then how many tokens of cache fit (Ctrl-C leaves it running)
docker logs -f vllm
# once it says it is ready
curl http://localhost:8000/v1/models

• The model's files stay in ~/.cache/huggingface on the machine, so the next start does not download them again. A gated model also needs -e HF_TOKEN, with your token in HF_TOKEN.
• -p 127.0.0.1:8000:8000 keeps the API on the machine. Reach it from your computer through an SSH tunnel (-L 8000:localhost:8000, in the bare metal guide), then use http://localhost:8000/v1 as the base URL of any OpenAI-compatible tool.

3. The settings that decide the memory

What each one does, and what we run Qwen3.8 27B (NVFP4) with on our own GB10:

With these, Qwen3.8 27B's 20 GiB cache holds 582,708 tokens, shared by the requests running at once: two at the whole 262,144, or 16 at about 36,000 each.

4. Free the file cache before a big start

A new GPU program on a GB10 needs memory that is really free. The available column of free -h also counts the file cache, which a big download fills, so a start can fail with CUDA error: out of memory while free -h still shows room. On our own GB10 (2026-10-20), the really free memory went from 1.9 GB to 8.6 GB once the cache was dropped, and the server started at the first try.

sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
free -h      # the free column is what a new GPU program can get

5. Keep it running

• docker run -d keeps the server running when you log out. docker stop vllm stops it, docker start vllm starts it again with the same settings.
• To have it start again after a reboot: docker update --restart unless-stopped vllm. Do this only once your settings have run under your heaviest load: a server that fills the memory would start again after every Reboot and hang the machine again.

6. The memory: stay under about 110 GB

The GPU and the system share this machine's 128 GB of memory, so about 110 GB is usable for your models. Loading more than that can make the machine unstable or hang it. Nothing on the machine stops a program that takes it all: if the machine stops answering, use Reboot in your console.

• Reboot cuts the power for a moment, like pulling the plug: whatever was not saved to disk is lost, and the machine takes a few minutes to come back.
• A container's memory limit (docker run --memory) does not protect the machine: it does not hold GPU memory on a GB10. On ours, a container limited to 12 GB took 24 GB for the GPU.
• Test with your heaviest load (the longest prompts, every user at once) while watch -n 1 free -h runs in a second terminal.
• What fits, and what we have seen happen when a GB10 is filled to the brim: What fits in a GB10's 110 GB.

7. When something does not work