Serving a Model on PACE ICE

Serve a model on a Georgia Tech cluster GPU when it will not fit on your laptop. GPU time is free. Turn on the GT VPN first. Assignment 2 ships its own batch script (scripts/run_pace.sbatch) built the same way.

File	Purpose
`setup.sh`	Send model downloads to scratch, run once
`serve.sbatch`	Start a server on one GPU

1. Setup

ssh <gtusername>@login-ice.pace.gatech.edu
mkdir -p ~/serving_pace && cd ~/serving_pace
curl -O https://kexinrong.github.io/fa26-cs6400/assets/llm_serving/setup.sh
curl -O https://kexinrong.github.io/fa26-cs6400/assets/llm_serving/serve.sbatch
bash setup.sh

Your home directory holds 30 GB and your scratch directory holds 300 GB. Each model is several GB. setup.sh therefore points the model cache at scratch. Run it once per account.

2. Serve a model

mkdir -p logs
sbatch serve.sbatch

squeue -u $USER                  # empty means finished
tail -f logs/serve-<jobid>.out
scancel <jobid>

The job starts Ollama, downloads both models, and checks the endpoint. Put your own script at the marked line in serve.sbatch.

Port. Each job serves on its own port (printed at the top of the log), because two jobs can land on the same node and would otherwise share 11434, one of them silently talking to the other’s server. Inside the job, use http://$OLLAMA_HOST/v1.
Python. module load llm-python-apis gives you Python with openai, but it unloads the ollama module. The server keeps running; serve.sbatch shows how to keep the ollama command available.
Context length. serve.sbatch sets OLLAMA_CONTEXT_LENGTH=8192 and prints how many prompts were truncated at the end of the job. Raise it if that number is not 0 (see memory in section 4).

Queue waits run from a minute to a few hours. A job showing the reason Priority is waiting behind other people’s jobs. Submit early rather than an hour before a deadline. The server starts and stops with the job.

Ask for short jobs. PACE caps the GPU time each user can have reserved at once. A job waiting with the reason MaxGRESRunMinsPerUser has hit that cap, and every running job’s whole --time counts against it. Several 1-hour jobs start sooner than one 8-hour job.

3. Interactive session

salloc -p ice-gpu --gres=gpu:v100:1 --cpus-per-task=4 --mem=32G -t 00:10:00

export HOSTNAME=$(hostname)            # the ollama module needs this
module load ollama
export OLLAMA_CONTEXT_LENGTH=8192
ollama serve &> ~/scratch/ollama.log &
sleep 3 && ollama pull qwen3:8b && ollama pull nomic-embed-text

curl http://127.0.0.1:11434/v1/models

Type exit to release the GPU. A GPU job runs for at most 16 hours. It gets 1 hour when you leave out -t.

4. Models and GPUs

Model	Size	GPU	Request
`qwen3:8b`	5.2 GB	V100 16 GB	default in `serve.sbatch`
`qwen3:14b`	9.3 GB	V100 16 GB	`--export=MODEL=qwen3:14b`
`qwen3:32b`	20.2 GB	L40S 48 GB	edit `--gres=gpu:l40s:1`
`gpt-oss:120b`	65 GB	H100 80 GB	edit `--gres=gpu:h100:1`

ICE has 22 V100s, 32 L40S, and 48 H100s. V100s are the least contended.

sinfo -p ice-gpu -t idle,mix -o "%G %D %T" -h | sort | uniq -c

PACE pre-downloaded llama3.1:8b and phi4:14b. Add the line below above ollama serve to use them with no download. That store rejects writes. Remove the line before you pull anything new.

export OLLAMA_MODELS=/storage/ice1/shared/d-pace_community/ollama/models

5. Concurrency

OLLAMA_NUM_PARALLEL sets how many requests the server answers at once. Send them from a thread pool in your own code. Each slot reserves its own context memory: OLLAMA_CONTEXT_LENGTH × OLLAMA_NUM_PARALLEL tokens of key/value cache on top of the model (per-token costs are in the laptop guide). qwen3:8b with 8k context and 4 slots needs about 5.2 + 4.8 GB, which fits a V100; 16k with 8 slots would not.

6. Reaching the server from your laptop

A job only listens on the compute node it runs on. Reach it from your laptop in three steps.

Set OLLAMA_HOST=0.0.0.0:$PORT in serve.sbatch so the server accepts outside connections.
End the script with sleep infinity so the job holds the GPU after its work finishes.
Read the node name and port from the job log and open a tunnel through the login node.

ssh -L 11434:<node>:<port> <gtusername>@login-ice.pace.gatech.edu

http://localhost:11434/v1 on your laptop now reaches the cluster GPU.

7. Common errors

Message	Fix
`could not connect to ollama`	Every job needs its own `ollama serve`
`bad argument #1 to 'find'` on `module load ollama`	`export HOSTNAME=$(hostname)` first
`ollama: command not found` after `module load llm-python-apis`	That module unloads ollama; see the commented lines in `serve.sbatch`
Answers ignore the start of long prompts	Context truncation: raise `OLLAMA_CONTEXT_LENGTH`
`Disk quota exceeded`	Rerun `bash setup.sh`
`404` on embeddings	Embedding model never pulled, check the job log
`read-only file system`	Remove the `OLLAMA_MODELS` line
`CUDA out of memory`	Smaller model, or lower `OLLAMA_NUM_PARALLEL`
`Permission denied`	Run as `bash script.sh`
Empty answer	Raise `max_tokens` above 1000

Email pace-support@oit.gatech.edu about problems with your account or with the cluster.