Serving an Open-Weight LLM
Assignment 2 and the course project both use open-weight models that you serve yourself, on your laptop or on a free PACE ICE GPU. Every server here speaks the OpenAI API, so your code is the same in both places; only the base URL changes.
| Page | Use it when |
|---|---|
| On your laptop | The model fits in your memory (a 7–9B model on a 16 GB laptop). Start here. |
| On PACE ICE | Your laptop is too small or too slow, or you need many long runs. GT VPN required. |
Common failures
These three cause most silent failures. Set them explicitly rather than relying on defaults.
| Setting | Why |
|---|---|
Context length (OLLAMA_CONTEXT_LENGTH, llama-server -c) | Ollama’s default is small (4096 tokens in the versions we tested) and it silently cuts the beginning of longer prompts. The only trace is a truncating input prompt line in the server log. |
| Temperature | Default sampling differs by model (for example 0.6 for Qwen3, 1.0 for Gemma 3 in Ollama). Pass temperature=0 when you want reproducible results. |
Parallel requests (OLLAMA_NUM_PARALLEL) | Sending requests from a thread pool is several times faster than one at a time, but every slot reserves its own context memory (see the memory notes on each page). |