Serving a Model on Your Laptop

The examples use a chat model and an embedding model, as in a RAG project. For Assignment 2, pull qwen2.5-coder:7b instead and use the settings in the assignment’s docs/LLM_SETUP.md.

1. Server choices

Tool	Windows	macOS	Linux	Choose if you
Ollama	Yes	Yes	Yes	want simple setup and model downloads
llama.cpp	Yes	Yes	Yes	want direct GGUF and quantization control
LM Studio	Yes	Apple Silicon	Yes	prefer a GUI

Ollama

Install: Ollama Quickstart

ollama pull qwen3.5:4b           # chat model
ollama pull nomic-embed-text     # embedding model
ollama list                      # what you have downloaded
OLLAMA_CONTEXT_LENGTH=8192 ollama serve   # see "Context length" below

A pull downloads once and reuses the file afterwards. ollama rm <model> frees the disk.

llama.cpp

Install: llama.cpp installation

llama-server -hf Qwen/Qwen3-4B-GGUF:Q4_K_M -c 4096 --port 8080
llama-server -hf nomic-ai/nomic-embed-text-v1.5-GGUF:Q4_K_M --embeddings --port 8081

LM Studio

Install: LM Studio download

lms server start --port 1234

Then load a chat model and an embedding model through the GUI.

2. Models to try

Any model your machine can hold will work. These are examples to start from rather than required choices.

Model	Family	Size
`qwen3:1.7b`	Qwen	1.4 GB
`gemma3:4b`	Gemma	3.3 GB
`qwen3.5:4b`	Qwen	3.4 GB
`llama3.1:8b`	Llama	4.9 GB
`qwen3.5:9b`	Qwen	6.6 GB

Embedding model	Size
`all-minilm`	46 MB
`nomic-embed-text`	274 MB
`embeddinggemma`	622 MB
`bge-m3`	1.2 GB

Start small. Browse more at Ollama models or Hugging Face GGUF.
GGUF is the single-file model format that llama.cpp and LM Studio read.

How much your machine can hold

Memory	Chat model up to	Example
8 GB	about 3 GB	`qwen3:1.7b`
16 GB	about 7 GB	`qwen3.5:9b`
32 GB	about 16 GB	`qwen3:14b`
64 GB	about 35 GB	`qwen3:32b`

On Apple Silicon the model shares one memory pool with everything else. A 16 GB laptop runs qwen3.5:9b comfortably alongside an embedding model.
With a discrete NVIDIA GPU the limit is VRAM. All three servers find the GPU automatically. A model larger than VRAM still runs. It splits across the GPU and CPU and slows down sharply.
Ollama keeps up to three models loaded at once and unloads each one after five minutes idle.

Quantization

Quantization stores a model’s weights at reduced precision so the model fits in less memory. Every model above is quantized. A bare tag such as qwen3.5:9b gives you q4_K_M. That is roughly 4 bits per weight. Pulling the explicit tag downloads the identical file.

Tag	Precision	`qwen3.5:9b`	`llama3.1:8b`	`gemma3:4b`
bare tag, which is `q4_K_M`	about 4 bits	6.6 GB	4.9 GB	3.3 GB
`q8_0`	8 bits	10.7 GB	8.5 GB	5.0 GB

In q4_K_M the K means a mixed scheme that keeps the more sensitive layers at higher precision. The M means medium.
Each family spells the explicit tag differently. The three above are qwen3.5:9b-q4_K_M, llama3.1:8b-instruct-q4_K_M, and gemma3:4b-it-q4_K_M.
q8_0 costs 50 to 75 percent more memory for a small quality gain.
A model needs roughly its download size in free memory. The context window needs more on top of that.

Context length and memory

Ollama’s default context is small (4096 tokens in the versions we tested), and a longer prompt is cut from the beginning without an error. Only a truncating input prompt line in the server log shows it. Set the context explicitly, and keep your prompts inside it:

OLLAMA_CONTEXT_LENGTH=8192 OLLAMA_NUM_PARALLEL=2 ollama serve
llama-server -hf <model> -c 16384 -np 2      # llama.cpp: -c is the total, shared by the -np slots

The context is memory reserved up front: Ollama reserves OLLAMA_CONTEXT_LENGTH × OLLAMA_NUM_PARALLEL tokens of key/value cache next to the model. At 16-bit precision that costs, per token:

Model	KV cache per token	8k context × 2 slots	16k × 4
`qwen2.5-coder:7b`, `qwen2.5:7b`	56 KB	0.9 GB	3.8 GB
`llama3.1:8b`	128 KB	2.1 GB	8.6 GB
`qwen3:8b`	144 KB	2.4 GB	9.7 GB

On a 16 GB laptop, keep model + cache under about 8 GB.

3. Connect to the server

All three servers support the OpenAI API format.

Server	SDK base URL	Chat path	Embeddings path
llama.cpp	`http://127.0.0.1:8080/v1`	`/chat/completions`	`/embeddings`
Ollama	`http://127.0.0.1:11434/v1`	`/chat/completions`	`/embeddings`
LM Studio	`http://127.0.0.1:1234/v1`	`/chat/completions`	`/embeddings`

Each base URL above uses that server’s default port. Set the port yourself when something else already holds the default or when you want two servers running at once.

OLLAMA_HOST=127.0.0.1:11500 ollama serve     # Ollama
llama-server -hf <model> --port 9000         # llama.cpp
lms server start --port 1234                 # LM Studio

python -m pip install openai

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:11434/v1", api_key="local")
print([m.id for m in client.models.list().data])

Use the model ID exactly as printed.

4. Send a request

This runs as written once your server is up.

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:11434/v1",  # Change server/port as needed.
    api_key="local",                       # Placeholder; stays local.
)

response = client.chat.completions.create(
    model="qwen3.5:4b",                    # Use the /v1/models ID.
    messages=[
        {"role": "system", "content": "Answer in one sentence."},
        {"role": "user", "content": "What is a vector embedding?"},
    ],
    temperature=0,                         # set it: defaults differ by model (Qwen3 0.6, Gemma 3 1.0)
    max_tokens=1200,                       # Qwen3 reasons first. One word cost 210 tokens.
)
print(response.choices[0].message.content)

vectors = client.embeddings.create(
    model="nomic-embed-text",
    input=["the first piece of text", "the second piece of text"],
).data
print(len(vectors), "vectors of", len(vectors[0].embedding), "numbers each")

A vector embedding is a numerical representation of data that captures semantic
relationships by mapping items into a continuous space where similar objects are
positioned close to one another.
2 vectors of 768 numbers each

Qwen3 and Qwen3.5 reason before they answer and spend max_tokens doing it. A budget under about 1000 could return an empty string instead of an error.

For RAG put your retrieved passages in the user message. Tell the model in the system message to answer only from those passages.

More details: OpenAI Python format · Ollama compatibility · LM Studio compatibility