How to Run a Local LLM With Ollama

Every call to a hosted LLM API sends your prompt to someone else’s server and bills you per token. Lose the network and it stops answering. Ollama takes both problems out of the loop: it pulls an open-weight model onto your machine and serves it over a REST API on localhost:11434. One binary is the model manager, the chat client, and the server.

What Ollama actually does

Ollama wraps llama.cpp and similar inference engines so you never touch a compiler flag or a CUDA library path. ollama pull llama3.2 fetches a quantized model file (GGUF format) from Ollama’s own registry rather than from Hugging Face, though it can import GGUF files from there too. Running the model hands those weights to a background server, which loads them into RAM (or VRAM, if it finds a compatible GPU), keeps them resident for a few minutes after your last request so the next one is fast, and unloads them once you stop asking.

The CLI, the API and the Modelfile system all sit on top of that one job: get weights into memory, route requests to them.

How to install Ollama and run your first model

On Linux or macOS:

1

curl -fsSL https://ollama.com/install.sh | sh

On Windows, run the installer from ollama.com, or use:

1

irm https://ollama.com/install.ps1 | iex

Either way you end up with the ollama CLI and a background server listening on port 11434. Pull a small model and talk to it:

1
2

ollama pull llama3.2
ollama run llama3.2

ollama run drops you into an interactive prompt. Ask a question, read the answer, /bye to leave. Three other commands you will use constantly:

1
2
3

ollama list   # models you've pulled, with their size on disk
ollama ps     # models currently loaded in memory
ollama rm llama3.2   # free the disk space back

llama3.2 is a good first pull: about 2 GB, fast enough to run on a laptop CPU. It is not the strongest model in its size class. gemma4 leads the small end now and handles images, and qwen3-coder is the better pick if you are testing coding assistance. Start with llama3.2 anyway, because it proves your setup works before you commit to a 20 GB download.

How to talk to Ollama over its REST API

The interactive prompt is for checking that things work. Anything you build calls the API. Ollama’s native format looks like this:

1
2
3
4
5

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Write a one-line commit message for a typo fix in README.md",
  "stream": false
}'

With "stream": false you get one JSON object back, including eval_count and eval_duration, which is enough to measure tokens per second on your own hardware. Set it to true and you get a series of partial-response objects instead, which is what a chat UI wants.

For multi-turn conversations, /api/chat takes a messages array in the same shape OpenAI’s API uses:

1
2
3
4
5
6
7

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    {"role": "user", "content": "What port does Ollama listen on by default?"}
  ],
  "stream": false
}'

Ollama also speaks the OpenAI API shape natively at /v1, so the official OpenAI SDKs work against it unmodified:

1
2
3
4
5
6
7
8
9

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Summarize this changelog in two sentences."}],
)
print(response.choices[0].message.content)

Ollama ignores api_key, but the SDK constructor demands one, so any non-empty string works. That makes this the fastest way to point an existing OpenAI-based tool at a local model: change base_url, leave the rest of the code alone.

One more endpoint matters if you are assembling a local retrieval-augmented generation pipeline. POST /api/embed turns text into vectors using an embedding model (nomic-embed-text is the usual pull), and a chat model on its own will not give you those.

How to run Ollama in Docker

If everything else on the machine is already a container, Ollama ships an official image instead of asking you to install anything on the host:

1
2
3
4
5

docker run -d \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

The named volume is not optional in practice: without it, every model you pulled disappears the moment the container is removed. To pull and run a model inside the running container:

1

docker exec -it ollama ollama run llama3.2

On a machine with an NVIDIA GPU, install the NVIDIA Container Toolkit on the host first, then add one flag:

1
2
3
4
5

docker run -d --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

AMD GPUs use a separate image tag and device mounts instead of --gpus:

1
2
3
4
5

docker run -d --device /dev/kfd --device /dev/dri \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama:rocm

The same thing as a Docker Compose service carries over the port and the volume, nothing more:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

services:
  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama

volumes:
  ollama:

Exposing that container past localhost is a job for a reverse proxy with TLS in front of it, the same as any other backend service. Ollama has no authentication of its own, so a port 11434 reachable from the network is a model server anyone can use.

How to customize a model with a Modelfile

A Modelfile pins a system prompt, a temperature and a stop sequence onto an existing model without retraining it. Create a file named Modelfile:

1
2
3
4
5
6
7
8

FROM llama3.2

PARAMETER temperature 0.3

SYSTEM """
You are a terse code reviewer. Reply with the diff and one sentence of
rationale. Never restate the code you were not asked to change.
"""

Then build and run it:

1
2

ollama create code-reviewer -f Modelfile
ollama run code-reviewer

code-reviewer now appears as its own entry in ollama list, though the weights are exactly the ones llama3.2 uses. Only the system prompt and the sampling settings are baked in. The payoff is that every client calling that name inherits the configuration without sending it.

Which model size fits your RAM

The number in a model’s name is its parameter count, and it roughly predicts how much RAM (or VRAM, for GPU inference) you need:

Those numbers assume the 4-bit quantization Ollama pulls by default, which trades a small amount of output quality for a download and a memory footprint around four times smaller than the full-precision weights. For summarizing text or drafting commit messages, quantization is not the thing you will notice. Running out of RAM is. Once the model spills into disk swap it goes from slower to unusable, which is a much bigger difference than a point of accuracy.

When Ollama is the wrong tool

Ollama is built for one user talking to one model on one machine. It does no request batching and none of the scheduling that lets a GPU serve dozens of concurrent users efficiently, which is what vLLM, TGI or a hosted API are for. If you are putting a local model behind real traffic, measure Ollama’s throughput under concurrent load before you commit to it; a single instance falls over well before a properly batched server would.

It is also not a fine-tuning tool. A Modelfile changes a system prompt and sampling parameters, never the weights. Getting a model to learn new behavior from your examples is a separate pipeline.

Neither limit applies to one developer on one laptop. Pull a model that sits comfortably under your RAM ceiling, run it once from the CLI, then hit the same model with curl on /api/generate. If the answer comes back in a couple of seconds and reads the way you expected, you have a local inference server that any OpenAI-compatible tool can point at.

Related articles

• What Is RAG: Retrieval-Augmented Generation Explained
• What Is Docker Compose and How to Use It
• What Is a Reverse Proxy and How It Works