AI Engineering

Every call to a hosted LLM API sends your prompt to someone else’s server and bills you per token. Lose the network and it stops answering. Ollama takes both problems out of the loop: it pulls an open-weight model onto your machine and serves it over a REST API on localhost:11434. One binary is the model manager, the chat client, and the server. What Ollama actually does Ollama wraps llama.cpp and similar inference engines so you never touch a compiler flag or a CUDA library path. ollama pull llama3.2 fetches a quantized model file (GGUF format) from Ollama’s own registry rather than from Hugging Face, though it can import GGUF files from there too. Running the model hands those weights to a background server, which loads them into RAM (or VRAM, if it finds a compatible GPU), keeps them resident for a few minutes after your last request so the next one is fast, and unloads them once you stop asking. ...

Ask a general-purpose LLM about your company’s refund policy or last week’s support tickets, and it will either say it doesn’t know or invent something plausible. Its knowledge stops at whatever was in its training data. It has never seen your documents, and it cannot go and look them up mid-conversation. Retrieval-augmented generation (RAG) is the fix: before the model answers, a separate step finds the relevant text in your own data and hands it to the model as part of the prompt. ...