AI Engineering

Ask a model to reply with JSON and most of the time json.loads() works fine. The failures are the ones that cost you a debugging session: a one-line comment inside the object, a markdown code fence wrapped around it, a key spelled userEmail when your code expects user_email. A stricter prompt (“reply with ONLY the JSON object, no other text”) makes these rarer. It doesn’t make them impossible, because the model can still ignore a request. Structured output removes the request from the equation: you hand the API a JSON Schema, and it constrains the model’s own token sampling so the reply has to match it. ...

Why “it looks right” isn’t a test for a prompt You tighten the wording of a ticket-routing prompt, run it against the three examples open in your scratch notebook, and ship it. A week later, billing complaints that used to land in the right queue start landing in “question” instead. Not the three cases you checked: the other four hundred you didn’t. Nothing crashed. There’s no stack trace. The prompt still reads fine out loud. ...

What LLM function calling actually does Ask a model “what’s the weather in Austin right now” with no connection to the outside world, and it has two options: say it doesn’t know, or make something up. Function calling gives it a third option. It describes what it needs, hands that description to your code, and waits for a real answer before replying. The name is slightly misleading. The model does not call anything. It emits a block of structured data saying which tool it wants and with what arguments, and your application decides whether to run it and what to do with the result. “Tool use” is the more accurate term, and it’s the one Anthropic’s docs use; OpenAI and Google mostly say function calling. The mechanism is the same under both names. The model only ever produces a request, never the side effect. ...

Search for “puppy” in a system that only matches keywords and a document about “dog training” will not show up, even though anyone reading both would call them closely related. Keyword search compares strings. It has no idea that “puppy” and “dog” mean nearly the same thing. Embeddings fix that. They turn a piece of text into a list of numbers, a vector, placed so that text with similar meaning lands near other text with similar meaning, whatever words it happens to use. ...

What makes one CLI coding agent different from another Ask a CLI coding agent to fix a failing test and it won’t reply with a suggestion. It reads the test output, opens the file, edits a line, reruns the test, and reports back once it passes. That loop is the product, not the chat window. The model inside the loop gets most of the attention, and it’s also the part that changes fastest: vendors swap in new ones every few months, and several agents let you pick. What stays put is the harness wrapped around the model — which tools it can reach, how it manages context, what it may run without asking, and where those commands execute. Two agents running the same model can behave very differently because of it. The harness is what you’re actually choosing. ...

Why so many LLM calls are really classification A lot of production LLM traffic returns one word. Escalate this ticket or not. Spam or not spam. Which of six handlers gets this event. To get that word, you send the whole prompt to a general-purpose model, wait while it decodes a reply token by token, and parse a JSON object out of the result. You pay the price and the latency of text generation, and what comes back is a decision that fits in an enum. ...

Every call to a hosted LLM API sends your prompt to someone else’s server and bills you per token. Lose the network and it stops answering. Ollama takes both problems out of the loop: it pulls an open-weight model onto your machine and serves it over a REST API on localhost:11434. One binary is the model manager, the chat client, and the server. What Ollama actually does Ollama wraps llama.cpp and similar inference engines so you never touch a compiler flag or a CUDA library path. ollama pull llama3.2 fetches a quantized model file (GGUF format) from Ollama’s own registry rather than from Hugging Face, though it can import GGUF files from there too. Running the model hands those weights to a background server, which loads them into RAM (or VRAM, if it finds a compatible GPU), keeps them resident for a few minutes after your last request so the next one is fast, and unloads them once you stop asking. ...

Ask a general-purpose LLM about your company’s refund policy or last week’s support tickets, and it will either say it doesn’t know or invent something plausible. Its knowledge stops at whatever was in its training data. It has never seen your documents, and it cannot go and look them up mid-conversation. Retrieval-augmented generation (RAG) is the fix: before the model answers, a separate step finds the relevant text in your own data and hands it to the model as part of the prompt. ...