Structured Output From LLMs: JSON Schema Explained

Ask a model to reply with JSON and most of the time json.loads() works fine. The failures are the ones that cost you a debugging session: a one-line comment inside the object, a markdown code fence wrapped around it, a key spelled userEmail when your code expects user_email. A stricter prompt (“reply with ONLY the JSON object, no other text”) makes these rarer. It doesn’t make them impossible, because the model can still ignore a request. Structured output removes the request from the equation: you hand the API a JSON Schema, and it constrains the model’s own token sampling so the reply has to match it.

JSON mode vs. structured output: what each one guarantees

“JSON mode” (response_format: {"type": "json_object"} in the older OpenAI API, and the equivalent on most other providers) guarantees one thing: the output parses as JSON. It says nothing about which keys exist, what type they hold, or whether a field you depend on is present at all. A model in JSON mode can return {"result": "yes"} when your code expects {"status": "approved", "confidence": 0.91}, and nothing tells you it went wrong.

Structured output (OpenAI calls it Structured Outputs, Anthropic calls it JSON outputs) is a separate guarantee: the reply matches a JSON Schema you supply, with the required keys present and the right type on each one. Function calling solves an adjacent problem, getting the model to request that your code run something. The two are easy to confuse because both pass a schema. What differs is what the schema shapes:

A production pipeline often uses the last two together. Strict tool use keeps an agent’s function calls from arriving malformed, and a strict schema on the final answer types the summary it hands back to the user.

How constrained decoding forces a match

The model never checks its answer against your schema after generating it. The schema changes what it’s allowed to generate, one token at a time.

The serving layer compiles your JSON Schema into a grammar: a state machine that tracks, at every position in the output, which tokens would still lead to something valid. Before each token is sampled, every vocabulary token the grammar rules out at that position gets its logit pushed to negative infinity, so it has zero probability after the softmax. The model samples from what’s left, and the relative probabilities among the surviving tokens are preserved. A misspelled key, a closing brace in the wrong place, a string where the schema demands a number: once the grammar forbids them, those tokens are not options.

That compilation step has a cost. Building a grammar for a new schema takes real work, so the first request against it is slower than the rest. Anthropic’s API caches the compiled grammar for 24 hours after its last use, and later calls with the same schema skip compilation.

The mechanism also explains the limit. Constrained decoding fixes shape, nothing else. The grammar knows which tokens are syntactically legal, not which one is true, so the model can still fill a valid "email" field with an address it invented.

How to get schema-valid JSON from OpenAI, Claude, and Ollama

OpenAI’s Responses API takes the schema in text.format, with strict turned on:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

from openai import OpenAI

client = OpenAI()

response = client.responses.create(
    model="gpt-6-astra",
    input="Extract the ticket details: 'Login fails with a 500 after the "
          "password reset flow, seen first on build 4.12, blocking release.'",
    text={
        "format": {
            "type": "json_schema",
            "name": "ticket",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "summary": {"type": "string"},
                    "build": {"type": "string"},
                    "severity": {"type": "string", "enum": ["low", "medium", "high", "blocking"]},
                },
                "required": ["summary", "build", "severity"],
                "additionalProperties": False,
            },
        }
    },
)
print(response.output_text)

Claude takes the schema in output_config.format, not in a tool definition:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

curl https://api.anthropic.com/v1/messages \
  -H "content-type: application/json" \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2026-10-21" \
  -d '{
    "model": "claude-sonnet-5-5",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Extract the ticket details: ..."}],
    "output_config": {
      "format": {
        "type": "json_schema",
        "schema": {
          "type": "object",
          "properties": {
            "summary": {"type": "string"},
            "build": {"type": "string"},
            "severity": {"type": "string", "enum": ["low", "medium", "high", "blocking"]}
          },
          "required": ["summary", "build", "severity"],
          "additionalProperties": false
        }
      }
    }
  }'

The JSON comes back in the response’s text content, not in a tool-call block. The older workaround (define a fake tool, force the model to call it, read the arguments) still works on models that only support tool use. Where native structured output exists you don’t need it: the decoder enforces the schema directly instead of hoping the tool arguments happen to match.

A model running locally under Ollama gets the schema through the format field of /api/chat:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "Extract the ticket details: ..."}],
  "format": {
    "type": "object",
    "properties": {
      "summary": {"type": "string"},
      "build": {"type": "string"},
      "severity": {"type": "string", "enum": ["low", "medium", "high", "blocking"]}
    },
    "required": ["summary", "build", "severity"]
  },
  "stream": false
}'

The mechanism is the same. Ollama’s llama.cpp runtime builds a grammar from the schema and masks tokens the way the hosted APIs do.

Which JSON Schema features do providers support?

Every provider enforces a subset of JSON Schema, and the subsets differ enough to break a schema you copy from one API to another.

OpenAI requires every object to carry "additionalProperties": false with every one of its properties listed in required — there’s no way to mark a field truly optional, so model one as required and nullable instead. Anthropic is more permissive: a property left out of required is genuinely optional, and required properties come first in the output, followed by the optional ones that are present. Past that, the two diverge further. Anthropic supports local $ref/$defs, anyOf, and const, but rejects recursive schemas, numeric bounds like minimum and maximum, and string-length constraints like maxLength. OpenAI’s examples include a recursive $ref: "#" for tree-shaped output, which Anthropic’s docs explicitly rule out.

When your schema needs a constraint the provider doesn’t support, move it into the field’s description (“must be a positive integer under 100”) and check it yourself after parsing. The SDK schema helpers on both platforms do this for you when you build the schema from a Pydantic or Zod model instead of writing raw JSON Schema.

How to validate structured output after parsing

A valid shape is not a correct answer. The schema guarantees severity is one of your four enum values. Whether “blocking” is the right call for this ticket is a separate question. Treat the parsed object the way you’d treat a form submission: input to validate, not ground truth.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

from pydantic import BaseModel, field_validator

class Ticket(BaseModel):
    summary: str
    build: str
    severity: str

    @field_validator("build")
    @classmethod
    def must_look_like_a_build_number(cls, v: str) -> str:
        if not v.replace(".", "").isdigit():
            raise ValueError(f"'{v}' doesn't look like a build number")
        return v

Instructor wraps a validate-and-retry loop around OpenAI, Anthropic, and most other providers. You give it a Pydantic model. On a validation failure it feeds the error back to the model and asks again, up to a retry limit you set, instead of raising a ValidationError on the first miss. It’s also the practical route to a schema guarantee on a provider or older model with no native strict mode: the retry loop stands in for decoder-level enforcement, and each miss costs an extra round trip.

Field validators catch malformed values in one response. To check whether the answers are right across hundreds of inputs, and whether a prompt change made them worse, you need an eval suite.

When structured output still fails

Strict mode doesn’t cover these cases. Check for them before you go hunting for a bug in your own code:

• Refusals. A model can refuse to answer, and the refusal text does not match your schema. Check the stop_reason (or equivalent) before you parse, not after the parse throws.
• Truncation. Hitting max_tokens mid-object produces incomplete JSON. The grammar controls which token comes next, not how much room is left to finish. Raise the limit or shorten the schema.
• Unenforced paths. JSON mode, a retry library on a model without strict mode, or a provider that silently ignores a schema keyword it doesn’t support can all return "Blocking" when your enum says "blocking". If any request in your system takes one of those paths, normalize values on the way in.
• Schema size. A large schema with many anyOf branches adds tokens to every request, on top of the compilation delay for a new schema. Keep schemas as flat and small as the task allows.

The provider’s guarantee is real, but your parsing code should not assume it held.

JSON mode, strict schema, or function calling: which to use?

Use plain JSON mode only against a model or SDK path with no schema support, as a fallback rather than a default. Use a strict JSON Schema when you’re shaping an answer for your own code or a user: extraction, classification, a typed summary. Use function calling when the model decides whether to run something. Turn strict mode on for the tool definition too, so a malformed call fails at the API instead of inside your code.

If all you need back is a label from a short enum, compare a system one model as well. When the output is restricted to a known set of answers, a model can pick one instead of generating it, and the latency and cost tradeoffs differ enough that you should measure them on your own traffic.

For everything else, the default is a strict schema plus a validator on the parsed result. The schema keeps the shape correct. Checking that the content is correct is still your job.

Related articles

• LLM Function Calling and Tool Use, Explained
• How to Build an LLM Eval: Testing Prompts Like Code
• What Is a System One Model and When to Use One
• How to Run a Local LLM With Ollama