Getting Reliable JSON From LLMs: Structured Outputs, Validation, and Retries

The short answer: To get reliable JSON from an LLM, stop asking nicely and start constraining. Use your provider's structured-output feature (schema-constrained decoding) so the response always matches your JSON Schema, then validate the parsed object in your own code with a library like Zod or Pydantic, because a valid shape can still hold wrong values. Check for truncation before parsing, design flat schemas with enums and explicit nulls, and send validation errors back to the model for a limited number of retries.

There's a particular kind of bug report that every team building on LLMs eventually receives. It usually arrives at an inconvenient hour, and the stack trace ends in something like SyntaxError: Unexpected token 'H', "Here's the"... is not valid JSON.

The model was supposed to return an object. It did, sort of. It returned a friendly sentence, then a markdown code fence, then the object, then another sentence offering to help further. Your parser read the first character, saw the letter H, and gave up. The feature had worked in every test. It worked for the first ten thousand requests in production. Then one day the model decided to be polite.

That's the core problem with structured data from language models. They don't produce data structures. They produce text, one token at a time, and "text that happens to be valid JSON" is a statistical tendency rather than a guarantee. The good news is that this is now a largely solved problem, as long as you use the right combination of tools. This guide walks through what goes wrong, which fixes address which failures, and how to build an extraction pipeline that holds up when it meets real traffic.

Five ways structured output fails

"The JSON broke" covers several distinct failures, and they need different fixes. It helps to name them.

Failure	Example	Fixed by
Not JSON at all	Prose before or after the object, markdown fences	JSON mode or constrained decoding
Malformed JSON	Truncated output, trailing comma, unescaped quote	Constrained decoding plus a truncation check
Valid JSON, wrong shape	Missing field, "price": "12.99" as a string, invented extra keys	Constrained decoding with a schema
Right shape, invalid values	A status outside your enum, a date in the past, an ID that doesn't exist	Validation in your code
Valid and wrong	Perfectly formed object containing the wrong customer's email	Evals, better inputs, human review

Notice how the fixes stack. Each level of tooling eliminates some failures and leaves the rest for the next layer. Teams get into trouble when they fix the first two rows and assume they've solved the whole table.

Level 0: asking for JSON in the prompt

Everyone starts here. You write "Respond only with valid JSON matching this format" and paste an example. Modern models follow that instruction most of the time, which is exactly what makes it dangerous. "Most of the time" passes every demo and then fails at a rate that's low enough to be hard to reproduce and high enough to page someone.

The typical failures at this level are the chatty preamble, markdown fences around the object, comments inside the JSON (which JSON doesn't allow), and single quotes instead of double quotes. People respond by writing cleanup code: strip the fences, find the first { and the last }, try to parse, try again with trailing commas removed. I've seen these helper functions grow to a hundred lines. They work until they don't, and they hide the real problem, which is that nothing is actually enforcing the format.

Prompting still matters, even when you use the stronger techniques below. A clear description of each field and a realistic example make the values better. But prompting alone should never be your format guarantee.

Level 1: JSON mode

Several providers offer a "JSON mode" that guarantees the output parses as JSON. That eliminates the first row of the table: no more preambles or fences.

What it doesn't guarantee is the shape. You'll get valid JSON, but it might be {"result": "success"} when you wanted a list of line items. It might use customerName when your code expects customer_name. It might nest things differently on Tuesdays. JSON mode turns a parsing problem into a validation problem, which is progress, but it's not the destination.

There's also one failure that no output mode can prevent: truncation. If the response hits your max_tokens limit halfway through an object, you get half an object. We'll come back to that, because it's one of the most common causes of "random" JSON errors in production.

Level 2: schema-constrained decoding

This is the technique that changed the game, and it's worth understanding how it works, because the mechanism explains both its strengths and its limits.

A language model generates text by repeatedly picking the next token from a probability distribution over its whole vocabulary. Constrained decoding adds one step: before each pick, the system works out which tokens would keep the output valid according to a grammar, and masks out every other token. If the schema says the next thing must be a number, tokens that would start a string are simply not allowed. If the object has three required keys and two have been written, the closing brace isn't allowed yet. The model can still choose among the valid options, but it physically cannot produce output that breaks the schema.

Most major APIs now offer this. OpenAI's structured outputs accept a JSON Schema and constrain the response to match it. Anthropic's API supports structured outputs for responses and a strict mode for tool definitions, which guarantees that tool-call arguments validate against the tool's input schema. If you run open models yourself, libraries such as Outlines and the grammar support in llama.cpp provide the same capability.

If your provider offers constrained decoding, use it. It eliminates the first three rows of the failure table in one move. But know the limits:

• Schema support is a subset. Providers typically support a large but incomplete part of JSON Schema. Some keywords, deep recursion, or very large schemas may be unsupported or rejected. Check your provider's documentation and keep schemas simple.
• Strict mode often has rules of its own. For example, some implementations require every property to be listed as required and additionalProperties set to false. Optional fields are then expressed as a type that allows null.
• Shape is not truth. A schema can say a field is a string in ISO date format. It can't say the date must be in the future, or must match the date mentioned in the email. The model will always produce a valid-looking value, even when the honest answer is "I don't know".
• Truncation still happens. Constrained decoding can't help if generation stops at the token limit.
• Refusals and safety stops exist. A model may decline a request, and the response then won't contain your object at all. Your code needs to check why generation stopped before assuming there's data to parse.

That third point deserves emphasis. Constrained decoding can make hallucination more convincing, not less, because the made-up value arrives in a perfectly formatted field. If the schema requires an invoice_number and the document doesn't have one, the model has to put something there. Your schema design needs to give it an honest way out, which we'll get to shortly.

Level 3: validate in your own code, always

Even with a guaranteed shape, parse the response into a typed object with a validation library and enforce your business rules there. In TypeScript that's usually Zod; in Python, Pydantic. The validation layer catches row four of the table, gives you typed data for the rest of your program, and acts as a safety net if the provider's constraint behavior ever changes or you swap to a model without constrained decoding.

Here's an example: extracting a support ticket's key facts from a free-text email.

import { z } from "zod";

export const TicketSchema = z.object({
  category: z.enum(["billing", "bug", "feature_request", "account", "other"]),
  urgency: z.enum(["low", "normal", "high"]),
  // null is an honest answer: the email may not mention an order.
  order_id: z.string().regex(/^ORD-\d{6}$/).nullable(),
  customer_email: z.string().email().nullable(),
  summary: z.string().min(10).max(280),
  refund_requested: z.boolean(),
});

export type Ticket = z.infer<typeof TicketSchema>;

export function parseTicket(raw: string, knownOrderIds: Set<string>) {
  const result = TicketSchema.safeParse(JSON.parse(raw));
  if (!result.success) {
    return { ok: false as const, errors: result.error.issues };
  }

  const ticket = result.data;
  // Business rule the schema can't express: the order must actually exist.
  if (ticket.order_id && !knownOrderIds.has(ticket.order_id)) {
    return {
      ok: false as const,
      errors: [{ path: ["order_id"], message: `Unknown order ${ticket.order_id}` }],
    };
  }
  return { ok: true as const, ticket };
}

Two details matter here. First, the schema mirrors the one you send to the provider, so the constrained output and the validator agree. Many libraries can generate a JSON Schema from a Zod or Pydantic model, which saves you from maintaining two definitions that slowly drift apart. Second, the semantic check (the order must exist) lives in code, because no schema language can know what's in your database.

Designing schemas that models fill in well

Schema design has a bigger effect on output quality than most people expect. A schema isn't only a contract for your parser. It's part of the prompt. Every field name and description is something the model reads and uses to decide what to write. A few principles go a long way.

Prefer enums to free text

If a field has a known set of values, make it an enum. "urgency": "high" is easy to route on; "urgency": "pretty urgent, customer seems upset" is not. Enums also make evaluation trivial, because you can compare exact values.

Always consider adding an escape value like "other" or "unknown". Without one, the model must force every input into one of your categories, and you'll get confident misclassifications instead of honest uncertainty.

Make "not found" a valid answer

Use nullable fields for anything that might not be present in the input, and say so in the field description: "The order ID if one appears in the email, otherwise null. Do not guess." This single change removes a large share of hallucinated values in extraction tasks. You're telling the model that leaving a field empty is correct behavior, not a failure.

Keep it flat and small

Deeply nested structures and huge schemas are harder for models to fill in correctly, and harder for constrained decoding implementations to support. If you need a complex object, consider whether it's really several simpler extractions. Two calls with small schemas are often more accurate than one call with a sprawling one, and easier to debug.

Order fields deliberately

Models generate fields in the order they appear. That means earlier fields can influence later ones, but not the other way round. If you want the model to think before it decides, put a short reasoning or evidence field before the decision field:

{
  "evidence": "Customer writes 'charged twice for March' and attaches two receipts.",
  "category": "billing",
  "refund_requested": true
}

Put the reasoning field last and it becomes a justification written after the fact, which is far less useful. Keep the field short, though; you're asking for a quote or a sentence, not an essay. And if you're using a model with built-in reasoning, you may not need this field at all.

Don't make the model do arithmetic

If you need a total, extract the line items and add them up in code. If you need a date difference, extract both dates. Language models are much better at reading than at calculating, and your code is perfect at calculating. Split the work accordingly.

Give it the IDs, don't ask it to remember them

When the output must reference something in your system, such as a product, a user, or a category ID, include the valid options in the prompt or the schema enum. Asking a model to produce the right internal ID from memory is asking for a hallucination.

Retries that actually converge

When validation fails, the most effective retry is a repair loop: send the model its previous output along with the specific validation errors, and ask for a corrected version. Generic "try again" retries waste money because the model often makes the same mistake. Specific error messages tell it exactly what to fix.

type Attempt = { raw: string; errors?: string };

async function extractTicket(
  email: string,
  knownOrderIds: Set<string>,
  callModel: (email: string, previous?: Attempt) => Promise<{ text: string; stopReason: string }>,
  maxAttempts = 3,
) {
  let previous: Attempt | undefined;

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    const { text, stopReason } = await callModel(email, previous);

    if (stopReason === "max_tokens") {
      // Half an object is never repairable. Raise the limit or shrink the schema.
      throw new Error("Extraction truncated: output hit the token limit");
    }

    const parsed = parseTicket(text, knownOrderIds);
    if (parsed.ok) return parsed.ticket;

    previous = {
      raw: text,
      errors: parsed.errors.map((e) => `${e.path.join(".")}: ${e.message}`).join("\n"),
    };
  }

  // Don't loop forever. Route to a human queue or a safe default.
  return null;
}

Some rules of thumb for retries:

• Cap them. Two or three attempts is plenty. If a case fails three times, it's probably genuinely ambiguous or broken input, and more attempts just burn money.
• Log every failure. Validation failures are a free source of evaluation cases. Each one shows you a weakness in your schema, prompt, or input data.
• Have a fallback. Decide in advance what happens after the last failed attempt: a human review queue, a default category, an error shown to the user. "Throw an exception into the void" is not a plan.
• Libraries can help. Tools like Instructor wrap this validate-and-retry pattern around Pydantic models. They're worth a look if you'd rather not maintain the loop yourself.

Truncation: the bug that looks random

A big share of "intermittent" JSON failures in production are truncation, and they're easy to diagnose once you know to look. The pattern: most requests succeed, but requests with longer inputs, or ones that produce longer lists, occasionally fail with a parse error at the very end of the output.

The fix has three parts. Always check the stop reason before parsing; every major API tells you whether generation ended naturally or hit the limit. Set max_tokens with headroom based on the largest realistic output, not the average. And for outputs that can grow without bound, such as "extract every line item from this invoice", either paginate the extraction or set a schema-level maximum on array length and handle the overflow explicitly.

Function calling vs. structured outputs

You'll often have a choice between two mechanisms. With structured outputs, the model's main response is the JSON object. With tool or function calling, you define a tool whose input schema is the object you want, and the model "calls" it with the extracted data as arguments.

For pure extraction, a response format is usually the cleaner choice: there's no pretend tool, and the intent is obvious to anyone reading the code. Tool calling makes more sense when the model is genuinely choosing between several actions, or when extraction is one step inside an agent that also uses other tools. Either way, turn on strict schema enforcement where your provider supports it. If you're designing a set of tools for an agent, the schema advice above applies just as much; I go deeper in the guide to designing tools for AI agents.

Streaming structured output

If you show structured results in a UI, you may want to stream them as they're generated rather than waiting for the whole object. The challenge is that JSON isn't valid until it's complete. Partial-JSON parsers solve this by parsing the incomplete prefix into a best-effort object (closing open strings and brackets temporarily) so you can render fields as they arrive.

Two cautions. Treat streamed partial objects as provisional display data only; don't act on them until the final, validated object arrives. And remember that fields arrive in schema order, which is another reason to put the fields users care about most near the top.

Testing your extraction pipeline

Structured output is one of the easiest LLM features to evaluate, because correctness is often a simple comparison. Build a dataset of real inputs (emails, documents, messages) with the expected extracted object for each, and measure field-level accuracy: how often is category right, how often is order_id right, how often does the model correctly return null when the field is absent?

Report those numbers per field rather than as one overall score. You'll usually find that most fields are near-perfect and one or two account for most of the errors, which tells you exactly where to improve the schema description or the prompt. If you haven't set up an evaluation harness before, the guide on how to test LLM features with evals covers the process from scratch, and extraction is one of the best places to start.

Make sure the dataset includes the uncomfortable cases: inputs where fields are missing, inputs in other languages, inputs with two conflicting values ("my old email was a@x.com, please use b@y.com"), and inputs that are really long. Those are where extraction pipelines fail.

Structured doesn't mean trusted

One last point that's easy to miss. A perfectly validated object can still carry content an attacker chose. If you extract data from emails, web pages, or uploaded documents, the values in your object came from untrusted text. A summary field can contain instructions aimed at the next model in your pipeline. A url field can point anywhere.

Treat extracted values exactly like user input: escape them before rendering, validate URLs against an allowlist, and don't feed them into another model with tool access without thinking about what they could do there. The article on prompt injection explains why this matters and what to do about it.

Production checklist

• ☐ Schema-constrained decoding (structured outputs or strict tools) is turned on where available
• ☐ The response is parsed and validated in code with Zod, Pydantic, or similar
• ☐ Business rules the schema can't express are checked after parsing
• ☐ The stop reason is checked before parsing; truncation is handled explicitly
• ☐ max_tokens has headroom for the largest realistic output
• ☐ Fields that may be absent are nullable, with descriptions that say "do not guess"
• ☐ Categorical fields use enums with an "other" or "unknown" option
• ☐ Arithmetic and ID lookups happen in code, not in the model
• ☐ Retries send specific validation errors back, are capped, and have a fallback
• ☐ Every validation failure is logged and reviewed as a potential eval case
• ☐ Extracted values are treated as untrusted input downstream

Frequently asked questions

What is the difference between JSON mode and structured outputs?

JSON mode guarantees that the response is syntactically valid JSON, but not that it has the fields or types you need. Structured outputs use schema-constrained decoding so the response must match a specific JSON Schema, including field names, types, required fields, and enums. Structured outputs are the stronger guarantee and the better choice when your provider supports them.

Do structured outputs guarantee correct data?

No. They guarantee the shape of the data, not its accuracy. A response can match your schema perfectly and still contain a wrong date, a made-up ID, or the wrong customer's details. You still need validation in code for business rules, nullable fields so the model can say a value is missing, and evaluations to measure accuracy.

Why does my LLM sometimes return invalid JSON?

The most common causes are relying on prompt instructions alone, so the model sometimes adds prose or markdown around the JSON, and truncation, where the response hits the max_tokens limit mid-object. Use your provider's structured-output feature, check the stop reason before parsing, and give max_tokens enough headroom for the longest realistic output.

Should I use function calling or structured outputs for data extraction?

For straightforward extraction, a structured response format is usually simpler and clearer. Function calling fits better when the model is choosing between several actions or when extraction happens inside an agent that also uses other tools. In both cases, enable strict schema enforcement if your provider offers it.

Can I get structured outputs from open-source models?

Yes. Libraries such as Outlines and grammar-based sampling in llama.cpp constrain open models to a JSON Schema or grammar during generation, giving similar guarantees to hosted structured-output features. Validate the result in code either way.

The takeaway

Getting JSON out of a language model used to be a game of prompt tricks and regex cleanup. It doesn't have to be anymore. Let the decoder enforce the shape, let your validator enforce the rules, and let your schema give the model an honest way to say "not found". Check for truncation, retry with real error messages, and log everything that fails.

Do that and the 3 a.m. parse error goes away. Not because the model became more polite, but because you stopped needing it to be.