Skip to content
JsonEval
FRExplore the demo
JSON QUALITY WORKBENCH

Compare AI models.
Field by field.

Evaluate their JSON outputs against the same documents and references.
Find the best model for your extraction.

01 / Define the use case
Try an example

Prepare without an account. You confirm the cost before any evaluation by a model.

PDF & scansEmails & textMultiple LLMsOne shared reference
YOUR USE CASE. YOUR FIELDS. YOUR CRITERIA.

Need more reliable extraction?
Start here.

One evaluation workbench.
Test datasets built around your needs.

OUR SPECIALTY

Invoices → JSON

Totals, VAT and every line item. Compare difficult scans, too.

invoice_numbertaxtotalitems[].quantity
Discover OCR eval
USE CASE / 02

Emails & agents

The right intent, order and deadline.

customerorder_idintentdue_date
Prepare this use case
USE CASE / 03

Receipts & expenses

Merchant, currency, taxes and amount. Small details matter, too.

merchantdatecurrencytotal
Prepare this use case
USE CASE / 04

Contracts & documents

Parties, dates and clauses. Separate what is written from what is invented.

parties[]effective_datejurisdictionrenewal_clause
Prepare this use case
USE CASE / 05

Orders & deliveries

References, quantities and addresses. Find matching lines regardless of their order.

order_iditems[].skuitems[].quantitydelivery_address
Prepare this use case
USE CASE / 06

Product sheets & catalogues

Prices, units, features and variants. A catalogue deserves reliable data.

skupricedimensionsvariants[]
Prepare this use case

Import a representative dataset and its expected answers. Evaluate emails and product sheets from text exports, PDFs or images.

THE EVALUATION WORKBENCH

See the big picture. Zoom in on what matters.

Open this benchmark
jsoneval / projects / demo-invoices / benchmarkSTATIC DEMO
JsonEval demo screenshot: JSON extraction scores by field, document and model
A fictional dataset. Multiple models. Errors made visible.Illustrative scores · no real model performance claims
VALID ≠ CORRECT

Your prompt changed.
What actually changed?

A slightly wrong amount. A poorly normalized date. A missing line item. One overall score is not enough: JsonEval shows where your extraction improves and where it regresses.

{ } Expected structure↔ Reference answers▦ Results by field
YOUR FIRST BENCHMARK, WITHOUT BUILDING A TEST HARNESS

Your first project.
Three steps. One guided flow.

You bring the context.
JsonEval organizes the comparison.

  1. 01

    Your prompt. Your JSON.

    Create a project, paste your prompt and an example of the expected JSON structure. AI can also propose a schema for you to adjust.

    prompt.txt + example.json
  2. 02

    Documents and their answers.

    Import a first sample of PDFs, images or text. Add expected values, or have a model prefill them and review them.

    documents + reference values
  3. 03

    Compare. Correct. Iterate.

    Choose models, confirm the cost estimate, then explore scores. Change a prompt and see exactly what evolves.

    benchmark → next iteration
✓

Start small. A first sample helps you check the workflow. Expand your dataset to measure quality on representative cases.

Get started
AVAILABLE IN THE WEB APP TODAY

Everything you need to
evaluate what your JSON says.

From preparing your dataset
to your next prompt version.

Fictional illustrations and data. Scores do not represent any real model performance.

01

Models and prompts, side by side

Compare multiple LLMs and prompt versions on the same documents. Find improvements and regressions, field by field.

02

Your JSON, your rules

Start with a JSON example, import it or use the visual builder. Configure a metric for every field, including nested objects and collections.

03

7 scoring metrics

Exact match, case handling, text similarity, meaning, numeric tolerance, dates and set membership. Continuous scores between 0 and 1.

04

A dataset that fits your needs

Import PDFs, images or text. Organize documents into groups and select those to include in the comparison.

05

Reference answers

Import or enter expected values, or prefill them with a model. Review this gold: it is what gives scores their meaning.

06

Correctly matched collections

Compare table rows even when their order changes. Identifiers and fallback text help find corresponding items.

07

An error and its context

Move from the heatmap to the document, prediction and expected value. The Reviewer helps correct and resolve uncertain references.

08

Versioned prompts

Keep your iterations, compare results and select an active version. One shared prompt for every model in a comparison.

09

Quality, cost and latency

See what a point of quality costs. Track extraction and judge costs, cost per document, latency and paid failures.

10

Rescore without extracting everything again

JSON predictions are stored. Reevaluate them after correcting gold or rules; a semantic judge may still incur a charge.

11

Comparable variants

Compare a native PDF with its pages rendered as images, or select a provider. Each variant appears separately in the benchmark.

12

A workbench for your team

Share projects in a workspace, discuss values and trace corrections. Keep a shared reference.

THE RIGHT METRIC FOR EACH FIELD
exactcase_insensitivefuzzysemanticnumber_tolerancedate_matchset_member
A SCORE IS THE START OF THE INVESTIGATION

From the result.
To the reason.

The prediction, the reference and the document.
The same context for your whole team.

Fictional invoice and reference answers in the Documents tab of the JsonEval demo
01 / DOCUMENTS & GOLD

References you can verify.

Read the expected answer alongside its document. Review, correct and discuss uncertain values.

Explore documents
Prompt versions and field scores in the static JsonEval demo
02 / PROMPTS & ITERATIONS

Keep track of what works.

Turn an instruction into a version. Compare its results before making it your active prompt.

Explore prompts
NEXT, IN YOUR DEVELOPMENT ENVIRONMENT

The web today.
Your terminal and assistant tomorrow.

CLI & AI PLUGIN · COMING SOON

A CLI for developers.

Run and inspect evaluations from your terminal, keep project context and prepare integration into your development workflows.

PROPOSED INTERFACE · NOT AVAILABLE
$ jsoneval evaluate
↳ project · dataset · prompt · models
↳ field scores · cost · regressions
Planned · commands and CI integration to be defined

A plugin for your AI tools.

Give your coding assistant the context of your evaluations: prompts, regressing fields and results to explore together.

PROPOSED WORKFLOW · NOT AVAILABLE

“Which fields regress
with my new prompt?”

Project context → evaluation results
Planned · supported tools and protocol to be defined

The web app is available. The CLI, AI plugin and CI integration are planned and not yet released.

THE PRODUCT KEEPS MOVING

Real improvements.
Every iteration.

Full changelog
Experience

An evaluation workbench built for developers.

A black-and-white identity, a need expressed in a few words and a demo to explore before creating an account.

See what’s new Benchmark

Compare how you send documents, too.

Native PDFs or image pages, a selected provider: model variants enrich the comparison. Cost analytics become more precise.

See what’s new Data

Fewer clicks to improve your references.

Find values in documents and correct repeated ones in the Reviewer. References remain at the heart of evaluation.

See what’s new
ANSWERS BEFORE YOUR FIRST RUN

The right questions.

Looking for a tool to evaluate the JSON outputs of your AI requests? Here is what JsonEval covers and what is coming next.

What is JsonEval for?

JsonEval evaluates JSON answers generated by AI models. It benchmarks structured extraction: you provide a prompt, schema, documents and reference answers, then compare LLMs and prompt versions field by field.

How is it different from a JSON Schema validator?

A validator checks structure and types. Yet perfectly valid JSON may contain a wrong amount, a missing line or an invented value. JsonEval compares content with expected answers and assigns field scores. It complements structural validation.

Do I need to write code to get started?

No. The web app guides project setup: prompt, schema from a JSON example, documents, reference values and a first comparison. The visual builder and assisted gold preparation help you build the evaluation dataset.

Can I compare models from different providers?

Yes. JsonEval uses the OpenRouter catalogue. Compare multiple models on the same dataset and prompt, along with PDF input or provider variants. Availability depends on the models and providers involved.

What are gold and field scores?

Gold is the expected answer: your ground truth. Each extracted field is compared with this reference using a suitable metric: exact match, text, meaning, numbers, dates or sets. Scores are continuous between 0 and 1; they do not replace human review of gold.

Are the CLI and AI tools plugin available?

The CLI and AI tools plugin are planned. They will bring evaluation to the terminal and coding assistants. The web application is available today; no installable command or CI integration has been released yet.

Does the demo call AI or use my data?

No. The public demo only uses fictional documents, answers, scores and costs prepared in advance. It requires no account or API key and runs no inference. Interactions stay in your browser.

How are evaluation costs handled?

The app shows an estimate before actions that call an LLM. Analytics separate extraction and judge costs and identify estimates. Prepaid credits, Stripe and connecting a personal key in JsonEval are still planned; the demo is not a pricing offer.

YOUR NEXT ITERATION STARTS HERE

Less “looks good”.
More “here are the results”.

Explore a fictional project. Then switch to your own documents.

Try the demo Create my project
No-account demo · Fictional data · No inference