Benchmarks

This page introduces Curator's Benchmarks feature for users, automation analysts, and operations leaders. The goal is to explain what the feature measures, when it is worth using, how it reaches its result, and how to interpret the numbers with confidence.

Overview

An interpreter is a reusable artificial intelligence recipe, made of chained steps: each step has an instruction, an artificial intelligence model, and receives the input information plus the answers from previous steps. The interpreter is what reads an official letter, extracts the authority, classifies the type of document, or summarizes a contract inside your automations.

The Benchmark answers a practical question: which artificial intelligence model should I use in this interpreter? To answer that, Curator takes a set of real cases whose correct answer you already know, runs exactly the same cases on several different models, and shows three numbers side by side:

Metric	What it tells you
Accuracy	What percentage of answers matched the correct answer you registered.
Cost	How much that model would spend, in dollars, to process that volume.
Speed	How long the model took to process everything.

At the end, Curator builds a ranking and points out the best model. You stop choosing a model by opinion or hype and start choosing based on evidence measured on your own document base.

How the feature works, in five stages

1. Build the batch       2. Choose the           3. Curator runs each case,
   real cases       -->     AI models       -->     step by step, on each model
   + reference answers                                         |
                                                               v
5. Ranking, charts    <--  4. Each answer obtained is compared with the reference
   and Excel report

1. Build the batch. You register the cases, which are real inputs from your daily routine, and the reference answers, which are the correct expected answers for each step.
2. Choose the models. Select two, three, or as many models as you want to compare. All of them receive exactly the same batch and the same instructions.
3. Execution. Curator processes the cases one by one. Inside each case, the steps run in a chain, from first to last. You watch live which case and which step is being processed.
4. Grading. Each answer obtained is compared with the reference answer. Grading can be automatic or manual, and you choose that when creating the batch.
5. Result. A model ranking, charts for accuracy, cost, and speed, and an Excel report with case-by-case detail.

Glossary

These are the terms used on the product screens. It is worth knowing them before starting, because they appear in every menu and report.

Term	What it means
Interpreter	The artificial intelligence recipe to be evaluated, made of chained steps.
Step	One stage of the recipe, with its own instruction and its own answer.
Batch	The set of cases used in the evaluation, with the reference answer for each one.
Case	One real input, for example a specific official letter or a specific contract.
Reference answer	The correct answer expected at a given step of that case.
Obtained	The answer the artificial intelligence actually returned at that step.
Job	One benchmark execution, with date, models used, and result.
Accuracy	The percentage of correct answers.
Multi-model comparison	A job that runs the same batch on several models at the same time.

When to use Benchmarks

The feature is useful whenever there is a decision about an artificial intelligence model to be made with money or quality at stake. The most common scenarios are:

• Choosing the model before going to production. Before putting an interpreter to work on thousands of documents, you find out with thirty or fifty cases which model delivers the accuracy you need.
• Cutting cost without losing quality. It is common for a cheaper model to be nearly as accurate as the expensive one on a specific task. The benchmark shows that difference as a number, which justifies the switch.
• Validating an instruction change. You adjust the text of a step and run the same batch again. If accuracy went up, the change was good. If it dropped, you revert based on data instead of impressions.
• Auditing quality periodically. Models are updated by their providers over time. Running the same batch each quarter shows whether the behavior remains the same.
• Accountability. The Excel report serves as formal evidence that the model choice was technical and measured.

When the feature is not the right instrument

Building the batch

A batch is the set of cases that will be used to evaluate the interpreter. Each case represents a real situation with two contents: the inputs, which are the information the interpreter needs to receive, and the reference answer, which is the correct answer expected at each step.

The same batch can be reused as many times as you want, with different models or on different dates. It is precisely this repeatability that allows comparing results over time. A well-built batch is what separates a trustworthy benchmark from a meaningless number.

How many cases to use

There is no mandatory minimum in the product: the benchmark runs with a single case. But one case says almost nothing, because the result will always be zero or one hundred percent. As a practical reference:

Batch size	What it is good for
Fewer than 10 cases	Only good for testing whether the interpreter works, not for deciding on a model.
20 to 50 cases	Usually enough to compare models on a classification or extraction task with reasonable confidence.
More than 100 cases	Increases confidence, but also increases the time and cost of the comparison, since each case is processed once per model.

The creation wizard, in four stages

Stage 1 — Batch name. Choose a name that recalls the scenario evaluated, for example "Letters Q3 2026" or "Authority extraction". This name appears in the history and in reports, and it is how you will recognize the batch months later.

Stage 2 — How to grade. Here you decide who judges whether the answer is right.

Mode	How it works	When to use
Automatic	Curator compares the obtained answer with the reference on its own.	Classification, field extraction, objective answers.
Manual review	A person marks each step as correct or incorrect after execution.	Summaries, free-form texts, answers with many valid forms.

With automatic grading the result is ready at the end of the execution. With manual review the job ends in the "Awaiting review" state and waits for you to judge each step.

Stage 3 — Which instructions to use. An interpreter evolves: you adjust its instructions over time. That is why the batch asks which version it should use.

Option	Behavior	When to choose it
Use current instructions	On each execution the batch uses the instructions the interpreter has at that moment.	When the question is "which model is best for my interpreter as it stands today". If you improve the instruction tomorrow, the next run already reflects it.
Freeze a copy now	The batch keeps a copy of the current instructions and always uses that copy, even if the interpreter changes later.	When the question is "does this specific scenario still behave the same". Freezing isolates the instruction variable: if the result changes, the cause was the model, not the text.

Stage 4 — Evaluation base. In this stage you import the spreadsheet with the cases. You can also skip it and register the cases later, directly in the batch editor.

The case spreadsheet

The safest path is to use the button that generates the Excel template. Curator creates a file already containing the right columns for that specific interpreter, because it knows which inputs that interpreter requires and how many steps it has. The file comes with two sheets: cases, with the header ready and two example rows for you to copy the pattern, and help, with a table explaining what each column means.

Column structure

The spreadsheet has three types of column.

Type	Header	Required
Case name	label	No. If left empty, Curator numbers them as "Case 1", "Case 2".
Inputs	One per interpreter input variable	Yes, for variables marked as required.
Reference answer	One per interpreter step	Recommended with automatic grading.

In the template generated by the product, each reference column takes the name of the corresponding step. If the interpreter has the steps "Espécie" and "Autoridade", the reference columns are named exactly that. This keeps the spreadsheet readable for whoever fills it in. If two steps share the same name, Curator differentiates them by adding a number in parentheses, for example "Extração (2)".

Example spreadsheet for an interpreter with one input named oficio and two steps named Especie and Autoridade:

label            | oficio                  | Especie  | Autoridade
Oficio 1024      | (texto completo aqui)   | ACORDAO  | TRT 2a Regiao
Oficio 1025      | (texto completo aqui)   | DESPACHO | TJSP

Accepted alternative headers

If you already have a spreadsheet, you do not need to redo it. Curator automatically recognizes several ways of naming the columns:

Field	Recognized headers
Case name	label, caso, nome, case, name
Generic input	insumo, input, input_text, texto
Reference by position	expected_1, esperado_1, gabarito_1, expected_step_1, esperado_passo_1, gabarito_passo_1
Reference by step name	The step name itself, or expected_<nome>, esperado_<nome>, gabarito_<nome>

Any column that does not fit any of these patterns is treated as an additional input variable.

Accepted file formats

Curator reads .xlsx, .xlsm, .xls, and .csv files. For CSV, the product detects on its own whether the separator is a semicolon, a comma, or a tab, and also tries the most common encodings, including UTF-8 and Latin-1. Even so, Excel remains the most predictable option, especially when the case texts contain line breaks or accented characters. Completely empty rows are ignored during reading.

Mapping columns and validating the import

After selecting the file, Curator opens the mapping screen. It shows three blocks, and in each one you indicate which column of your spreadsheet corresponds to which field of the case:

• Case identification: which column holds the case name.
• Interpreter input variables: which column feeds each input.
• Reference answers per step: which column holds the correct answer for each step.

Suggestions come pre-filled automatically, based on column names and step names. You only adjust what differs. A preview with the first rows shows the result of the mapping before you confirm.

Two validation rules on this screen:

• You must link at least one input variable to a column. Without input there is nothing to process.
• If any required variable is left without a column, Curator warns which ones and asks whether you want to import anyway. If you proceed, those cases will probably fail or run without context.

The validation report

Once the import finishes, Curator analyses the cases against the interpreter contract and shows a summary: how many cases were imported, how many have at least one reference answer filled in, how many are missing a required input, and how many reference cells were left empty. Three warnings deserve special attention:

Warning	What it means	What to do
Empty batch	No case was read from the file.	Check that the header is on the first row and that the columns were mapped.
No case has a reference answer	With automatic grading the accuracy will be zero.	Fill in the reference columns before running.
Empty reference cells	Some steps will not be evaluated.	Acceptable if intentional. See the rule in the next section.

The most important rule about reference answers

This rule has a practical consequence worth understanding well. Imagine a batch with 50 cases and 3 steps per case, which would give 150 evaluable steps. If you only filled in the reference answer for the first step, the benchmark will calculate accuracy over 50 steps, not 150. The percentage remains correct, but it describes only the first step of the interpreter, not the whole recipe.

The behavior is also useful when intentional: if a step of the interpreter produces free-form text that you do not want graded automatically, just leave that step's reference answer blank and it drops out of the calculation without harming the result.

Editing cases later

The batch is not frozen after creation. In the editor you can add cases manually, fix a reference answer that was wrong, import a new spreadsheet, or remove cases that no longer make sense. Each case card shows how many inputs and how many reference answers are filled in, which helps you quickly find incomplete cases.

Running and comparing

Three conditions must be met before starting:

1. The interpreter must have at least one step. With no step there is nothing to run.
2. The batch must have at least one case. If it is empty, Curator warns you and does not start.
3. Each chosen model must have its access key configured. Models whose provider has not been configured yet appear blocked in the selection list.

Choosing the models

On the selection screen you check as many models as you want to compare. All of them get exactly the same treatment, and that is what makes the comparison fair: the same cases, in the same order, and the same instructions, according to the mode chosen when the batch was created.

How the execution happens

Job start
   |
   |-- Model A --> Case 1 --> Case 2 --> Case 3
   |-- Model B --> Case 1 --> Case 2 --> Case 3
   |-- Model C --> Case 1 --> Case 2 --> Case 3

Inside each case:  Step 1 --> Step 2 --> Step 3   (chained)

• Models run in parallel. Each selected model has its own execution lane. Model B does not wait for model A to finish.
• Within a model, cases run one at a time. This keeps consumption predictable and avoids exceeding the provider's request limit.
• Within a case, steps run in a chain. One step's answer feeds the next step, exactly as the interpreter works in production. If a step fails, that case's chain is interrupted, because the following steps depended on that answer.

A useful detail: even when you start with a single model, Curator already treats the execution as a comparison. That is deliberate, because it lets you add other models to the same job later without redoing anything.

Following live, pausing and stopping

During execution the screen shows progress at three levels.

Area	What it shows
Model list	All the models in the job and which case each one is on.
Case being processed	The current case and the input being processed.
Current step	The step the artificial intelligence is answering right now.

Clicking a model in the list makes the detail area follow that model. As each step finishes, the result appears in that case's step log, so you can notice early if a model is doing poorly, without waiting for the end. You can also leave the screen: the benchmark keeps running in the background and progress stays visible on the job card in the Benchmarks screen.

Pausing and stopping

The pause and stop buttons apply to the model currently selected in the list, not to the whole job. This is useful when a specific model is taking too long or costing more than you expected: you interrupt only that one and let the others finish.

Action	Effect
Pause	The model stops after the current step and the queue waits.
Resume	The model continues from where it stopped, without redoing what was already done.
Stop	The model ends after the current case or step, and partial results are saved.

The most important point about stopping: nothing is lost. The cases already processed remain recorded, with their answers, their grading, and their cost. The job is marked as interrupted and you can analyse what was already measured.

Adding models to an existing job

You can add more models to a comparison, even while it is running. The new model joins in parallel and starts processing the same batch from the beginning, without disturbing those already running.

This feature solves a very common situation: you start comparing two models, see the partial result, and remember a third one you would like to test. Instead of discarding the work and starting over, you just add it. In the end, all three appear in the same ranking, measured over exactly the same cases.

Job states and execution cost

Each job carries a state that summarizes where it stands.

State	Meaning
Running	Still processing cases.
Paused	Temporarily interrupted, ready to resume.
Completed	Finished normally and the result is available.
Awaiting review	Finished, but the batch uses manual grading and awaits your judgement.
Interrupted	Stopped by you, with partial results preserved.
Failed	Ended due to an error, for example an invalid access key or provider unavailability.

About the "Failed" state, a distinction is worth making: if only some steps errored out, the job finishes normally and the errors appear in the detail. The job is only marked as failed when the problem prevented execution in general, such as a rejected key, in which case no step could be answered.

When a model fails inside a comparison, the others carry on. The failed model appears in the ranking marked as such, always at the bottom of the list, so it does not take a good position without having actually been measured.

Execution cost

Each execution consumes real credit at the artificial intelligence provider. The math is straightforward:

calls = number of cases x number of steps x number of models

A batch of 50 cases with 3 steps comparing 4 models generates 600 calls. It is not a frightening number, but it justifies the earlier recommendation of testing with a small batch before running the full one.

The three metrics, in detail

Every benchmark job answers three questions about each model: how accurate, how expensive, and how slow.

Accuracy

It is the percentage of correct answers. The calculation is a simple division:

accuracy = correct steps / steps with a reference answer x 100

Two remarks about the denominator of that calculation, because that is where the most common confusion lives:

• Steps without a reference answer are left out. They count neither as correct nor as wrong. If you only filled in the reference for one of the three steps, the accuracy describes only that step.
• Steps that hit an execution error do not score automatically either. A communication error with the provider does not become a "wrong answer" without someone confirming it.

Cost

It is the estimated sum, in dollars, of everything that model consumed in the job: the text sent in each step plus the text received back, multiplied by that model's rates.

Duration

It is the total time the model took to process the whole batch, measured from the start to the end of the execution. This number is precise.

The time shown per step, however, is an approximation: Curator divides the case duration equally among that case's steps. It gives a sense of magnitude, but do not use that value to claim that a specific step is the bottleneck.

Two cautions when reading the duration. It is influenced by your connection and by how busy the provider is, so a small difference between two models may be just noise. And it grows with batch size, so it only makes sense to compare durations measured in the same job.

How automatic grading compares texts

Automatic comparison does not require the answer to be identical character by character. It tolerates differences that do not change the content:

• Upper and lower case are ignored: "Acordao" and "ACORDAO" match.
• Repeated spaces and tabs collapse into a single space.
• Leading and trailing spaces are discarded.
• Excess line breaks are reduced.

In addition, Curator understands answers structured as JSON, which is the format interpreters often return:

Situation	How it is compared
Reference and answer are both JSON	Field by field, ignoring field order.
The answer is JSON and the reference is plain text	Curator looks for the reference text among the JSON values.
Both are plain text	Direct comparison, with the tolerance described above.

The middle case is the one that shows up most in practice. You write only ACORDAO in the reference, and the artificial intelligence answers {"especie_publicacao": "ACORDAO"}. Curator recognizes the match and marks it as correct.

The ranking and the charts

Models are ordered by three criteria, applied in this order:

1. Highest accuracy first.
2. In case of a tie, lowest cost.
3. If the tie persists, shortest duration.

Models that failed always go to the bottom of the list, even if they happened to have seemingly good numbers. A model that was not truly measured cannot take the top spot.

The logic of this order is deliberate: quality first, savings second. The cheapest model is useless if it gets things wrong. But between two models that are equally accurate, the cheaper one wins.

The four charts

The result brings four visualizations. All of them use the same color per model, so you can follow the same model across different charts.

Chart	What it shows	How to read it
Accuracy	Percentage of correct answers per model.	Taller bar is better, scale from 0 to 100%.
Cost	Estimated value per model.	Shorter bar is better, value in dollars.
Speed	Total time per model.	Shorter bar is better, time in seconds or minutes.
Grouped comparison	The three metrics side by side.	Taller bar is always better. See the explanation below.

The grouped chart deserves a separate explanation because it inverts an intuition. In it, the three metrics appear together and a taller bar means better in all three, including cost and speed. That is possible because the bar height is not the raw value: it is a comparative index from 0 to 100, in which the best model for that metric gets 100 and the others get a proportional score.

The real value remains visible: it is written on top of each bar, with the percentage, the dollar value, or the time. So the height lets you compare at a glance who is doing better, and the label tells you the exact number. When a model fails, its bar appears in an alert color and the label shows a dash, making it clear that no measurement happened there.

Manual review

When the batch was created with manual grading, the job ends in the "Awaiting review" state and the accuracy is only defined after you judge the steps.

The review screen presents one step at a time, showing side by side what was expected and what was obtained, along with the case input. You mark it correct or incorrect, and you can record the reason when marking it incorrect. That reason is valuable later: it turns a number into a diagnosis, showing whether errors cluster in one type of document or in a specific step.

Two rules govern this stage:

• Human marking always prevails over the automatic one. If Curator marked it wrong and you mark it right, your judgement wins.
• Reopening the job does not erase your work. The automatic evaluation can be recalculated, but manual markings remain.

It is worth noting that automatic grading also runs on batches marked as manual. It works as a pre-marking that speeds up the review: obviously correct steps come already flagged and you focus on the doubtful ones.

The Excel report and the history

The export generates a file with three sheets.

Sheet	Content
Summary	Batch, interpreter, date, best model, and total costs.
Ranking	One row per model, with position, accuracy, correct steps, cost, duration, and state.
Details	One row per case, step, and model, with the expected value, the obtained value, the automatic grading, the manual grading, and the final verdict.

The Details sheet is the most useful day to day. With it in a pivot table you answer questions the screen does not answer directly, such as which step concentrates the errors, which cases no model got right, or whether errors cluster in a specific type of document.

The warning that the cost is an estimate also appears inside the spreadsheet, so the file remains correct even when it circulates outside Curator.

Job history

Curator keeps previous jobs so you can compare results over time. The history holds up to 500 jobs and up to 200 comparisons; the oldest are discarded as new ones come in.

A five-question reading script

To wrap up, a practical sequence to follow when a fresh result comes out:

1. Was the base sufficient? Look at the count of evaluated steps, not just the percentage. A result over 12 steps does not support a decision.
2. Did any model fail? Failed models were not measured and do not enter the comparison.
3. Is the accuracy difference relevant? Two percentage points between models are usually normal variation. Ten points are a clear signal.
4. Is the cost difference worth it? Compare the accuracy gain with the cost increase and think about your operation's real monthly volume.
5. Where are the errors? Open the Details sheet. If errors concentrate in one step, the path is improving that instruction, not switching models.