How Do We Find Out How Good AI Is Across the Task Taxonomy?

The Legal Help Task Taxonomy names 52 tasks, and the cohort has taken one of them through a full evaluation that another team could repeat. Other teams are measuring too, on their own rubrics and their own cases, so their results sit beside each other without being comparable. This report sets out what common measurement would take, the choices for getting there, the partners the work needs, and the question of how the issue-area agendas and the task taxonomy fit together.

About the cohort. We ran a cohort of legal help website teams for 18 months, working together on common approaches to AI for legal help online. Where these reports say “the cohort,” that is who they mean.

The Legal Help Task Taxonomy names 52 tasks across 7 areas of legal help work. The cohort measured legal question-answering in full, and even that evidence is early and rests on synthetic questions and five rounds of testing in May. The classification task is partly measured, against a 197-case ground-truth set. Four more tasks the cohort can speak to from building the tools, deploying some of them, and watching where they work, which is real knowledge and is not a scored evaluation. Vendors and legal aid teams are running their own evaluations on other tasks, and we have not found published results that use a shared rubric, a shared case set, or a shared way of reporting, so there is no way to put any of them side by side. That is the true state of the field's read on AI performance, and this report is about how the field changes it.

What knowing one task actually takes

For legal question-answering the cohort needed four things before it could measure anything. It needed a clear task definition, which the taxonomy supplies. It needed a dataset of test cases, built synthetic so it could be shared. It needed a rubric for what a good answer looks like, written so a person and a machine can both apply it. And it needed a protocol for running models against the dataset and scoring them, which was promptfoo plus a deterministic verifier. Put those four together and you have an evidence pack for a task. JusticeBench is built to hold exactly this, because each task page can carry the task's definition, its datasets, its rubric, and its results. The inventory this report is about is one of these packs per task, kept current.

The number is never just a number

A single score for a task hides the thing the grounding report found, which is that AI performance depends on the content the model is grounded in and the jurisdiction it is answering for. A model may explain a standard federal form well and a county's local cover sheet badly. It may answer a housing question well in a state with a strong tagged knowledge base and badly in a state without one. So the real unit is a task, in an issue area, in a jurisdiction, against a particular model. Scoring the task on its own averages across all of it. That is a large matrix, and no one is going to fill all of it.

This is where the field's two ways of slicing the work meet. The Legal Help Commons runs issue-area agendas: housing, debt, reentry, public benefits. Those are vertical, and they ask what it takes to deliver good help in one domain from start to finish. The task taxonomy slices the other way. It names the task divorced from the issue area, so that document explanation or deadline calculation can be looked at across every domain at once. The two are complements, and the cleanest design connects them on purpose. The issue-area cohorts are where the cells of the matrix get filled, because a housing cohort building and testing its tools is already measuring the housing instances of classification, routing, and document explanation. The task taxonomy on JusticeBench is where those results get gathered by task, so the field can see how document explanation does across housing, debt, and family side by side. The vertical work produces the evidence. The horizontal taxonomy organizes it.

Who does the measuring

There are three ways to get the work done, and they are not exclusive.

The first is a central benchmark team. A national group runs evaluations across tasks and publishes them. This gives consistency and comparability, because the same hands run the same protocol every time. It also asks one team to keep up with 52 tasks across issue areas and jurisdictions while the models change every month, which no one team can do.

The second is distributed contribution. Individual teams run their own evaluations on the tasks they care about, using the shared protocol and rubrics, and send their results back to a common registry. This is the model the measurement report is built for, and it is the only one that scales, because the work spreads across everyone who has a stake in the answer. Its cost is comparability, because results are only poolable if teams use the same rubrics and report the same way, which means the standards have to come first.

The third is task stewardship. Each task, or a cluster of related tasks, gets an owner: an organization or a working group that builds the dataset, maintains the rubric, runs the benchmark, and keeps it current. This is how open standards and open-source projects survive. It gives depth and accountability for each task. Its cost is recruitment and sustainability, because a steward role needs a clear reason to exist and someone to fund it.

The recommendation is a hybrid. Hold the shared protocols and rubrics centrally, so results stay comparable. Assign task stewards for the priority tasks, so someone is accountable for depth. Use distributed contribution for everything else, so the work scales past what any one team could do alone. JusticeBench is the registry that ties the three together.

How often

A measurement is a snapshot of a model that will be replaced. An evaluation run in May can be stale by autumn. So the field has to choose a cadence, and there are two honest options that pair well.

The first is a yearly read. Once a year the field publishes a state of AI on legal help tasks: where each task sits, what improved, what slipped, what got measured for the first time. This is feasible, it is legible to leaders and funders, and it gives the field a rhythm to work to. The second is event-triggered re-runs on the priority tasks. When a major model is released, the stewards of the highest-stakes tasks re-run their benchmarks, because those are the tasks where a regression reaches a person with a real problem. The two together give a steady annual picture plus fast checks where the stakes are highest, without pretending the field can keep a live leaderboard running for every task in the taxonomy.

The partners the work needs

The measurement does not happen without people who are not only engineers. Each task needs subject-matter experts to write and validate its rubric and to do blind review, because a rubric for whether eviction advice is sound is a lawyer's judgment before it is a metric. The outcomes question, which is whether a better answer actually produces a better result for the person, needs researchers with causal methods, and there are credible homes for that in the Rhode Center, RAND's civil justice group, and the American Bar Foundation. Running evaluations across models needs access to those models, which is a place for provider relationships held so that no single provider sets the terms. And the whole inventory needs a neutral registry that belongs to the field rather than to a vendor, which is what JusticeBench is for. None of these is a solo job, and that is the case the national-leadership report makes for a tier above any single state.

Where to start

The field cannot open every task at once, so the choice is which few to take to full depth first. The sensible answer is to start where the issue-area agendas are already working, so the measurement rides on building that is already funded and underway. The housing, debt, reentry, and public benefits cohorts each touch a handful of the high-frequency tasks, and those instances are the first cells to fill. Take those to full depth, publish the protocol and the rubrics so the next team can extend them, and let teams add the tasks they care about through their own promptfoo runs. The tasks nobody has measured in a comparable way are the invitation, and they are the reason to set up shared measurement now rather than after another year of tools shipping on evidence no one else can read.

What this report is

This is a set of choices, not a plan the field has agreed to. Whether to fund task stewards, whether to commit to a yearly read, and how tightly to wire the issue-area cohorts to the task inventory are decisions for the field and its funders to make together. What the cohort can say with confidence is narrower and still useful. The method exists. The registry exists. A task has been measured end to end against a protocol another team could run, which is the proof that the rest can be. The work now is agreeing on how the field measures, then deciding who measures what, how often, and with whom.