When Grounding Helps a Legal Help Answer, and When It Hurts

Over five rounds in May 2026 the cohort tested whether grounding an AI answer in state legal help content makes the answer better. The early rounds said yes, and the later rounds said it depends on whether the content actually covers the question that was asked. One round showed that the scoring rubric, rather than the architecture, was deciding what we could see. This is the full record of those rounds, including the ones that reversed an earlier result, and an account of how much weight each finding can carry.

About the cohort. We ran a cohort of legal help website teams for 18 months, working together on common approaches to AI for legal help online. Where these reports say “the cohort,” that is who they mean.

The short version of this work, with the headline finding on its own →

A person asks an AI tool a legal question. Three things can happen with a state's legal help content. The model answers from its training alone. The model is handed a list of in-state articles, with titles, links, and short descriptions, and answers with those in front of it. Or the model is handed the full text of two or three in-state articles and answers from that. The cohort calls these the bare condition, directory grounding, and full-content grounding, and every round compared some combination of them on the same questions.

Three models were tested across the rounds, chosen to cover the range a team might actually deploy: Claude Sonnet 4.5, Gemini 2.5 Pro, and GPT-4o-mini. The questions were synthetic, written from a representative sample of legal issues and phrased from a database of live chat queries that cohort partners shared. Each round cost under twenty-five dollars and ran in under ninety minutes.

The rounds in order

The first round put twenty-two Illinois housing questions through the bare condition and directory grounding. Directory grounding won on every model. The second round scaled the same comparison to forty-eight questions across six states and seven issue areas, and the result held everywhere, which is what turned a single-state result into something the cohort could rely on.

A separate single-question run on a person with an eviction hearing the next morning shows what the composite hides. Two of the three models failed that question without grounding and all three passed with it. One question proves nothing on its own, and it is the kind of question where failing matters most.

The third round introduced full-content grounding on ten high-stakes Illinois housing questions, and the answer came back as a tie. Reading the answers side by side told a different story from the scores. The full-content answers quoted the article text instead of paraphrasing it, said plainly when the articles did not cover something, and gave form numbers and deadline counts. The rubric in use measured citation accuracy and general quality, so none of that was visible to it.

The fourth round rescored the same ninety answers with a sharpened rubric that added four things: whether the answer is specifically actionable, whether its factual claims are grounded in the source, whether it admits what it does not know, and whether its quotation marks contain text that is actually in the article. Under that rubric full-content grounding beat directory grounding on all three models. The tie in the previous round was a measurement result rather than an architectural one.

The fifth round took forty high-risk questions across six states and sixteen scenarios, drawn from a set of queries that subject-matter experts had flagged, and scored them with a judge that also checks whether the answer asks the clarifying questions the expert said the scenario requires. The clean result from the fourth round did not survive it.

The judge passed almost everything

Each answer is scored by a model acting as a judge and by a deterministic verifier that checks the links, organizations, and phone numbers against a list of known-good resources. Across the second round the judge passed essentially every answer under every condition, and all of the separation between bare and grounded answers came from the verifier.

This has been the same in every round the cohort has run, which is now four separate runs across three models. A model judge reading for prose quality cannot tell a well-written answer with working links from a well-written answer with none, because both read well. Any team evaluating a legal AI tool needs a deterministic check alongside whatever judge it uses, or it will see a flat line where the differences are.

What the verifier caught across the two hundred and eighty-eight answers in that round is a short list, and the two most dangerous categories did not appear at all.

The links that no longer load are worth separating from the rest. Those are correctly named in-state resources whose pages have moved or been removed since the content index was built, so roughly six to seven percent of the verified links in that round were stale. That is a maintenance signal for the content index rather than a fault in the model, and the verifier surfaces it on every run.

What the answers cited

The score difference has a plain mechanism behind it. Without grounding, the models write fluent paragraphs and name no resources a person could use. With the directory in front of them, they name in-state resources with working links.

The clearest case is GPT-4o-mini, which cited nothing at all in all forty-eight bare answers and cited a verified link in thirty-nine of forty-eight grounded ones. A person can act on the second kind of answer and cannot act on the first. This is also why the cheaper model with grounding scored above the premium model without it, which is a finding about deployment economics more than about model quality.

Where the full article helps and where it hurts

On the high-risk round, adding the full article text to the directory listing helped one model and did nothing or slightly hurt the other two.

At the highest-risk tier, which covers decisions that cannot be undone such as a plea with immigration consequences or a custody matter crossing state lines, the pattern was sharper. The strongest model dropped more than two tenths of a point when it was given the article text, and only the smallest model gained. The explanation the cohort reached from reading the cases is that a partial article pulls a strong model off a better answer it would otherwise have given. One example is an article that covers standard custody modifications and says nothing about exclusive continuing jurisdiction, which is the thing the question turned on.

Splitting the same round by whether an expert had confirmed that the article actually matched the scenario separates the two outcomes.

Where an expert confirmed the match, every model gained from the article text. Where the content was flagged as uncertain or known to have gaps, two of the three models lost ground. The sample behind the confirmed slice is four questions, which is small enough that the pattern should be treated as a lead rather than a result. It is a lead the cohort takes seriously because it points at content review as the thing that decides whether grounding helps.

How much weight each finding can carry

Each of these findings rests on a different amount of evidence, and they should not be quoted as though they were equally firm.

Directory grounding beats the bare model. This is the firmest finding here. It held across five rounds, six states, seven issue areas, and three models, on several hundred answers. What would make it firmer is a test on questions written by people rather than generated, and a test against a deployed system rather than a prompt.

A model judge alone cannot see safety failures. This has now replicated four times on the cohort's own runs, which is enough to design around. It has not been tested against other judges or other rubrics, so the honest claim is that this judge, on these rubrics, could not separate the conditions. A team using a different judge should check whether its own judge behaves the same way before assuming it does.

Index quality matters more than index size. The state with the largest content index produced one of the smallest gains, and two small well-organized indexes produced the largest. That comparison rests on six states, with one observation each, and no measure of index quality other than the cohort's reading of it. It is a hypothesis with suggestive evidence behind it, and it would take a deliberate test, holding size constant and varying structure, to call it a finding.

Full-content grounding depends on whether the content fits. The direction of this result is consistent across the cases the cohort read, and the numbers behind the confirmed slice come from four questions. Treat the mechanism as plausible and the size of the effect as unknown.

Expert content review pays for itself downstream. This follows from the slice above and inherits its small sample. The cohort is acting on it because the cost of being wrong is low, and the claim should be presented that way rather than as a measured return.

What we did not test

Retrieval was not tested. The articles in the full-content condition were matched to each question outside the model, either by hand or by a separate matching step, so none of these results say anything about how a production retrieval system would choose articles. Answer stability over time was not tested, and the runs are single samples per cell, so there are no variance bands behind any of these numbers. No model outside the three named was tested. No deployed system was tested end to end. All of the questions are in English.

Sixteen cells in the high-risk round came back as provider errors rather than substantive failures, and they are excluded from the means. An error rate of about four percent from a production model is worth knowing for any team considering it for a deployed tool.

The judge scores in the high-risk round collapse ten dimensions into one composite, which means the round cannot say which criterion drove the drop at the highest-risk tier. Recovering that requires rescoring the answers already generated, which is the next run the cohort has planned.

What a team can do with this

A vendor saying that it grounds every answer in your content is describing an architecture, and this testing says the architecture alone does not settle whether the answers are better. The test that answers it is to run your own highest-risk scenarios through the tool and compare its grounded answers against ungrounded ones on the same questions. If the grounded answers do not win on the questions that matter most, the tool is not ready for those questions.

For a team maintaining its own content, the queries where the content was flagged as incomplete are a content roadmap. Each one names a topic where the state's material does not yet cover the hard edge of the issue. The same testing also produces a list of links that no longer load, which is ordinary maintenance that nobody was tracking before.

The infrastructure behind these rounds is open and reusable. That includes the deterministic verifier, the list of known-good links built from the cross-state content index, the sharpened rubric, the quotation check, the scenario-specific judge prompt, and the set of expert-flagged high-risk queries. The measurement report describes how to stand the harness up, and the citation verifier guide covers the deterministic piece on its own.

What comes next

Two runs are planned. The first rescores the answers already generated in the high-risk round to recover the per-criterion scores, which should say whether the drop at the highest-risk tier comes from the model missing the failure mode the expert named. The second tests a new condition that puts the expert's clarifying questions and named failure modes into the model's own prompt rather than only into the judge's. If that condition reverses the drop, the content safety wrapper has evidence behind it. If it does not, the problem is in which article gets retrieved rather than in how the prompt is built, and that is worth knowing either way.