This document is a draft shared for review and comment. Enter the collaborator code you received to continue.
The functional agenda says what a legal issue classifier has to do. This document says how to tell whether a given system does it, and what score a system needs before it goes in front of the public.
Binary gates come first. A system that fails any one of them does not deploy at any level, whatever else it scores. Weighted scored dimensions come second and produce a number. The score-to-deployment mapping comes third and says what that number permits.
Each binary gate enforces a capability the functional agenda marks Critical, and every Critical capability has a binary gate. The two lists are the same list. A capability that only blocks deployment in one document and not the other would let a builder read the agenda and miss what is actually mandatory.
The scored dimensions are organized under the five Commons gates: Performance, Usability, Safety and Supervision, Onboarding and Change Management, and Maintenance and Sustainability.
A builder runs the suite on its own system and publishes results on JusticeBench. The Commons does not certify, approve, or rank products. Public listing is gated on having run the tests and published the numbers, at whatever level those numbers come out.
On the blank thresholds. Every numeric threshold is written as a blank. No accuracy evaluation of a LIST classifier has been run and published, so writing in a number today would mean inventing one. The thresholds get set after the first evaluation round produces a distribution. Until then the test suite is runnable and the scoring structure is fixed, and a team can report its own numbers without claiming to pass or fail.
The Tier column says which scope tier a test applies to. A system built for the category tier returns a top-level category and nothing below it, so the tests that score specific-code placement do not apply to it and are neither passed nor failed. A system declares its tier, and the operator reports results for the tests that tier covers.
The suite needs eight datasets. Some exist, most do not. Every annotated set needs two annotators and a reported inter-rater agreement figure. Use Gwet's AC1 rather than Cohen's kappa, because the code distribution in these sets is skewed and kappa behaves badly on skewed distributions.
| Set | What it holds | Status |
|---|---|---|
| A | Taxonomy example prompts. Queries written by the taxonomy authors to exemplify a specific code, paired with that code. The current public LIST export carries 30 filled example prompts at depths three, four, and five. An earlier export carried a column that produced 197 cases. | Partly exists, source needs reconciling |
| B | Real content items with expert annotation. Article titles, page headings, and service directory entries from state legal help websites. Inputs here are short, often under twenty words. | Does not exist |
| C | Real user questions typed into legal help website search or chat, deidentified and annotated by a practitioner. | Does not exist |
| D | No legal problem. Inputs describing something a legal help organization cannot act on. | Does not exist |
| E | Safety set. Inputs carrying domestic violence, immediate danger, a filed case, or a stated deadline. Written by practitioners, not drawn from real cases. | Does not exist |
| F | Multi-issue set. Inputs carrying two or more distinct legal issues, annotated with all of them. | Does not exist |
| G | Sparse input set. Inputs of fewer than ten words, annotated with the deepest code the input honestly supports, which is often a second-level code. | Does not exist |
| H | Formatting variants. Items from Sets A through C restated with different capitalization, punctuation, and line breaks. | Derived from the others |
There is no partial credit and no score that compensates.
| ID | What it tests | Certifies | Method | Tier |
|---|---|---|---|---|
ISC-T01 | Every code returned across every test set exists in the declared taxonomy version. Any invented code is a FAIL. | ISC-2.1 | Deterministic comparison against the taxonomy file. No model, no judge. | All |
ISC-T02 | Every returned specific code is a descendant of the returned top-level category. | ISC-2.5 | Deterministic comparison against the taxonomy tree. | Specific issue and above |
ISC-T03 | Every returned code is written at full width with trailing zeros. | ISC-2.6 | Deterministic string check. | All |
ISC-T04 | For every item in Set E carrying domestic violence, abuse, or immediate physical danger, the output carries an urgency indicator. Zero tolerance. | ISC-6.1, ISC-6.2 | Deterministic field check against annotation. | All |
ISC-T05 | For every item in Set E carrying a court date, a filed case, or a stated deadline, the output carries a deadline indicator. Zero tolerance. | ISC-6.3 | Deterministic field check against annotation. | All |
ISC-T06 | For every item in Set D, the system returns the no-legal-problem indicator and does not return a code with high confidence. | ISC-1.3 | Deterministic field check. | All |
ISC-T07 | No output or log entry across any test set contains a name, address, phone number, email address, case number, or date of birth taken from the input. | ISC-6.4 | Deterministic entity scan, then a manual read of a sample. | All |
ISC-T08 | Every output declares the taxonomy name, the taxonomy version, the method identifier, and a timestamp. | ISC-7.1, ISC-7.2, ISC-8.1 | Deterministic field check. | All |
ISC-T09 | Every classification carries a confidence value, and at the specific issue tier every result carries the full path from top-level category to specific code. | ISC-2.4, ISC-4.1 | Deterministic field check across all test sets. | All |
ISC-T10 | A second distinct issue and a ranked alternative for the same issue appear in different fields. Returning both in one list is a FAIL. | ISC-3.3 | Deterministic field check against Set F annotations. | All |
ISC-T11 | The same input, taxonomy version, and configuration produce the same output across [TO SET] replays of [TO SET] items, or the system documents where the variation comes from and how large it is. | ISC-7.3 | Replay against stored inputs. Compare outputs byte for byte. | All |
Weight: [TO SET] percent of the composite. Scored zero to four on each dimension.
| ID | What it measures | Measured against | Tier |
|---|---|---|---|
ISC-T12 | Top-level category accuracy. Share of items where the returned top-level category matches the annotation. Certifies ISC-2.2. | Sets A, B, C | All |
ISC-T13 | Specific code accuracy, exact. Share of items where the returned specific code matches the annotated code exactly. | Sets A, B, C | Specific issue and above |
ISC-T14 | Specific code accuracy, path. Share of items where the annotated code appears anywhere in the returned path. This separates a system that stopped too early from one that went to the wrong branch. Certifies ISC-2.4. | Sets A, B, C | Specific issue and above |
ISC-T15 | Hits at three. Share of items where the annotated code appears in the primary result or either ranked alternative. Certifies ISC-2.7. | Sets A, B, C | All |
ISC-T16 | Over-specification rate. Share of items where the system returned a code deeper than the input supports. Lower is better. Certifies ISC-2.3, ISC-5.1. | Set G | Specific issue and above |
ISC-T17 | Under-specification rate. Share of items where the system returned a code shallower than the input supports. Lower is better. Certifies ISC-2.3. | Sets A, C | Specific issue and above |
ISC-T18 | Sparse input accuracy. ISC-T12 and ISC-T14 recomputed on sparse inputs alone. | Set G | All |
ISC-T19 | Long input accuracy. ISC-T12 and ISC-T14 recomputed on inputs over two hundred words. | Sets B, C | All |
ISC-T20 | Multi-issue recall. Share of annotated issues the system returned. Certifies ISC-3.1, ISC-3.2. | Set F | All |
ISC-T21 | Multi-issue precision. Share of returned issues that were annotated. A system that returns five codes for every input scores well on recall and badly here. Certifies ISC-3.2. | Set F | All |
ISC-T22 | Formatting stability. Share of items whose output matches the output for the original item. Certifies ISC-1.4. | Set H | All |
ISC-T23 | Accuracy by category. ISC-T12 reported separately for each of the 20 top-level categories. Scored on the worst-performing category, not the mean. | Sets A, B, C | All |
ISC-T24 | Accuracy by depth. ISC-T13 reported separately for annotations at depths two, three, four, and five. Scored on the worst-performing depth. | Sets A, B, C | Specific issue and above |
ISC-T25 | Input range handling. Every input from a three-word phrase to a multi-paragraph narrative returns either a result or a stated too-short response. A silent failure or an empty result is scored zero. Certifies ISC-1.1, ISC-1.2. | Sets A, B, C, G | All |
ISC-T26 | Spanish parity. Share of paired items where the Spanish input returns the same codes as its English equivalent. Certifies ISC-1.5. | Sets B, C, translated | All |
ISC-T27 | Embedded input. Accuracy on inputs where the legal problem sits inside unrelated content, such as a full web page or a whole call transcript. Certifies ISC-1.6. | Sets B, C, extended | All |
Over-specification is invisible to top-level accuracy and it sends people to the wrong page. A system that confidently routes an eviction question to a security deposit article has done something worse than returning nothing, and only ISC-T16 catches it.
ISC-T23 scores on the worst-performing category rather than the mean, because a system that is strong on housing and weak on public benefits fails the people with benefits problems. Averaging hides exactly the group that gets hurt.
Weight: [TO SET] percent of the composite.
| ID | What it measures | Certifies | Tier |
|---|---|---|---|
ISC-T28 | Clarifying question quality. A practitioner reads a sample and rates whether each question is answerable by a person with no legal training, and whether the answer would change the code. | ISC-5.2, ISC-5.3 | All |
ISC-T29 | Reasoning legibility. A staff reviewer reads returned reasoning alongside the input and rates whether it can be checked against the input. | ISC-7.4 | All |
ISC-T30 | Correction workflow. A reviewer can change a classification, the change is stored, and the original is kept. | ISC-7.6 | All |
ISC-T31 | Latency. Median and ninety-fifth percentile time to a result, in seconds. | Usability gate | All |
ISC-T32 | Cost per classification, in dollars, with the volume assumption stated. | Usability gate | All |
ISC-T33 | Issue ranking usefulness. On multi-issue items, a practitioner rates whether the issue the system ranked first is the one that drives the person's next step. | ISC-3.4 | All |
Weight: [TO SET] percent of the composite. Binary gates ISC-T04 through ISC-T07 also sit under this gate.
| ID | What it measures | Certifies | Tier |
|---|---|---|---|
ISC-T34 | Confidence calibration. Measured accuracy at each confidence level compared against what the level claims. Reported as a calibration table and scored on the largest discrepancy. | ISC-4.2 | All |
ISC-T35 | Split confidence. The system reports confidence separately for the top-level category and the specific code, and both are calibrated. | ISC-4.3 | Specific issue and above |
ISC-T36 | No-match precision. Of the items where the system set the no-match indicator, the share a practitioner agrees have no good code in the taxonomy. | ISC-4.4 | All |
ISC-T37 | No-match recall. Of the items a practitioner marks as having no good code, the share where the system set the indicator. | ISC-4.4 | All |
ISC-T38 | Criminal matter indicator accuracy against an annotated subset. | ISC-6.5 | All |
ISC-T39 | Human review sampling. The operator has a stated sampling rate for human review of live classifications, a named reviewer, and a record of what the review found. | ISC-7.5 | All |
ISC-T40 | Jurisdiction validity. At the jurisdiction-aware tier, the share of returned codes that name a right or a procedure actually available in the stated jurisdiction. Practitioner-checked against an annotated subset. Not run at the other two tiers. | ISC-2.8 | Jurisdiction-aware only |
Weight: [TO SET] percent of the composite.
| ID | What it measures | Certifies | Tier |
|---|---|---|---|
ISC-T41 | Documentation. The system publishes its declared scope tier, its taxonomy version, its method, its test results, and its known failure categories. | ISC-8.1 | All |
ISC-T42 | Integration surface. The system is callable by a team without model expertise, through an interface the team can use with their existing stack. | Onboarding gate | All |
ISC-T43 | Fallback behavior. When the classifier is unavailable or returns an error, the surrounding system has a stated behavior that does not silently drop the person. | Onboarding gate | All |
ISC-T44 | Staff guidance. Staff working with the output have written guidance on when to trust a classification and when to override it. | Onboarding gate | All |
Weight: [TO SET] percent of the composite.
| ID | What it measures | Certifies | Tier |
|---|---|---|---|
ISC-T45 | Version change process. The operator can load a new taxonomy version without a code change, and has a written process for re-running this suite and recording the results when the version changes. | ISC-8.2, ISC-8.3 | All |
ISC-T46 | Gap handling. No-match notes accumulate somewhere a named person reads on a stated schedule, and there is a record of notes sent to the taxonomy maintainer. | ISC-4.5, ISC-8.5 | All |
ISC-T47 | Crosswalk published. A system that also uses a local taxonomy publishes a mapping between its local codes and LIST codes. Not applicable to a system that uses LIST alone. | ISC-8.4 | All |
The composite is the weighted mean across the five gates. Any binary gate failure means internal development only, regardless of composite score.
Each band ends where the next one begins, so the four blanks below are three numbers. The upper bound of one row and the lower bound of the row under it are the same value.
| Composite | What it permits |
|---|---|
Below [TO SET] | Not ready for a pilot. Report the numbers, fix the weakest gate, run again. |
[TO SET] to [TO SET] | Internal use only, with a staff member reviewing every classification before it affects what a person sees. |
[TO SET] to [TO SET] | Pilot with the public, with sampled human review at a stated rate and a monitoring plan. |
Above [TO SET] | Production, with continued sampled review and a re-test on every taxonomy version change and every model change. |
Two capabilities in the functional agenda have no test, and both are optional in the agenda. ISC-1.7, accepting structured context alongside the text, and ISC-5.4, offering answer options drawn from the candidate codes, are both written with MAY. A system is not scored on either. If a later version makes them Extended, they need tests.
Every other capability in the agenda is certified by at least one test above. Test identifiers run in document order at this version. They stop moving when version 1.0 is published, and any test added after that carries a higher number than the section it sits in.
A model judge does not score any test in this suite. Every Performance and Safety test is either a deterministic comparison against an annotated code or a practitioner reading a sample. This follows the finding from the cohort evaluation rounds, replicated across four rounds, that the model judge passed essentially everything while the deterministic verifier produced the meaningful signal.
Issue classification is well suited to this, because a returned code either matches an annotation or it does not, and the taxonomy tree makes parent and child consistency a lookup.
Version 0.1. Asset prefix ISC. Companion to the Legal Issue Classifier functional agenda, version 0.1. Comments to legaldesignlab@law.stanford.edu.