This document is a draft shared for review and comment. Enter the collaborator code you received to continue.
This standard evaluates a system that takes a legal document containing personal information and produces a version of that document safe to send somewhere else.
The systems in scope include commercial detection services, open source libraries, self-hosted deployments, and scripts a legal aid organization wrote itself.
The destinations in scope are foundation model providers, researchers, developers, and open datasets.
The standard applies to the masking step alone. Optical character recognition quality drives masking performance in scanned legal documents and has to be evaluated end to end with masking rather than in isolation, so a system that ingests scans is evaluated on its output after recognition, and the recognition standard is scored separately.
A system is scored against the tier assignments below. A conformance result reported without naming the tier version it was scored against is not a conformance result.
| Tier | Handling | Entities |
|---|---|---|
| High or definite | Masked by default | Social security number, bank and financial account numbers, driver's license or state identification, passport number, phone number, email, date of birth, signature, home and mailing and property address, insurance policy number, criminal history details. |
| Undecided | Proposed for the high tier, not yet agreed | Person name, minor name. Do not score a system against these entities until the tier assignment is settled. |
| Medium or possible | Masked depending on mode and context | Last three digits of a social security number or identification number, case or court number, initials, age, income amount and financial assets, medical condition and pregnancy status, employer, school and daycare names, license plates and vehicle identification numbers, place of birth, immigration identifiers including alien registration number, client identifier. |
| Low or not personal information | Usually left readable | Hearing date and time, document date, rent amount and generic financial totals, judge name, insurance company name, laws and citations and legal labels, court name, court location, case type, filing date, county and courthouse, relative name, unique incident details. |
Open decision. Does person name belong in the high tier, and does minor name belong with it? PIM-G1 does not currently name either one, so a reviewer following the gate as written can pass a document that still carries the tenant's full name in the caption.
Open decision. Four entities have competing assignments. Driver's license and state identification are proposed as high and as medium. Employer and school names are proposed as medium and as low. Initials are proposed as a medium entity of their own and as part of the person name span.
Opposing party name, meaning the landlord, the debt buyer, or the plaintiff company. Attorney name and bar number. Process server name. Unit or apartment number handled separately from the street address. Photographs of people inside exhibits. Internet protocol address, device identifier, and account handles.
Open decision. Which of these entities enter the tier table in the first agreed version, and which are deferred.
A system runs in a named mode, and the mode decides how the medium tier is handled. PIM-1 asks whether the system caught the medium-tier items the selected mode called for. PIM-2 asks whether it left the low tier readable, which strict mode overrides.
The medium tier is defined by what the mode does to it, so the mode set has to be named before a medium-tier entity can be scored.
Open decision. What is the full set of modes, and what does each one do to each of the three tiers? Strict and balanced are named. Whether a third mode exists has not been settled. A related question is whether modes are the same thing as the sharing contexts of internal collaboration, external expert review, and public dataset release, in which case one set of names replaces the other.
| ID | Failure condition | How it is checked |
|---|---|---|
PIM-G1 |
A high or definite item is still visible. The gate covers social security number, date of birth, address, identification number, phone, email, and signature. | The document-level rate of any missed high-risk item from the evaluation protocol. Any missed high-risk item fails the sample automatically. |
PIM-G2 |
Personal information can still be found by searching or copy-pasting, because the text layer was not cleaned. | Extract the text layer from the output file and run a string match for every gold-labeled high-tier span from the source document. Any hit fails the gate. Repeat the extraction after copy and paste, and after running character recognition on a rendered image of the output. |
PIM-G3 |
The masking is reversible, because annotations or layers can be removed. | Inspect the output for annotation objects, layers, and redaction marks that carry the original content beneath them, and remove them programmatically to confirm nothing is recoverable. |
Gates two and three are checked deterministically, without a model and without a human reading the document.
PIM-G1 covers social security number, date of birth, address, identification number, phone, email, and signature. It does not cover person name. A reviewer following the gate as written can pass a document that carries the tenant's full name in the caption. Settling the tier assignment for person name settles this gate at the same time.
A score of 1 is Not OK, 2 is OK with cleanup, and 3 is Good.
| ID | Criterion | The question | Score 3, Good | Score 2, OK with cleanup |
|---|---|---|---|---|
PIM-1 | Catches personal information | Did it find what is there? | Catches high and definite items throughout the document, including repeats. Catches key medium items as expected for the selected mode. | Catches every high and definite item, and misses medium items that the selected mode called for. |
PIM-2 | Masks the right things | Did it mask what it should, and leave what it should not? | Masks high-tier items. Does not mask low or not-personal content, including dates, rent totals, judge name, and citations, unless the user chose strict. | Masks high-tier items, and also masks low-tier content the mode called for leaving readable, without destroying a deadline, a court location, or an amount owed. |
PIM-3 | Masking is real | Is the information truly gone rather than hidden? | The output cannot be searched or copied for personal information. The masking holds across exports to PDF, PNG, and TXT without leakage. | The masking holds in the primary output format and leaks in a secondary export or under character recognition of a rendered page. |
PIM-4 | Avoids overmasking | Did it avoid hiding important content that is not personal information? | It does not black out whole paragraphs because of one identifier. It keeps legal meaning readable, with dates, amounts, and key events still visible. | It removes surrounding words along with an identifier, and a reader can still find the deadline, the court, and the amount. |
PIM-5 | Consistent and reviewable | Is it easy to verify and fix? | It masks all instances, including headers, footers, tables, and exhibits. It shows what it masked through a log or highlights, or it makes manual correction easy. | It misses instances in headers, footers, tables, or exhibits, and it reports what it did well enough that a reviewer can find and fix them. |
A criterion scores 1 when the conditions for 2 are not met.
Open decision. Do the anchors for a score of 2 hold up in use? They are written so that two reviewers scoring the same output reach the same score. No one has scored a system with them yet, and they should be confirmed or replaced before results are reported.
Open decision. Do the five criteria carry weights, or are they scored equally? The bands below treat them equally and derive the result from how many criteria scored 1. Other Legal Help Commons conformance standards weight their criteria, and matching that pattern would change how a band is reached.
Any hard fail, or two or more criteria scored 1.
No hard fail, and criteria mostly scored 2.
No hard fail, and criteria mostly scored 3.
A band tells a reviewer what they are looking at. It does not yet say what an organization may do with a system that lands in each band.
Open decision. What does each band permit? One proposal is that Good permits public dataset release, OK with cleanup permits external expert review with a human in the review queue, and Not OK permits internal collaboration only. That proposal reuses the three sharing contexts as the deployment tiers.
A result is reported as a single record. A result that omits any field below cannot be compared to another result and will not be listed.
| Field | What it holds |
|---|---|
| System name and version | The service, library, or deployment that was scored, at the version that ran. |
| Mode | The masking mode the system ran in. |
| Benchmark version | The named benchmark set the samples came from. |
| Tier table version | The version of the entity tiers the scoring used. |
| Gate outcomes | Pass or fail for PIM-G1, PIM-G2, and PIM-G3, with the check that produced each result. |
| Criterion scores | The score of 1, 2, or 3 for each of PIM-1 through PIM-5, with the reviewer notes. |
| Band | Not OK, OK with cleanup, or Good. |
| Metrics | The measured values from the metrics table. |
| Date and reviewer | When the run happened and who ran it. |
A result carries the standing of whoever produced it. A self-scored result is reported by the team that built the system. A peer-scored result is reported by another cohort member. A Lab-scored result is reported by the Stanford Legal Design Lab. All three are publishable and the record names which one it is.
Open decision. Do the three kinds of result map onto the JusticeBench engagement tiers of Spotted, Documented, and Evaluated? A second question follows from it: what evidence must a self-scored result attach before it is listed?
A system that produces none of these cannot be scored on PIM-5 and cannot be audited after the fact.
| Artifact | What it contains |
|---|---|
| Masked artifact | A new PDF, image, or text file where sensitive strings are removed or obscured. |
| Masking manifest, machine-readable | What was found and how it was transformed: entity type, position and span, confidence, method used, and version of the recognizers. |
| Risk flags | Possible missed items, low confidence, handwriting detected, recognition quality low, non-English names likely. |
| Audit log | Who ran it, when, on what, and what policy version. |
The masking policy is treated as a versioned contract, because what an organization redacts today may change as its governance matures. The manifest and the audit log both carry the policy version.
A conformance result is reported against a named benchmark version.
The benchmark holds 300 samples, split as 150 text-first, meaning clean digital PDFs or copied text, and 150 scan-first, meaning character recognition is required at varying quality. Samples are stratified by document type, language mix, and presence of tables, stamps, and signatures.
Document types are notices carrying addresses and names, pleadings carrying names and case numbers, intake notes carrying phones and emails and dates of birth, letters in free text, and scans with stamps and signatures.
Gold labels are human-labeled spans with start, end, and type, a severity tier per entity type, and must-preserve annotations recording what content cannot be destroyed.
Ten to twenty percent of records are double-labeled, disagreements are resolved, and the labeling guide is updated whenever an ambiguity appears.
The benchmark set is frozen and no improvement work touches it.
The cohort collected filled-in or blank documents from partners, inventoried them by issue type, document type, and personal information status, and produced synthetic duplicates by populating realistic but fake names, addresses, case numbers, dollar amounts, and case details into the same templates.
The synthetic table covers Illinois, Michigan, Minnesota, Ohio, and Oregon, with two counties in each state, real court names, and case numbers in the form IL-UD-2025-00009.
Each courthouse appears twice, once with an ASCII name and once with a name carrying diacritics or non-Latin ordering, at the same street address with a different unit. Cook County has William Moore at 3428 Justice Ave, Apt 1C and Tünde Kovács at Apt 1D.
Open decision. Is the paired-name design a requirement of the benchmark? If the purpose is to measure the difference in recall by name origin while jurisdiction and address are held constant, this standard should say so, and the metrics should report recall separately for the two halves of each pair.
Not yet recorded. The count of templates in the set, the count of populated documents produced from them, and the count that carry gold spans.
| Metric | How it is measured |
|---|---|
| Recall, precision, and F-beta by entity type | Against gold-labeled spans. |
| Document-level rate of any missed high-risk item | The share of documents with any missed high-risk item. |
| Over-redaction rate | Precision together with human utility ratings. |
| Utility score | A human answering whether a legal worker can still use the document. |
| End-to-end throughput and cost | Documents per minute at the required quality. |
A system is tuned toward high recall, and precision is managed with policy and post-processing afterward. Missing a high-risk item is the primary safety risk. An overmasking problem is an inconvenience, and a missed identifier is a disclosure.
Low recognition quality pages. Handwritten fragments. Non-English names and addresses. Personal information in margins, headers, and footers.
Hard cases inside the benchmark: all-caps notices, scanned pleadings, headers and footers, signatures, stamps, and tables.
Names including initials, multiple last names, and hyphenations. Addresses with apartment and unit patterns and post office boxes. Case numbers in court-specific formats.
A judge name and a party name appearing in the same paragraph, which tests whether name masking is role-aware. A figure that is both the rent owed and the tenant's income, which tests the boundary between the medium-tier income amount and the low-tier rent amount. Two households at the same street address with different unit numbers, which the synthetic dataset already supplies.
Any missed high-risk item fails the sample. Any over-redaction that deletes required operational content, including deadlines and court location, fails the sample.
Masking runs in the cloud or locally. Cloud is faster and easier to maintain. Local is safer because the document never leaves the user's environment. Both are available and the choice is made per deployment based on the sensitivity of the use case. This standard scores either configuration.
Where recognition confidence is low or the document is high risk, the system routes to human review rather than sharing automatically.
The interface tells the user what was removed, by type and count, what might still be present through low confidence warnings, and what to do next through a review queue.
The standard, the entity tier table, and the benchmark set each carry their own version number, because a change to any one of them changes what a score means.
A change that alters what a system has to do to pass raises the first number. A change that adds an entity, a stress test, or a clarification raises the second number. Every version stays published, so a result reported against an earlier version stays readable.
Open decision. Who approves a version change? A published result can either be rescored when the tier table changes or left standing with the version it ran against, and the standard should say which.
Version 0.2. Asset prefix PIM. Comments to legaldesignlab@law.stanford.edu.