Find and remove personal information from legal documents before they are stored, logged, shared, or sent to a model.
Personal information has to come out of a legal document before that document can go to a model provider, a researcher, a vendor, or an open dataset. This page holds what the cohort is building so that a team does not have to work out masking on its own: a standard for judging a masking system, the data to test one against, and the guidance for building or buying one.
A masking system takes a legal document containing personal information and produces a version of that document safe to send somewhere else. The destinations that matter here are foundation model providers, researchers, developers, and open datasets.
Every organization doing legal help AI work hits this problem, and most of them solve it alone. The pieces are the same each time: find the identifiers, decide which ones to remove, remove them so they cannot be recovered, and leave enough of the document that a legal worker can still use it.
The last part is what makes legal documents difficult. A masker that blacks out a whole paragraph because it found one identifier has destroyed the deadline, the court location, or the amount owed. Both failures matter, and they pull in opposite directions, which is why the standard scores catching and overmasking as separate criteria.
Character recognition quality drives masking performance in scanned documents, so a system that ingests scans is evaluated end to end on its output rather than on the masking step in isolation.
This serves legal aid organizations sharing documents with researchers or vendors, courts preparing filings for analysis, teams building evaluation sets, and anyone sending legal text to a model they do not host.
A team sends a document in, as a clean digital file or as a scan. The system reads the text, including whatever the character recognition can pull off a photograph or a fax. It then finds the identifiers, which means both the ones any general detector knows, such as a name, a phone number, or an account number, and the ones particular to legal filings, such as a case caption, a signature block, or the names of children in a family matter.
Each identifier it finds is sorted by how sensitive it is, because a Social Security number and a hearing date do not call for the same treatment. The system then masks according to the mode the team asked for, since a document going to an expert reviewer inside a partner organization can keep more than one going into a public dataset. What comes back is the masked document, plus a record of what was changed, so a person can check the work and a team can show how a file was treated.
The review step is the one teams underestimate. Nothing in this package assumes a masker runs unattended, and the open questions below include what a reviewer is allowed to do with a result once they have one.
Here is where each shared asset stands. The conformance standard is drafted and open for comment. The benchmark set is partly built. The rest follow from them.
This is how to evaluate a masking system. A system has to clear the hard gates before anything else is scored, since failing one of them makes a score meaningless. The scored criteria cover what the system catches and what it wrongly hides, and the result bands say how to read the total. The standard also sets out the entity tiers a system is scored against.
Read the standard →The benchmark holds samples split evenly between clean digital text and scans of varying quality, stratified by document type, language mix, and whether the page carries tables, stamps, and signatures. Each gold label records the span, the type, the severity tier, and the content that has to survive masking.
Join the cohort to contribute →These are partner templates filled in with realistic but invented names, addresses, case numbers, and amounts, covering Illinois, Michigan, Minnesota, Ohio, and Oregon with a pair of counties in each. Every courthouse appears at the same address under two spellings, one plain and one carrying diacritics, so a system can be checked for treating them the same way.
Join the cohort to contribute →This is the list of repositories worth copying, and it records the license and the maturity of each one. Commercial detection services, open source libraries, and self-hosted deployments are all in scope, and the standard scores any of them.
Coming as the cohort formsThis covers the recommended architecture, where the work splits between deterministic code and the model, how to choose between cloud and local deployment, and when a team should buy rather than build. It also says which layers should survive a model swap and which are expected to be replaced.
Coming as the cohort formsThese are case studies from organizations running masking in production. Each one says what the team chose, what the review queue costs it in staff time, and what it would do differently.
Coming as the cohort formsThe conformance standard says what good looks like. The benchmark set is what you score against. The synthetic set is what you can share and test with when real documents cannot leave the building. The reference implementations are systems the standard has been run against.
You do not need all of them. Pick the resources that match your role.
The cohort is collecting this. If you are running masking in production, or you have written your own script and would rather not maintain it alone, tell us and we will connect you with the others working on it.
These are unresolved and marked so nobody mistakes them for settled. The full list sits at the end of the conformance standard.
Before a masking tool can be judged, somebody has to say what it is supposed to find. A group of people who work with legal documents every day went through the fields that appear on a filing and marked each one as clearly personal information, possibly personal information, or not personal information. They could also say that it depends on the document, and they used that option often.
Agreement was complete on the classic identifiers. A person's name, their email address, their Social Security number, a driver's licence or state ID, and a bank account number were marked clearly personal by everyone who gave an answer. Date of birth, home address, phone number, property address, insurance policy number, and signature followed close behind.
The middle of the list is where a masking tool actually has to make decisions. Criminal history split the group evenly between clearly personal and possibly personal. Court numbers, medical conditions, the last digits of a licence or a Social Security number, income, assets, rent, pregnancy status, age, gender, and initials all drew mixed answers. One respondent summed up the pattern: most of the possible answers are in the camp of it depends, since a field like an address matters differently depending on who the person is and what kind of case it is.
The written answers added fields the list had missed. People named passport and state identification numbers, address components such as a zip code or census tract when they appear with other details, vehicle identification and licence plate numbers, identifying marks, professional licence numbers, school and day care names, employers, place of birth, alien registration numbers, financial account numbers beyond bank accounts, other ongoing court orders, and the names and birth dates of family members, with children in domestic relations cases named more than once. Several people pointed at the narrative itself, where a person tells their story in their own words, as the hardest part of a document to mask.
The same exercise mapped each field against what general-purpose detection services already label. The classic identifiers are well covered, since every service has a label for a name, an email address, a Social Security number, a licence number, a bank account, an address, and a phone number. Coverage thins out quickly after that. Signatures, criminal history, court numbers, income, rent, and assets have no standard label in the services checked, and those are exactly the fields the group found hardest to agree on.
This is consultation input rather than a finding. The group was small, it answered once, and nobody checked the answers against how these people actually redact in practice. The point of asking was to see where the field agrees and where it does not, so that the conformance standard can be explicit about the fields in the middle instead of assuming them away.
The PII masking working group is forming now. Members settle the open decisions, contribute documents to the benchmark, and run the standard against the systems they already use.