Why it matters to a town
Every response to a records request is read by a person for the details the law keeps out of a release: home addresses Daniel’s Law protects, and personal details the Open Public Records Act excludes, such as Social Security numbers and bank account information. A missed address can put someone at risk. An unneeded redaction withholds what the public has a right to see. His note “AI redaction and Daniel’s Law” sets out where a tool helps and where it fails; this test counts both.
Materials
Synthetic documents only, written for the test: permit files, correspondence and meeting packets, seeded with made-up protected details in known places. Every miss can be counted, and no real person’s information is ever used.
Method
The assistant marks each document. Luis scores every mark against the answer key, then reviews the marked set as he would a real release, timing the review, and checks that the redacted text can’t be copied out of the released files.
Measures
- Recall on protected items: the share of seeded protected details it marks.
- False redactions: marks on text that should stay public.
- Time to a reviewed release.
The baseline to beat
A person redacting the same set without the assistant, timed the same way.
What counts as a pass
The assistant marks every protected home address, misses no more of the other seeded details than the baseline does, leaves no recoverable text and cuts the time to a reviewed release. One missed protected address is a failure.
What will be published
The synthetic set and its answer key, every mark, every miss, every false redaction and the times, exactly as measured, with a plain note on which steps an AI coding agent carried out under his direction.