Current work
Teaneck was selected for its public records corpus. Document review and preparation of the question set are in progress; the comparisons and final evaluation are still ahead. The sections below describe the planned protocol.
Why it matters to a town
Residents, applicants and staff ask the same questions of Planning Board records every week. Was the variance granted? On what conditions? Which version of the resolution is current? A fluent answer drawn from the wrong version is worse than no answer, because someone will act on it. That is the argument of his note “A correct source matters more than a fluent answer”; this test puts numbers on it.
Materials
Teaneck’s posted Planning Board minutes and resolutions, from the Township’s own website. Public documents only: nothing personal is added, and nothing that isn’t already public goes into any tool.
The question set
A planned set of 30 questions, each answered by hand from the records before any tool sees it. Some have answers that changed over time, such as an approval later amended or extended, so the right version matters. Some can’t be answered from the records at all, and the right response is to say so.
Method
Every question goes to the cited search tool and to the baseline. For each answer Luis records what it said, the source it cited, whether that source supports it, whether it’s the current version and how many minutes it took him to verify it against the record.
Measures
- The share of answers whose cited source actually supports them.
- The share that return the right version of a record.
- The share of unanswerable questions correctly answered with “can’t tell from these records.”
- Minutes to verify each answer.
The baseline to beat
The town’s own website search, plus a text search of its posted PDFs: what staff and residents use today.
What counts as a pass
The cited search has to beat the baseline on all four measures, with time counted from the question to a verified answer. Faster alone isn’t a pass. An answer that cites a source that doesn’t support it counts as a failure, however good it sounds.
What will be published
The question set and its answer key, every answer from the tool and the baseline, every failure, the measures and the time spent, exactly as measured, with a plain note on which steps an AI coding agent carried out under his direction. The label moves to “Evaluated prototype” only when the written test, its results and its failures are all on this page.