Methods
A report is only as good as the reader's ability to check it. This page describes the standard every NorthLaw model and report is held to, so that opposing counsel's expert finds nothing that was not already disclosed.
Every model is built from decided cases coded for the factors courts actually cite. Each decision in a comparable set is identifiable by citation and can be listed on request. No report depends on undisclosed data.
Models are frozen, then tested on held-out decisions that played no part in training. We report how often the estimate falls within a stated distance of the actual outcome, not just an average error.
Legal data leaks in three ways: facts known only after the outcome, citations to the decision itself, and time ordering. Each model is audited against all three before deployment, using the framework in our working paper on temporal leakage.
Point estimates are accompanied by prediction intervals whose coverage is checked on held-out data. An 80% interval is expected to contain the actual outcome about four times in five, and we say so when it doesn't.
Each report identifies which facts move the estimate most and by how much, so counsel knows what the argument is about before the other side does.
Every report states what the analysis is not: not a prediction of a particular judge, not legal advice, not a substitute for the evidence. Settled cases are unobserved and may differ from litigated ones; we say that too.
These engines are used by NorthLaw to prepare reports for counsel. They are not offered as self-serve tools, because a case-specific estimate requires judgment about the facts, the data and the law, which is what the report provides.
Validation notes are maintained per model and per data refresh. They include the held-out sample size and period, coverage of the stated interval, error by subgroup, and the leakage audit. They are provided to counsel before engagement and are appended to every expert report.