Independent evaluation for benchmarks built on SEC filings.
AITrailblazer audits and builds evaluations for SI systems that read financial documents. Every grade is checked against the full filing, exhibits included, never against the excerpt a dataset happens to ship with.
Fixed-fee engagements, scoped in writing. Findings go to the benchmark owner privately first. (c) 2026 AITrailblazer
What we do
Two services, one standard: a claim about a filing stands only if the filing itself supports it.
Benchmark audits
We re-grade a sample of your benchmark under blind conditions and report what holds up. The delivery is a per-item flaw list under a fixed taxonomy (wrong gold answer, ambiguous question, evidence mismatch, contaminated item, stale source, difficulty mislabeled), a severity roll-up of clean, fixable, and invalid items, and the agreement statistics behind both. After you fix the items, we re-verify the corrections.
Eval construction
We author evaluation tasks from SEC filings: a question, a gold answer, the filing and page evidence behind it, a weighted scoring rubric, and a difficulty tier. Sets ship with public and private splits. Private holdouts stay private, with scheduled refresh so a set does not go stale or leak into training data.
Method
The protocol is the product. It runs the same way on our own record as it does on yours.
Primary sources only
Grades rest on the complete filing package, exhibits included. Dataset excerpts and snippets are locators. They are never treated as evidence.
Two blind passes
Two graders work independently under one written rubric. Neither sees the other pass, any prior labels, or the benchmark owner's gold calls.
Adjudication
Where the passes disagree, or either one abstains, a fresh adjudicator decides, seeing the competing calls anonymized. Items that stay unsettled ship as abstains, never as forced labels.
Agreement, reported
Every delivery reports raw agreement, Cohen's kappa, the adjudication rate, and abstain counts, with the sample's composition stated next to the numbers. A percentage without its set is not a result.
The record
We grade our own work in public, on the same standard we sell.
AITrailblazer runs DeltaSignal, a public SEC filing intelligence practice. Its calls are graded on a public scoreboard, wins and misses both, and its filing briefs ship through DeltaSignal Radar on this site. The evaluation practice grew out of grading our own output first and publishing the results either way.
The proof asset behind these offers is the DeltaSignal Decider, our blind two-pass SEC filing grading program, with its record stated in public.
How engagements run
Owner-private first, fixed fee, documented method.
Private first
Findings go to the benchmark owner privately, with a response window, before anything is published. What we publish afterward is method and aggregate rates, with the sample size attached.
Fixed fee
A standard audit covers a 150 item sample at a fixed fee. Construction pilots are fixed fee as well. Remediation and re-verification are scoped separately, in writing, before work begins.
Confidential
Client benchmarks and data stay inside the engagement. Our method is documented and shared with the client. Client material is never published, reused, or trained on.
The practice is led by Constantine Vassilev, founder of AITrailblazer, with Rada Vassileva. Engagements start with a scoping conversation, not a purchase.