No description
_s split without an LLM in the retrieval loop, and demonstrates zero retrieval regression across 20 consecutive gbrain releases.| Rating | Summary | |
|---|---|---|
| Quality | solid (17/24) | Actively committed, well-documented benchmark suite with strong adoption signals, let down only by no formal releases and HTML as the GitHub-reported primary language |
| PAI Relevance | integrate (0.50) | BrainBench's sealed-qrels enforcement, planted-perturbation corpus design, and multi-adapter harness pattern offer concrete methodology to extract for PAI's Evals skill |
Composite is exactly at the 0.50 INTEGRATE threshold; the practical integration action is methodology study and pattern extraction rather than running gbrain-evals as-is against PAI, since all adapters call into gbrain/* subpath exports and would need full replacement.
17/24 — maintained / adequately-documented / solid
Failed:
Passed:
Failed:
eval:* scripts appear only in package.json, not a README headingPassed:
git clone and bun install stepsdocs/benchmarks/ directory tree and to individual benchmark reportsFailed:
Passed:
"test": "bun test test/eval/" and multiple eval:* runner scripts| Dimension | Score | Assessment |
|---|---|---|
| Harvest Value | 1 | The sealed-qrels enforcement pattern (gold _facts metadata that never crosses the adapter boundary), planted-perturbation corpus design (10 contradictions, 5 stale facts, 5 paraphrased-injection poison items), and 12-category BrainBench structure each offer concrete techniques worth extracting for PAI's Evals skill, which currently handles general agent evaluation rather than retrieval-quality benchmarking specifically. |
| Integration Readiness | 1 | All runner scripts are TypeScript and Bun-native (stack match), but every adapter imports from gbrain/* subpath exports; adapting the harness to PAI's Knowledge subsystem requires replacing the adapter layer — moderate glue, not drop-in. |
| Overlap Risk | 1 | Partial overlap with PAI's existing Evals skill (AI agent evaluation); gbrain-evals focuses specifically on retrieval-quality metrics (P@5, R@5, per-type F1, latency p95), a narrower sub-domain the Evals skill does not deeply cover. |
| Gap Fill | 1 | PAI has no dedicated retrieval-quality benchmark harness for its Knowledge subsystem; gbrain-evals addresses that functional gap but requires adaptation away from gbrain internals before it can measure PAI's retrieval pipeline. |
Composite: 0.50
Capture-to-Knowledge Pipeline (currently at manual validation stage): Run gbrain-evals' BrainBench suite against the pipeline's retrieval layer now, before implementation solidifies — clone the repo, substitute your own committed corpus for the included 240-page fictional one, and capture R@5 as a baseline. Manual validation gets replaced by a repeatable numeric gate that fires on every architecture change rather than on an ad-hoc schedule.
Capture-to-Knowledge Pipeline (retrieval adapter selection): Use the multi-adapter comparison runners included in gbrain-evals to benchmark at least two candidate adapters (e.g., Contriever vs. whatever dense retriever the pipeline currently targets) against your actual knowledge corpus before finalising the architecture. The SOTA 97.60% R@5 result is claimed without an LLM in the retrieval loop — replicating or refuting that on your own document distribution produces a concrete, defensible architecture decision rather than a design assumption.
Capture-to-Knowledge Pipeline (regression gate for future releases): Wire the BrainBench runner into the pipeline's CI as a required check, mirroring gbrain's own zero-regression record across 20 releases. Any future change to chunking, embedding model, or index configuration that drops R@5 below the baseline fails loudly — this converts the current clean-room Haiku validation step from a one-time sanity check into a durable quality floor.
Category: LLM & Prompt Tooling
In this category: mattpocock--evalite (decent, 15/24)
Standing: gbrain-evals scores higher than evalite on every dimension and addresses a more specific domain — personal knowledge agent retrieval benchmarking with committed corpora — versus evalite's general LLM output evaluation; both lack formal releases.
Density: 7/10 — Available: full README, complete package.json, repo metadata (stars, forks, dates, license, open issues, archived flag), dependency manifest, scripts catalog, benchmark results table. Missing: actual TypeScript source files, CI configuration, CHANGELOG, contributor graph, test output examples.
The HTML-as-primary-language tag is misleading — it reflects committed benchmark report files rather than the TypeScript evaluation codebase; actual runner and adapter code is entirely .ts. The version string "v0.40.6.0" in results tracks the parent garrytan/gbrain release, not a gbrain-evals release tag. The claimed SOTA LongMemEval _s result (97.60% R@5) compares against MemPalace's published 96.6% rather than a fresh same-environment re-run of MemPalace, which is a methodology caveat worth noting when reading the cross-system comparison table.