Built with Evals and benchmarks
typed_evals
Fast, typed Python framework for Jev-as-a-Judge/Decision-model evaluation for LLMs, RAG, and AI agents along with human-label calibration.
17 ★ on GitHub
What it is
Evaluate responses. Guard actions. A Python toolkit for checking LLM answers, evaluating agent runs, and reviewing tool calls before execution
Getting started
From the README:
pip install typed-evals
export TYPESAFE_API_KEY='your-key'Where the decision sits
typed_evals calls the hosted decision API rather than running its own weights. That keeps the project small and means its behaviour moves whenever the vendor ships a new model version. It also means an API key and a network round trip sit in the path of every decision.
Models it references: Jev, jev, laya, laya.
How active it is
Created 2026-09-20, last pushed 2026-10-29, 17 stars and 1 forks when we read it on 2026-11-02. Everything in this category is weeks old at most, so a stale push date means something different here than it would on a mature project. Check the repository before you depend on it.
Find out what your data actually supports
We turn your historical decisions into a frozen test set, then tell you which model, which threshold, and how much of it you can safely automate.