Built with Evals and benchmarks
jev-column-race
Jev vs Gemini 3.8 Flash: labelling 1,000 app reviews, 4.1× faster and 7× cheaper
23 ★ on GitHub
What it is
Four AI columns over 1,000 real app reviews, raced live: TypeSafe's Jev against Gemini 3.8 Flash.
The homepage listed on the repository, jev-column-race.vercel.app, puts it like this: Column Race
What the README says it does
- Per-request latency decides the race. Both lanes send 50 requests at concurrency 8. Jev's median request took 528 ms; Gemini's took 2,353 ms. When Jev finished, Gemini had 200 of 1,000 rows.
- A short tail. Jev's slowest request was 1,516 ms. Gemini's was 8,256 ms, and seven of its requests took over 4 s.
- Cheap input, free output. Jev costs $0.042 per million input tokens, and output tokens are free. Gemini costs $0.75 per million input and $3.75 per million output. $0.109 of Gemini's $0.158 is output tokens.
From the project’s README.
Where the decision sits
jev-column-race calls the hosted decision API rather than running its own weights. That keeps the project small and means its behaviour moves whenever the vendor ships a new model version. It also means an API key and a network round trip sit in the path of every decision.
Models it references: Jev.
How active it is
Created 2026-09-17, last pushed 2026-09-17, 23 stars and 3 forks when we read it on 2026-11-02. Everything in this category is weeks old at most, so a stale push date means something different here than it would on a mature project. Check the repository before you depend on it.
Find out what your data actually supports
We turn your historical decisions into a frozen test set, then tell you which model, which threshold, and how much of it you can safely automate.