Built with Evals and benchmarks

jev-column-race

Jev vs Gemini 3.8 Flash: labelling 1,000 app reviews, 4.1× faster and 7× cheaper

23 ★ on GitHub

What it is

Four AI columns over 1,000 real app reviews, raced live: TypeSafe's Jev against Gemini 3.8 Flash.

The homepage listed on the repository, jev-column-race.vercel.app, puts it like this: Column Race

What the README says it does

  • Per-request latency decides the race. Both lanes send 50 requests at concurrency 8. Jev's median request took 528 ms; Gemini's took 2,353 ms. When Jev finished, Gemini had 200 of 1,000 rows.
  • A short tail. Jev's slowest request was 1,516 ms. Gemini's was 8,256 ms, and seven of its requests took over 4 s.
  • Cheap input, free output. Jev costs $0.042 per million input tokens, and output tokens are free. Gemini costs $0.75 per million input and $3.75 per million output. $0.109 of Gemini's $0.158 is output tokens.

From the project’s README.

Where the decision sits

jev-column-race calls the hosted decision API rather than running its own weights. That keeps the project small and means its behaviour moves whenever the vendor ships a new model version. It also means an API key and a network round trip sit in the path of every decision.

Models it references: Jev.

How active it is

Created 2026-09-17, last pushed 2026-09-17, 23 stars and 3 forks when we read it on 2026-11-02. Everything in this category is weeks old at most, so a stale push date means something different here than it would on a mature project. Check the repository before you depend on it.

Find out what your data actually supports

We turn your historical decisions into a frozen test set, then tell you which model, which threshold, and how much of it you can safely automate.