Evals

Each eval is a public repository pinned at one commit, with a manifest that says how it runs and how it is checked. A new commit is a new eval version.

Coming soon

Community evals

darkmux evals is meant to be the community’s: people who run models locally define the evals, and share results from their own Apple Silicon Macs, so the same task can be read across chips, memory sizes and engines. Until then, the evals here are approved by the darkmux maintainers.

  1. 1Define an eval: a public repository pinned at one commit, with a manifest that says how it runs and how it is checked.
  2. 2Run it with darkmux on your Mac and share the result with a free account.
  3. 3Read it beside everyone else’s, machine by machine and engine by engine.