language models
I let LLMs play a 1988 detective gamebook 646 times
In a hot-air balloon with a storm coming, almost every model made the sensible choice and landed. About ten decisions later, that choice cost them the case.
You can't generate evals on autopilot
Writing eval criteria is hard, so you ask an LLM to write them. Minutes later, you have hundreds of reasonable-looking rubrics. But whose idea of a correct answer are they measuring?
Your eval judge doesn't know what you mean
Does the solution follow best practices? Your LLM judge answers without hesitation and explains its verdict convincingly. So which best practices did it check?
What's your minimal viable model?
A better AI model appears, so you switch. But how would you know whether its extra intelligence makes any difference to your work?
Benchmark models using OpenAI-compatible APIs
Learn to benchmark language models with OpenAI-compatible APIs using our updated Jupyter Notebook for optimal performance evaluation.