language models

  • You can't generate evals on autopilot

    Writing eval criteria is hard, so you ask an LLM to write them. Minutes later, you have hundreds of reasonable-looking rubrics. But whose idea of a correct answer are they measuring?

  • Your eval judge doesn't know what you mean

    Does the solution follow best practices? Your LLM judge answers without hesitation and explains its verdict convincingly. So which best practices did it check?

  • What's your minimal viable model?

    A better AI model appears, so you switch. But how would you know whether its extra intelligence makes any difference to your work?