Home / Forward deployed engineering / LLM evaluation
Test the task, not the demonstration
LLM evaluation
LLM evaluation tests how a language-model workflow behaves on a defined task and under documented conditions. A useful evaluation examines the complete system: its sources, instructions, tools, output and human review. This guide explains a practical test-set method. It is different from a consulting service page or a leaderboard of model scores, and it does not claim that a small set of successful examples establishes general reliability.
Create a repeatable evaluation record that supports a specific release or change decision.
NIST describes its AI risk framework as voluntary guidance for design, use and evaluation. NIST: AI Risk Management Framework ↗
Government guidance explains retrieval-augmented generation and discusses evaluating retrieval and generation separately. GOV.UK: AI Insights — RAG systems ↗
New to the topic? Read the background on artificial intelligence.
Test the task, not the demonstration
what should the evaluation measure?
Define the intended task and unacceptable failures before selecting metrics.
An illustrative document assistant may need to cite the correct policy, preserve uncertainty and decline unsupported requests. A generic quality score can obscure those requirements. The voluntary NIST framework provides risk-management context, while AI evaluation consulting helps deliver a task-specific assessment. State who reviews the output and which decision the evaluation supports, such as accepting a prototype or approving a source change.
Keeping these checks running after release is the job of an AI operations consultant.
Define the task.
Describe material failures.
Name the decision supported.
Test the task, not the demonstration
how should a test case be recorded?
Keep the input, expected behaviour, evidence and reviewer criteria together.
A useful case includes the source version, relevant user permissions and what the system should do when the answer is absent. Avoid requiring identical wording where several answers could be valid. Record mandatory facts, prohibited claims and the intended escalation. AI knowledge base describes retrieval-specific dependencies. Keep a separate set of examples outside development so the team can inspect whether an apparent improvement applies beyond the prompts and sources used during tuning.
Test the task, not the demonstration
what failures should the set include?
Add normal work, ambiguous requests, missing sources and failure paths.
For a document assistant, include outdated material and conflicting versions. For an agent, include a failed tool, a repeated action and a request outside its permitted scope. AI agent development services defines that action boundary. The GOV.UK RAG guidance discusses evaluating retrieval and generation separately. Check the evidence selected and the final answer rather than assuming a plausible citation proves the complete response is supported.
| Check | What to establish |
|---|---|
| 1 | Include counterexamples. |
| 2 | Inspect retrieval and answer stages. |
| 3 | Test failed and repeated actions. |
Test the task, not the demonstration
how should people and automated checks work together?
Automate clear checks and preserve human judgement for contextual quality and consequences.
A field-format check can be automated, while deciding whether a caveat has been misleadingly omitted may need a knowledgeable reviewer. A model-based judge can assist a review but its decisions need validation against the task’s criteria. AI risk assessment helps prioritise failures by consequence. Keep the reviewer’s reason, disagreements and unresolved cases so the evaluation remains inspectable rather than becoming an unexplained number.
Test the task, not the demonstration
when should the evaluation be repeated?
Rerun relevant cases after material changes and compare the complete task outcome.
A new model, instruction, tool permission or source set can alter behaviour. Record the version tested, observed failures and review effort. AI implementation roadmap places evaluation at delivery gates; AI operations consulting supports continuing checks. An improved average result should not conceal a new consequential failure. The release owner should decide whether to accept, narrow, investigate or revert the change using the evidence and its coverage limits.
Track tested versions.
Inspect regressions.
Record the release decision.
Make the decision concrete
What does the current task cost in time?
Choose a task connected to llm evaluation and enter its current workload. This gives you a baseline to discuss before buying training or commissioning a build.
A baseline is useful even if you decide to keep the process manual. Measure the work that matters, then compare quality and review effort as well as time.
Assess the workflow before changing it, or plan business AI training if the gap is how people use existing tools.
Your current workload
Estimate how much time the existing task takes before discussing llm evaluation.
16 hours per week
People × minutes × weekly tasks ÷ 60. A workload baseline from your inputs, not a prediction of AI time savings. Include review and support time when comparing a proposed workflow.
A useful AI investment leaves your team able to do the work and judge the result.
Continue with the right question
Where should you go next?
Use the overview for context, then follow the pages that match your delivery decision.
See our forward deployed engineers service or read the forward deployed engineering overview.
FAQ
Questions, answered
Evidence and further reading
Which sources support the guidance?
Source links support the specific background described here. The practical checklists and illustrative examples are predeploy’s guidance.
NIST describes its AI risk framework as voluntary guidance for design, use and evaluation. NIST: AI Risk Management Framework ↗
Government guidance explains retrieval-augmented generation and discusses evaluating retrieval and generation separately. GOV.UK: AI Insights — RAG systems ↗
Background reading on the broader field of artificial intelligence. Wikipedia: artificial intelligence ↗
Choose your next step
Explore forward deployed engineering
Start with a topic overview, or go directly to the question your team needs to answer.
Forward deployed engineering
Understand the role, compare hiring routes and commission an FDE.
Start with the decision
More questions in this topic
- Forward deployed engineering
- Hire forward deployed engineers
- How to hire forward deployed engineers
- Forward deployed engineers for hire
- Forward deployed AI engineer
- Forward deployed software engineer
- Forward deployed engineer vs solutions engineer
- AI agent development services
- AI knowledge base
- AI chatbot development services
- Claude Code training
