A support team wants an LLM to summarise escalated tickets for engineers. The customer has already tried restarting the device, but a related replacement part has not been tested. A fluent summary can accidentally present both steps as completed and send the engineer in the wrong direction. The evaluation should ask what a reader could reasonably conclude about each action.
Design a pair with one meaningful difference
Write two tickets using the same product and symptoms. In the first, the customer tested the replacement and the fault persisted. In the second, the replacement was only proposed. Ask reviewers to label the completed steps, unresolved hypotheses and next appropriate question. Keep proposals distinct from observations when scoring the output. Overlap with an expected sentence cannot establish that the distinction survived.
Run both tickets with the same system instructions and output limits. Have reviewers compare the candidate with the current workflow without seeing the model names. Keep the better summary and the reason for the preference. If reviewers disagree about whether a restart counts as a meaningful diagnostic step, clarify the rubric before interpreting that disagreement as a model error.
Measure what happens at the handoff. Can an engineer identify the untested hypothesis without reopening the whole conversation? Does the summary preserve uncertainty where the customer gave an incomplete answer? Include one representative failure with the score so the team can choose between changing the prompt, improving the input or retaining human review for that ticket type.