Why agent distillation needs separate success, constraint, and cost metrics

Distillation results often arrive as one number: final-task accuracy. That number is useful, yet it hides the reason a small model succeeded, the constraints it violated on the way, and whether the resulting system fits the deployment budget. When I read the recent work on reasoning and agent distillation, the missing part is usually the reporting scheme rather than another training recipe.

The scale reduction can be substantial. Distilling Step-by-Step reports a 770M T5 student outperforming a few-shot prompted 540B PaLM model on one benchmark while using 80% of the available data. That difference equates to roughly 700 times fewer parameters; a remarkable transfer of capability from one model to another. The method extracts natural-language rationales from the teacher and trains the student to predict both the answer and the explanation. Fine-tune-CoT and Symbolic Chain-of-Thought Distillation follow the same general direction, using teacher-generated reasoning paths and, in the latter case, many sampled chains per instance. Model specialization makes the trade-off explicit: concentrating a small model on multi-step mathematics improves that target ability while reducing generic capability.

These papers establish a strong case for transferring a bounded skill into a smaller student. They also show why accuracy cannot be the only result. A student can produce the correct label with an invalid rationale, or preserve the answer while losing the general ability that made the original model useful elsewhere.

The tool-use literature moves the unit of transfer closer to behavior. Toolformer trains a model to decide which application programming interface (API) to call, when to call it, what arguments to pass, and how to use the result. AgentTuning trains Llama 2 models on filtered interaction trajectories mixed with general instructions, targeting planning, memory, and tool utilization. More recently, Agent Distillation transfers reason-act-observe behavior into 0.5B, 1.5B, and 3B students equipped with retrieval and code tools. The evaluation still centers on whether the task was solved.

For a bounded workflow, I would report four metrics separately. Consequently, the evaluation table describes both the result and the path used to obtain it:

(1) Task success. Did the system return the required result, under the same test distribution and budget as the teacher? (2) Constraint adherence. Did it use only the permitted tools, state transitions, files, and action formats? A correct final answer reached through an invalid transition should be recorded as a constraint failure. (3) Trajectory quality. Were the intermediate actions relevant, consistent, and grounded in the available state? ROSCOE already provides metrics for semantic consistency, logicality, informativeness, fluency, and factuality in reasoning paths. A workflow version would add tool-call validity and state-transition checks. (4) Deployment efficiency. What were p50 and p95 latency, peak memory, energy or cost per successful task, and the training cost of generating and filtering teacher trajectories?

A composite score can help rank systems after these measurements exist. It should not replace them. A single weighted number makes it impossible to tell whether a student is cheaper because it is faster, because it uses fewer tools, or because it silently violates the workflow. The distinction matters when the workflow controls a real experiment or a production change.

I want to apply this table to the autoresearch keep-or-discard loop and to the coordination levels in my agent orchestration guide. For the first, success is an improved validation score, constraint adherence is respecting the editable-file and timeout rules, trajectory quality is the usefulness of each experiment, and efficiency is cost per accepted improvement. Further work should measure those four columns together and test whether the ranking of small students changes once invalid but accurate runs are counted.