7.a.ii Prompt chaining with gate checks

Split a task that decomposes cleanly into fixed steps into a chain of model calls, one goal per call, with a gate between steps: a schema check, an assertion, or a run against the real system, and an LLM check only for subjective criteria. On a gate failure, rerun only the failed step with its output and the error. The trade is latency and tokens for accuracy and inspectable intermediate outputs; it pays on tasks beyond what one call does reliably, and less as models improve.

Tier 4 because it decomposes one call into a gated chain.

When it applies

In the code: A fixed sequence of model calls with checks between them rather than one prompt.
In traces: Stage outputs; gate failures; retries per stage; the whole chain rerun when one step fails.
In eval results: Per-stage and end-to-end quality.
With a judge: Rate each stage's output.
With probes: Inputs that should fail at each gate.

How

Start from a single call tuned with the same effort; chain only if the task splits cleanly into fixed steps.
Give each step one goal and a handoff the next step can validate against a schema.
Gate in code first: schema validation or assertions; where you can, run the output against the real system (execute the query) instead of asking the model whether it is valid; use an LLM check with hard thresholds only for subjective criteria.
On a failure, rerun only the failed step with its previous output and the error message; cap retries, then stop or continue with a warning; keep outputs that already passed.
Run new gates in warn-only mode first and review each halt for false alarms.
Log each step's output and gate result; compare the chain with the single call on success, tokens and latency, per model.

How to measure it

experiment (tier 4): paired eval of the chain against a single call tuned with the same effort, per model: success, tokens, latency
monitor: gate failure and retry rate per step; false-alarm rate on reviewed halts; each error traced to the step that caused it

To compare the change against your agent's current version, with the same cases and enough repeats to see past run-to-run noise, use aot-evals.

Expected impact

Quality: improves, medium: tasks beyond what one call does reliably: well-formatted JSON 37.6% to 98.8% with assertions and retry (DSPy Assertions, 2023); over a tuned single call, orchestration gained at most 4.6 points (Leins, Aug 2026); 37% of gate halts were false alarms in one study (Elina, Sep 2026)
Cost: degrades, medium: more calls: orchestration took about 2 to 4 times the tokens of a single call (Leins, Aug 2026); a fixed chain that reran downstream steps cost 1,632 tokens per retry against 904 for one prompt, and 436 when only the failed step reran (Asthana, May 2026)
Performance: degrades, medium: serial calls; with gates, one model got faster and the other markedly slower (Dhage, Aug 2026)
Security: improves, small (unmeasured): Gates can check policy.
Governance: improves, medium (unmeasured): Checkpoints are auditable.
Brand: improves, small (unmeasured)

Caveats

Chains cost latency and tokens, and a fixed chain that reruns downstream steps on a failure can cost more per retry than one prompt; rerun only the failed step. Gates misfire: in one study 37% of halts were false alarms, enough to turn a gain into a loss. The benefit depends on the model and shrinks as models improve: Anthropic dropped a step-by-step breakdown when a stronger model arrived, and tuned single calls came within a few points of orchestration. Several studies are small (one app, one run).

Evidence

Evidence comes from: Structured output, Retrieval, Data analysis, Code generation, Reasoning, Incident response.

Lists this practice in its agent-optimization pattern library (Workflow Patterns group)Not measured

Names the practice as a known agent optimization; reports no measured effect for it.

Scale AI, "VeRO: A Harness for Agents to Optimize Agents" (ICML 2026) · Feb 2026 · paper Table 6; pattern library file master-agent-optimization-cookbook.md
Describes prompt chaining with programmatic gates between stepsNot measured

"You can add programmatic checks (see "gate" in the diagram below) on any intermediate steps to ensure that the process is still on track." Use it when "the task can be easily and cleanly decomposed into fixed subtasks"; "The main goal is to trade off latency for higher accuracy, by making each LLM call an easier task." No measurement.

Anthropic (Erik Schluntz, Barry Zhang), "Building effective agents" · Dec 2024Older evidence · Workflow: Prompt chaining
Introduces chaining LLM steps so each output feeds the next

"In a 20-person user study, we found that Chaining not only improved the quality of task outcomes, but also significantly enhanced system transparency, controllability, and sense of collaboration."

Tongshuang Wu, Michael Terry, Carrie J. Cai, "AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts" · Oct 2021Older evidence · Abstract
Assertions between pipeline modules; on failure, backtrack and retry the failing module with the erring output and error message in its prompt

Well-formatted JSON quizzes 37.6% to 98.8% with assertions and backtracking; valid quiz questions 30.5% to 87.2% with all assertion-driven optimizations. Across four case studies, constraints passed "up to 164% more often" and responses were "up to 37% more higher-quality".

Singhvi et al., "DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines" · Dec 2023Older evidence · Abstract; §1; §5
Draft, critique and refine as three chained prompts against the same steps in one stepwise prompt

Chained refinement with GPT-4 won 77 of 100 comparisons against a one-step GPT-4 summary; the stepwise prompt won 53. The authors suggest the stepwise prompt "might produce a simulated refinement process".

Sun et al., "Prompt Chaining or Stepwise Prompt? Refinement in Text Summarization" · Jun 2024Older evidence · Abstract; Table 1
Fixed planner, generator and evaluator steps with hard-threshold gates per sprint, against one agentMixed

Each criterion "had a hard threshold, and if any one fell below it, the sprint failed". One app: solo 20 min and $9, full harness 6 hr and $200, with better output judged by the author. The sprint breakdown was later removed for a stronger model; the evaluator "is worth the cost when the task sits beyond what the current model does reliably solo."

Prithvi Rajasekaran (Anthropic), "Harness design for long-running application development" · Mar 2026 · Running the harness; Removing the sprint construct
Two-stage text-to-SQL pipeline gated by executing the query; on failure or an empty result, retry with the database error

The retry loop "yields up to +9.3pp accuracy gains with zero regressions on the best model (Llama 4 Scout 17B, 57.3%)". Passing queries are accepted without re-evaluation, and the best partial result is kept across retries.

Ijaz, "SQL Query Engine: A Self-Healing LLM Pipeline for Natural Language to PostgreSQL Translation" · Apr 2026 · Abstract
One prompt against a fixed-step chain and against steps with schema-validated outputs where only failed steps rerunMixed

Retry cost on Kubernetes root cause analysis: 1,632 ± 145 tokens for the fixed chain against 904 ± 17 for one prompt, "because failures forced reruns of downstream subtasks"; 436 ± 132 when only failed subtasks reran. Multi-file debugging: 933, 703 and 460 tokens.

Asthana, Zhang, DeLuca, Patel, et al., "Runtime-Structured Task Decomposition for Agentic Coding Systems" · May 2026 · Abstract
Self-Refine (draft, critique, refine), Best-of-N and Debate against single calls, each tuned with GEPA under the same budgetMixed

Averaged across backbones, the largest gain was 4.6 points over optimized chain-of-thought and 4.5 over task-only inference, "while requiring approximately 2 to 4 times the mean total tokens"; gains were benchmark-dependent and did not grow with task difficulty.

Leins, Pelleriti, Gonnermann-Müller, Pokutta, "When Does LLM Orchestration Pay Off? A Controlled Evaluation of Accuracy, Cost, and Task Difficulty" · Aug 2026 · Abstract; §4
Wrap an agent in fixed states, output validation, bounded retry, then a schema check on the plan before any tool runsMixed

Validation and bounded retry alone improved reproducibility in one of four model-task cells and degraded it in two. With the plan checked against a schema, three of four cells reached a reproducibility rate of 1.000 and task success of 100%. Token cost fell in every cell; latency fell for one model and rose markedly for the other.

Dhage, "Harness Engineering for Predictable Agentic Systems: An Empirical Study of Deterministic Execution Constraints" · Aug 2026 · Abstract; §6
Check every agent-to-agent handoff against a workflow contract in code, with no model in the checkMixed

On paired trials, rubric score +7.7% to +29.1% in five workflows and flat in one; model-call cost fell 17-53% where halts came early. Of 94 halts, 35 (37%) were false alarms from validator defects; counting them as failed work, the gated runs scored below ungated ones in four of six workflows.

Elina, "Maat: Independent Deterministic Contract-Based Governance for Multi-Agent LLM Workflows" · Sep 2026 · Abstract; §6

9.a.iFeed back a sound external verifier's verdict; never the model's own critique
7.c.iiiEvaluator-optimizer loop
5.c.iDiscrete phase separation
4.b.iiInitializer-maintainer dual agent
7.d.iiInversion of control

In Claude Code

This optimization is a skill in aot-kit, our open-source Claude Code plugin: /aot-kit:prompt-chaining-with-gates. Its library entry holds every source. Have new evidence, or a correction? Suggest a change.

Prompt chaining with gate checks