# Agent optimizations (Mahal Systems): the signs each one applies

A catalog of every agent optimization Mahal Systems knows of: what each change is, where in the agent it lands (the lever), what kind of work it is specific to (the domain), the signs that it applies, how to make it, how to measure it, and the published evidence with dates and sources.

It is a catalog, not a list of recommendations: most of these changes help some agents and hurt others, and much of the evidence comes from coding and web-research benchmarks. Whether one helps a particular agent is something to measure on that agent.

Each entry below links to its full page in Markdown (how to make the change, how to measure it, the evidence). Everything in one file: https://mahalsystems.ai/aot/optimizations/llms-full.txt

## [1.b.i Match the model and its effort to each call site and query class](https://mahalsystems.ai/aot/optimizations/model-and-effort-per-call-site.md)
Tier 3 (Tuning) · levers: Model, Prompt · domains: General
Pick the model per role or call site rather than one model everywhere, and set its reasoning effort and sampling parameters per site; tune each role's prompt to its model (GEPA raised every model tested at lower cost).
- Signs in code: one model and one effort setting for every call site; no per-query-class budget
- Signs in traces: simple queries using as many steps, tokens or sub-agents as complex ones; a call site whose cost dominates with no quality reason; retries exhausted or timeouts at one site
- Signs in evals: quality and cost per call site, per model; price not predicting quality across the measured configurations (in an email-triage agent we measured: Spearman 0.12)

## [2.a.i Reduce the tool surface: remove and consolidate tools, then defer the rest behind search](https://mahalsystems.ai/aot/optimizations/reduce-tool-surface.md)
Tier 1 (Hygiene) · levers: Tool surface, Context · domains: Tool use
Offer the agent only the tools the task needs.
- Signs in code: more than about 10 tools, or more than about 10k tokens of tool definitions, in every prompt; duplicate or near-duplicate tool names or descriptions (73% of audited MCP servers have repeated names); tools the agent's credentials cannot use
- Signs in traces: tools never called across the eval runs; tool-use coverage below the eval's needs (share of needing cases where the right tool was loaded and chosen; GitHub measured 69% for a static list); calls to a tool where none was needed; failure rows labeled incorrect-tool-selection concentrated on one pair of tools
- Signs in evals: accuracy falling as tools are added (an ablation over catalog size); paired eval, current tool set against a reduced one

## [2.a.ii Rewrite the tool surface: names, schemas, descriptions](https://mahalsystems.ai/aot/optimizations/rewrite-tool-surface.md)
Tier 1 (Hygiene) · levers: Tool surface · domains: Tool use
Change what the model sees of each tool without touching its implementation: a distinct name, a strict schema (enums, required fields), and a description with fixed sections (what it is for and when to use it versus sibling tools, defaults and limits, error modes, side effects), plus a usage example for complex parameters.
- Signs in code: descriptions that do not state the tool's purpose (56% of audited MCP tools) or when to use it; duplicate or overlapping names; schemas without enums or required fields; no example for complex parameters
- Signs in traces: misrouted cases per tool (false positives and false negatives); failure rows labeled incorrect-tool-selection, malformed-arguments or suboptimal-arguments; per-tool error rate and retries (Datadog tracks these per tool and server)
- Signs in evals: failures attributed to tool selection or arguments; step count per task after a description change

## [2.a.iii Choose how the agent acts: typed tools, code execution, or shell, matched to the model and task](https://mahalsystems.ai/aot/optimizations/choose-action-interface.md)
Tier 3 (Tuning) · levers: Tool surface, Model · domains: Tool use
Decide the action interface deliberately, by model and by task, and provision what it needs.
- Signs in code: the interface in use (typed tools, code interpreter, bash), the model it serves, and whether the environment has the packages and fixtures the tasks need
- Signs in traces: bash-only runs emitting tools that do not exist; typed-tool runs re-patching the same file; code errors in routine operations (reading a file, searching); import-error or command-not-found spans at the start of runs; calculations done in the model's own text; independent commands serialized
- Signs in evals: success and cost by model under each interface

## [2.b.i Make tool calls and tool results fail soft, and retry transient failures](https://mahalsystems.ai/aot/optimizations/fail-soft-tools.md)
Tier 2 (Reliability) · levers: Tool surface, Control flow · domains: Tool use
Keep a failed call from ending the run: validate tool calls and model responses, return tool errors to the agent as results, and retry only failures that are transient and safe to repeat.
- Signs in code: parsing that assumes a fixed call shape; model responses used without checking finish_reason, empty content or tool-call arguments against the schema; tool bodies with no try/except; custom HTML or table extraction code; retry wrappers that retry every error, including permanent ones and writes without an idempotency key the server honors
- Signs in traces: runs ending in an unhandled exception (AttributeError, parse or fetch errors) rather than an answer; tool spans returning empty or garbled text; failure rows labeled tool-recovery-failure or observation-failure; requests failing on rate limits or transient server errors with no retry, or with retries exhausted; failure rows labeled service-failure or recovery-failure-external; runs ended by the harness rather than the model (a repeat detector after malformed calls, a provider stream error, the output cap); three or more consecutive calls to a tool that keeps failing; a write retried after a timeout and committed twice
- Signs in evals: crash outcomes as a share of failures; request-failure outcomes
- Signs in probes: Faults at the tool and model-API boundaries: malformed calls, errors, timeouts, empty and truncated responses, permanent failures, and lost replies on writes.

## [2.b.ii Tell the agent when to use each tool, in context it always sees](https://mahalsystems.ai/aot/optimizations/tool-use-guidance.md)
Tier 3 (Tuning) · levers: Tool surface, Prompt · domains: Tool use
For each tool or skill that is under-used or misused, add an explicit trigger rule where the model always sees it (system prompt, AGENTS.md): the situation that calls for the tool, and what to use it instead of (e.g.
- Signs in code: tools or skills whose intended use is stated nowhere the model always sees; overlapping tools with no rule for choosing between them
- Signs in traces: tool spans: a tool called on under about 5% of cases where its trigger condition held (e.g. a docs skill never invoked in 56% of runs); bash spans where a dedicated tool exists; failure rows labeled incorrect-tool-selection
- Signs in evals: failures where the right tool existed but was not called

## [2.c.i Bound tool output: cap it, or write it to a file the agent can read](https://mahalsystems.ai/aot/optimizations/bound-tool-output.md)
Tier 3 (Tuning) · levers: Tool surface, Context · domains: General
Cap the characters of tool output kept per call (VeRO settled near 15-20 KB on TerminalBench), or, better, write long output to a file and return its path with a short head, so the agent can read more if it needs to.
- Signs in code: tool results passed to the model whole, with no cap or file offload
- Signs in traces: a single tool result over about 10% of the context window; context-overflow outcomes; a truncated result followed by re-running the same command
- Signs in evals: context-overflow outcomes as a share of failures

## [3.a.i Add worked examples to the prompt (few-shot), or curated example queries for data agents](https://mahalsystems.ai/aot/optimizations/few-shot-examples.md)
Tier 3 (Tuning) · levers: Prompt · domains: General
Put a few solved examples of the task in the prompt (the input and the exact output wanted), or give a data agent commented example queries for its custom metrics and an index of past queries it retrieves per question.
- Signs in code: A prompt with no worked examples on a task with a fixed output shape; a data agent with no example queries for its custom metrics; examples kept on a reasoning or strong model with no zero-shot comparison; examples retrieved per request placed before static prompt content, so the cached prefix changes on every call.
- Signs in traces: Outputs that miss the expected shape or metric definition; repeated queries for the same custom metric; retrieved examples with low similarity to the question, or answers that copy a retrieved pattern that does not fit.
- Signs in evals: Failures that cluster on custom metrics, business logic or an output format the examples would show; no paired zero-shot baseline for the examples in use.

## [3.a.ii Avoid few-shot ruts](https://mahalsystems.ai/aot/optimizations/vary-few-shot-examples.md)
Tier 3 (Tuning) · levers: Prompt · domains: General
An agent's own history, and any examples in its prompt, act as few-shot demonstrations: when the context fills with near-identical action-observation pairs (twenty similar items in a row, a retried call), the model tends to copy their surface form instead of judging the current item.
- Signs in code: Every action and tool result serialized with one fixed template; harness messages with identical wording; few-shot examples that share one format and one structure or were picked as the most similar to the input; many similar items processed in one context; failed calls replayed verbatim in the history.
- Signs in traces: Identical or near-identical consecutive actions; repeats of a call that already failed; later outputs that copy wording or structure from earlier items; decisions that drift with position in the batch or toward the batch majority, or that differ from the same item run alone.
- Signs in evals: Accuracy and outlier detection by position in the batch; decision flip rate against each item run alone; spread across several equivalent templates.
- Signs in judge: Detect over-generalization: later outputs that carry over content or conclusions from earlier items instead of the current one.
- Signs in probes: A batch of similar items with one outlier.

## [3.a.iii Dynamic context injection](https://mahalsystems.ai/aot/optimizations/inject-runtime-context.md)
Tier 3 (Tuning) · levers: Prompt, Context · domains: General
Put the runtime facts the model cannot know into its context each run: today's date and timezone, the working directory, OS and git state, the tools available, the user and locale.
- Signs in code: Prompt templates contain runtime placeholders (date, time, user, locale, tool availability). Absence where tasks are time- or user-sensitive is the signal.
- Signs in traces: Model guesses the date or year, asks the user for information the system already has, builds queries with stale dates.
- Signs in evals: Failures concentrated on time-sensitive or user-specific items.
- Signs in judge: Flag outputs that assume stale state.
- Signs in probes: Tasks whose answer depends on today's date, relative dates, locale or the current user.

## [3.a.iv Restructure the system prompt: every requirement once, up front, in named sections; nothing the model doesn't need](https://mahalsystems.ai/aot/optimizations/restructure-system-prompt.md)
Tier 3 (Tuning) · levers: Prompt · domains: General
Rewrite the system prompt so that each requirement the task has (constraints, the answer format, the completion check, how to work efficiently, the domain procedure) is stated once, up front, in a named section, and remove instructions the model does not need.
- Signs in code: system prompt with no named sections; requirements repeated, contradictory, or absent (no answer format, no completion criteria); prompt plus tool descriptions over about a quarter of the context window
- Signs in traces: failure rows labeled instruction-following-failure or instruction-grader-mismatch where the requirement was never in the prompt; prompt tokens the largest share of input per step; context-overflow outcomes
- Signs in evals: an ablation that removes a section and shows no loss (cut it), or failures that cluster on one step of the task procedure (add it)

## [3.b.i Keep the prompt prefix cache-stable](https://mahalsystems.ai/aot/optimizations/cache-stable-prefix.md)
Tier 1 (Hygiene) · levers: Context · domains: General
Put static content (identity, rules, tool definitions) before a boundary and per-turn content (date, state, memory) after it; keep the prefix bit-for-bit identical across turns; strip thinking text from persisted history; and track per-turn cache-miss cost.
- Signs in code: a timestamp, session id or per-turn state before the static block; non-deterministic serialization; a marked-for-caching prefix shorter than the provider's minimum (in one image-labeling agent, 504 tokens against Haiku 4.5's 4,096)
- Signs in traces: cache_read tokens near zero across calls that share a prefix; cache-hit rate below about 50% on multi-turn runs

## [3.b.ii Mask tools instead of removing them](https://mahalsystems.ai/aot/optimizations/mask-tools-not-remove.md)
Tier 3 (Tuning) · levers: Context, Tool surface · domains: General
When the tools an agent may use change mid-task (by mode, phase or permission), declare every tool once, keep the definitions byte-stable, and restrict the choice another way: OpenAI's allowed_tools or tool_choice "none", Anthropic's appended tool_removal and tool_addition blocks, a decode-time logit mask or grammar on self-hosted models, and a harness check that rejects calls outside the allowed set.
- Signs in code: Tool list rebuilt per turn from mode, phase, permissions or connected servers (a tools array filtered by state, or prompt text that co-varies with the active toolset); no allowed-tools or tool_choice restriction, appended tool_removal or tool_addition blocks, decode-time mask, or harness check of each call against the allowed set.
- Signs in traces: A different tools-array hash between turns of one session, followed by cache reads near zero (provider cache diagnostics name the tools as the point of divergence); on Claude, tool_choice changing between turns; calls to tools no longer defined or not allowed; on recent Claude models, 400 errors on replayed thinking blocks after a tool edit.
- Signs in evals: Cache hit rate; invalid tool call rate.
- Signs in probes: Sessions where a tool becomes unavailable midway.

## [3.c.i Self-Discover reasoning structures](https://mahalsystems.ai/aot/optimizations/self-discover-reasoning.md)
Tier 3 (Tuning) · levers: Model, Prompt · domains: Reasoning
Self-Discover has the model compose a reasoning structure once per task type (select useful modules from 39 seed heuristics, adapt them to the task, write them as a step-by-step plan), then follow that cached structure on every instance, so each instance costs one call.
- Signs in code: A module-selection or planning step before solving, or a reasoning structure (key-value or numbered steps) per task type in the prompt. Flag a structure regenerated for every instance where one cached per task type would do, JSON structures where a natural-language plan would do, a structure stacked on a reasoning model with thinking on that was never compared against thinking alone, and no CoT or self-consistency baseline.
- Signs in traces: Structure id and version per call and the task type it was discovered for; modules selected; whether the output fills every key or rephrases or skips them; tokens and latency per call against plain CoT.
- Signs in evals: Accuracy per task type against CoT, the model's own thinking at matched tokens and self-consistency at matched cost; computation-heavy split from knowledge-heavy, short answers from long-form.
- Signs in judge: Before caching, check each discovered structure: would following it solve labeled examples of the task? (12.5% of MATH structures were wrong in the original study.)
- Signs in probes: A mix of task types needing different reasoning.

## [4.b.i Filesystem-based agent state and checkpoints](https://mahalsystems.ai/aot/optimizations/filesystem-checkpoints.md)
Tier 2 (Reliability) · levers: State · domains: General
Save the agent's state at each step boundary (the conversation, the workspace files, process state when it matters, and a ledger of external side effects) so a crashed or stuck run resumes or rolls back instead of starting over.
- Signs in code: What a checkpoint covers (transcript only, workspace files, sandbox processes, a ledger of external effects) and when it is written; whether resume serves recorded tool results or re-executes them; whether conversation and workspace can be restored from different points; whether a parked step or approval is claimed atomically before it runs; resume logic on start.
- Signs in traces: Resume and rollback events; steps re-executed after a resume; an external effect issued twice for one step; checkpoint latency exposed on the critical path; rollbacks per run and what followed them.
- Signs in evals: Recovery from injected crashes (a real kill, not only an exception) on tasks the agent already solves, against restart and transcript-only resume: recovery correctness, redo time and tokens, duplicate effects in an external ledger.
- Signs in probes: Kill mid-run and resume.

## [4.b.ii Initializer-maintainer dual agent](https://mahalsystems.ai/aot/optimizations/initializer-maintainer-agents.md)
Tier 4 (Restructuring) · levers: State · domains: Planning, Code generation
Give the first session of a multi-session build a setup-only prompt (expand the spec into a JSON list of end-to-end tests, all failing, and write an init script, a progress file and a first commit) and every later session a maintainer prompt that reads that state, re-tests what passed, builds one failing item, verifies it end to end and commits.
- Signs in code: Two entry points; a progress file and feature list; bootstrap and per-session flows.
- Signs in traces: Session handoffs; progress file updates; whether each session starts by reading state and running the smoke test; items removed or reworded in the feature list; sessions declaring the project done with items failing; a first-session plan never revised after tests expose new defects.
- Signs in evals: Multi-session completion rate.
- Signs in probes: A multi-session project.

## [4.b.iii Keep plan, progress and results in files when work outlives one context window](https://mahalsystems.ai/aot/optimizations/multi-window-task-state.md)
Tier 4 (Restructuring) · levers: State · domains: Planning
When a task spans context resets (compaction, a truncation limit, a new session), have the agent keep what the next window needs in files and read them back at the start of each window: the plan and a progress list, decisions and open items, intermediate results too large to hold in context, and for coding a structured test list and an init script.
- Signs in code: A plan or todo file, a progress log, a decisions and open-items file, and for coding a structured test list and an init script, kept outside the context window, plus a start-of-window step that reads them back.
- Signs in traces: Context resets followed by re-exploration or redone steps; notes written but never read back; user constraints or decisions violated after compaction; work declared done after a reset; state files older than the code they describe.
- Signs in evals: Completion rate and redone work on tasks longer than one window, with a reset forced midway.
- Signs in probes: Tasks longer than one context window.

## [4.c.i Episodic memory retrieval](https://mahalsystems.ai/aot/optimizations/episodic-memory.md)
Tier 4 (Restructuring) · levers: State · domains: Memory
Log every agent session as a raw trace, distill a few verified lessons from them (user corrections, definitions, workflows that worked, mistakes) into short memory files, show the agent an index of those files at the start of each task, and let it search both.
- Signs in code: Raw session traces kept across tasks or sessions; distilled memory files or a vector store of lessons; an index or retrieval step at task start; writes after each task or on user request; a consolidation, TTL or decay job.
- Signs in traces: Memories retrieved per task, whether the agent read and followed them, and whether they applied; memory writes and their source; repeated mistakes the store should have prevented.
- Signs in evals: Repeat-mistake rate over time; pass rate with and without memory on a task stream, split into rescues and regressions.
- Signs in judge: Rate retrieved memories for relevance.
- Signs in probes: A task pair where the first teaches a lesson the second needs, an unrelated task the lesson could mislead, and untrusted content that asks to be remembered.

## [4.c.ii Skill library evolution](https://mahalsystems.ai/aot/optimizations/skill-library.md)
Tier 4 (Restructuring) · levers: State · domains: Memory
Have the agent turn runs that passed a verifier into reusable skills (a code function or a SKILL.md procedure), store them in a library, and on later tasks show it only the few a separate selection step picks, reviewing what goes in and pruning what goes unused.
- Signs in code: A skills directory or store the agent writes to after runs; skills written without a verifier outcome, or from the model's own knowledge before it attempts the task; new skills retrievable with no review or quarantine; authoring that reads retrieved skills as templates; every skill's name and description listed to the agent on every run.
- Signs in traces: Skill reuse rate; skills authored per session; loaded-skill recall and zero-load rate per run; per-skill opens and outcomes; skills never opened; skills distilled from failed or unlabeled runs; authored skills that copy retrieved code verbatim.
- Signs in evals: Paired runs with and without the library on held-out tasks from recurring families (pass rate, tokens, time); pass rate and loaded-skill recall as the library is padded with irrelevant skills.
- Signs in judge: Review each new skill before it becomes retrievable: task-specific paths or values baked in, confident assumptions no verifier checked, boilerplate copied from retrieved skills, overlap with existing skills.
- Signs in probes: Recurring task families; a growing library; planted skills.

## [5.a.i Compact conversation history in stages: elide stale tool outputs, then keep an anchored summary](https://mahalsystems.ai/aot/optimizations/staged-compaction.md)
Tier 2 (Reliability) · levers: Context · domains: General
Manage the history before it overflows.
- Signs in code: no compaction, or compaction that summarizes the whole history at once; a context window under about 64k
- Signs in traces: context-overflow outcomes; peak context over about 85% of the window; summaries that drop file changes or decisions; failure rows labeled context-rationale-erosion
- Signs in evals: failures concentrated in the longest trajectories

## [5.b.i Give the model the schema it queries, in context: descriptions, keys, codes, sample values](https://mahalsystems.ai/aot/optimizations/ground-queries-in-schema.md)
Tier 3 (Tuning) · levers: Knowledge access, Context · domains: Data analysis
When the model writes SQL or structured filters, put the schema it is querying in its context: table and column descriptions, key relationships, what enum codes and nulls mean, sample values.
- Signs in code: a SQL-writing agent with no table and column descriptions, keys or allowed values in its context; schema context reachable only through a search tool
- Signs in traces: queries against non-existent columns, wrong joins, guessed enum codes; failure rows labeled tool-hallucination (names) or domain-knowledge-deficit
- Signs in evals: failures from wrong table, column or value

## [5.c.i Discrete phase separation](https://mahalsystems.ai/aot/optimizations/discrete-phase-separation.md)
Tier 4 (Restructuring) · levers: Context · domains: General
Run research, planning and implementation as separate agent sessions, each starting from a fresh context that holds only the previous phase's written output (a research document, then a plan file), and have a person review that document before the next phase starts.
- Signs in code: Phases with distilled handoffs rather than shared history.
- Signs in traces: Phase boundaries and what crosses them; handoff size against the full history; files the implement phase re-reads that research already covered; edits made to the plan at review; research or plan files older than the code they describe.
- Signs in evals: Quality vs monolithic run.
- Signs in judge: Rate handoffs for completeness.
- Signs in probes: Multi-file tasks whose request leaves out requirements the repository reveals, where a design decision must precede coding.

## [5.c.ii Sub-agent architectures with clean contexts](https://mahalsystems.ai/aot/optimizations/isolated-subagent-context.md)
Tier 4 (Restructuring) · levers: Context · domains: General
Hand a context-heavy side task (code search, reading many files, documents or logs) to a sub-agent that starts with a fresh context and only the tools it needs, and have it return a short result or file references so the parent's context stays small.
- Signs in code: A sub-agent call with a fresh context, a scoped tool list and a summary or file-reference return; or a single agent that reads large search results, file sets, documents or logs into its own context. Warning signs: generic lifecycle sub-agents, sub-agents that inherit the full history or can delegate further, a parent that paraphrases sub-agent output.
- Signs in traces: Sub-agent spans: tokens consumed inside vs returned; the parent's peak context; empty returns; facts present in the sub-agent transcript but missing from the parent's answer (hand-off breakdowns, usually silent); share of the corpus read.
- Signs in evals: Single agent vs delegation (and vs an indexed retriever) on search-heavy tasks: success, cost per correct answer.
- Signs in judge: Summary fidelity against the sub-agent's full transcript: facts found but not returned, claims returned but not supported.
- Signs in probes: Broad search then synthesis.

## [6.a.i Oracle-worker multi-model](https://mahalsystems.ai/aot/optimizations/oracle-worker-escalation.md)
Tier 4 (Restructuring) · levers: Model · domains: General
Let a cheap model do the work and call an expensive one only when a trigger fires: a cascade escalates when the cheap answer looks unreliable or fails a check, a router picks the model before the call, and an advisor tool lets a cheap agent consult a stronger model mid-task.
- Signs in code: Two models with an escalation call from worker to oracle.
- Signs in traces: Escalation or consult rate and when it fires (turn of the first consult, share in the last third of the run); strong-model share of tokens and cost; escalated tasks whose outcome changed; advice the worker then ignored.
- Signs in evals: Success and cost per completed task against each model alone across effort settings and against a random mix of the two at the same spend.
- Signs in judge: Whether escalations were warranted.
- Signs in probes: Tasks the worker solves, tasks only the oracle solves, and tasks neither solves.

## [6.b.i Parallel tool execution for read-only calls](https://mahalsystems.ai/aot/optimizations/parallel-read-only-tools.md)
Tier 3 (Tuning) · levers: Control flow · domains: General
Let the model put several independent read-only tool calls (searches, fetches, file reads) in one turn and have the harness run them at once, while calls that write, share state or need another call's result run one at a time in the order the model emitted them.
- Signs in code: Each tool marked read-only (safe to run together) or as having side effects, shared state or ordering needs, with unknown tools defaulting to serial; read-only calls in a batch dispatched concurrently; the rest run in model-emitted order; all results returned in one message before any text; a parallel-calls instruction in the prompt.
- Signs in traces: Average tool calls per tool-calling turn near 1 on tasks with independent lookups (parallelism unused); overlapping spans for calls that touch the same resource; calls that failed because a prerequisite in the same batch had not run.
- Signs in evals: Accuracy, turns, cost and p95 latency with parallel calls off and on, on tasks with and without dependent calls; race-condition failures.
- Signs in probes: Batches that mix reads and writes to the same resource, or hold a call that needs another call's output.

## [6.b.ii LLM map-reduce](https://mahalsystems.ai/aot/optimizations/llm-map-reduce.md)
Tier 2 (Reliability) · levers: Control flow · domains: General
Give each piece of a large or untrusted input its own tool-less model call that sees only that piece and must answer in a fixed schema, then combine the answers in code or in a reducer that never sees the raw text.
- Signs in code: Chunking, a sandboxed model per chunk with a constrained output schema, a deterministic reducer. Flag map calls that hold tools, free-text map outputs read by a privileged or tool-using reducer, and chunking applied to tasks whose answer depends on relations across chunks.
- Signs in traces: Per-chunk outputs conform to schema.
- Signs in evals: Injection containment (attack success with one poisoned item); accuracy against the whole-input call at several input lengths, with cost and latency.
- Signs in probes: One injected item among clean items; long inputs with the evidence placed at different positions and split across chunks.

## [6.b.iii Use multi-agent orchestration only for breadth-first, independent sub-tasks](https://mahalsystems.ai/aot/optimizations/multi-agent-for-breadth.md)
Tier 4 (Restructuring) · levers: Control flow, Context · domains: General
Spawn parallel sub-agents when the work splits into independent, breadth-first parts with little shared context; expect about 15x the tokens of a chat and up to 90% less wall-clock.
- Signs in code: a multi-agent design whose sub-tasks share context or depend on each other; or a single agent on a breadth-first task with many independent parts
- Signs in traces: sub-agents duplicating work or making conflicting decisions; failure rows labeled delegation-failure-subagent or communication-failure-subagent; token use per task far above a single agent's
- Signs in evals: single-agent against multi-agent on the same tasks

## [6.c.i Code-then-execute (CaMeL)](https://mahalsystems.ai/aot/optimizations/code-then-execute.md)
Tier 4 (Restructuring) · levers: Guardrails · domains: General
Have a privileged model turn the trusted request into a small program before any untrusted data is read, then run it in an interpreter that labels every value with where it came from and checks a per-tool policy before each call, so injected text can neither add actions nor send private data to the wrong place.
- Signs in code: The privileged model sees only the request, trusted context and tool descriptions with return schemas, and writes the task as a program; a custom interpreter labels every value with its sources and readers and propagates labels through operations, string formatting and control flow, including untaken branches; untrusted content is parsed only by a model with no tools and a typed output; each side-effecting tool has a policy on argument sources and readers; repair retries and error messages stay inside the checked boundary.
- Signs in traces: Policy denials and user confirmations per task; privileged-model retries per task; no executed call whose arguments carry untrusted labels unless a policy allows it; no tool-derived text in another agent's instruction.
- Signs in evals: Static and adaptive injection suites (AgentDojo with per-tool policies, plus data-dependent tasks); benign utility and utility under attack; tokens and latency per task; share of tasks with a policy denial; excess privilege left open.
- Signs in probes: Injected data that tries to add a call, set a write argument, leak a private value through a branch or a retry, or cross an agent boundary inside instruction text.

## [6.c.ii Plan-then-execute](https://mahalsystems.ai/aot/optimizations/plan-then-execute.md)
Tier 4 (Restructuring) · levers: Guardrails, Control flow · domains: General
Have the agent fix its tool calls (an ordered list or a dependency graph) from the trusted request before it reads any tool output, and let a controller execute only those calls, so injected text cannot add an action.
- Signs in code: A plan (tool calls with their dependencies) is generated from the request before any untrusted data is read; a controller executes only planned calls plus read-only lookups; write-call arguments and destinations are bounded before execution; raw tool output and error messages never go back to the planner.
- Signs in traces: Executed calls equal the planned calls plus read-only lookups; no new write call after data ingestion; each write argument traces to the request or a trusted record, not to tool output.
- Signs in evals: Injection suite attack success, benign utility and utility under attack, on tasks that include data-dependent ones; tokens and latency per task.
- Signs in probes: Injected data that tries to add a tool call, change the arguments of a planned call, or steer a planned branch.

## [7.a.i Route metric questions through a deterministic semantic layer; fall back to text-to-SQL](https://mahalsystems.ai/aot/optimizations/semantic-layer-routing.md)
Tier 3 (Tuning) · levers: Control flow, Knowledge access · domains: Data analysis
Where a semantic layer exists, have the agent answer metric questions by choosing metrics and dimensions that the layer compiles to SQL, and write SQL only when the layer cannot answer.
- Signs in code: a data agent that writes SQL directly although a semantic or metrics layer exists
- Signs in traces: wrong numbers on questions whose metric the layer defines; failure rows labeled reasoning-failure on metric questions
- Signs in evals: accuracy on metric questions, split by whether the layer defines them

## [7.a.ii Prompt chaining with gate checks](https://mahalsystems.ai/aot/optimizations/prompt-chaining-with-gates.md)
Tier 4 (Restructuring) · levers: Control flow · domains: General
Split a task that decomposes cleanly into fixed steps into a chain of model calls, one goal per call, with a gate between steps: a schema check, an assertion, or a run against the real system, and an LLM check only for subjective criteria.
- Signs in code: A fixed sequence of model calls with checks between them rather than one prompt.
- Signs in traces: Stage outputs; gate failures; retries per stage; the whole chain rerun when one step fails.
- Signs in evals: Per-stage and end-to-end quality.
- Signs in judge: Rate each stage's output.
- Signs in probes: Inputs that should fail at each gate.

## [7.b.i Detect stalls and loops: warn, then stop early](https://mahalsystems.ai/aot/optimizations/detect-loops-and-stalls.md)
Tier 2 (Reliability) · levers: Control flow · domains: General
Track repeated identical actions (same tool, same arguments; same search query) and turns with no new information.
- Signs in code: no comparison of a tool call or query against previous ones; the loop ends only at a fixed turn cap
- Signs in traces: tool spans with identical name and arguments repeated 3 or more times in a run; the same query string issued twice; runs ending at the turn cap; failure rows labeled redundancy or state-tracking-failure
- Signs in evals: turn-cap and timeout outcomes as a share of failures

## [7.b.ii Keep errors in context](https://mahalsystems.ai/aot/optimizations/keep-errors-in-context.md)
Tier 2 (Reliability) · levers: Control flow, Context · domains: General
Return every failed action to the model as an error it can read (the call, the cause, a capped trace) instead of swallowing the exception, retrying silently, returning an empty result or deleting the failed attempt, and collapse each error once it is resolved.
- Signs in code: Exceptions swallowed or retried silently; failures returned as empty or default results; a generic "operation failed" in place of the cause; failed attempts deleted before a retry; full stack traces or test logs passed untruncated; third-party error text passed through with steps the agent cannot take (a terminal command, a config change, a web page).
- Signs in traces: Errors absent from the next model input; exact repeats of a call that already failed; runs of failed fixes of the same error; agents stopping after error text that asks for a step they cannot take; error observations thousands of lines long.
- Signs in evals: Recovery rate per injected failure and repeat rate of failed calls, alongside task success.
- Signs in probes: A tool that fails deterministically on first use.

## [7.b.iii Size run budgets so productive runs aren't cut off: turn caps, deadlines, command timeouts](https://mahalsystems.ai/aot/optimizations/size-execution-budgets.md)
Tier 2 (Reliability) · levers: Control flow · domains: General
Set the turn or step cap, the wall-clock deadline and per-command timeouts from what runs that succeeded actually needed, not from framework defaults, so a run that is still making progress is not stopped short.
- Signs in code: caps, deadlines and command timeouts at framework defaults, unchanged since the agent was built
- Signs in traces: runs ending at the turn cap or deadline while still producing new information in their last turns (a run that ends at the cap after repeating itself is 7.b.i's signal, not this one); commands killed by a timeout that finish when re-run with a longer one
- Signs in evals: limit-ended outcomes (turn cap, deadline, command timeout) on cases where the run was still progressing

## [7.b.iv Track explicit progress: a task list with a status per item, re-injected each turn](https://mahalsystems.ai/aot/optimizations/track-task-progress.md)
Tier 3 (Tuning) · levers: Control flow, Context · domains: Planning
Give the agent a task-list tool (each item with a status, one in progress at a time), ask for the list on the first turn of non-trivial tasks, and re-inject the current list after the system prompt each turn instead of keeping old copies in history.
- Signs in code: no task list or todo mechanism in the harness
- Signs in traces: runs ending with no edit or output, and short aimless trajectories (weaker models); long post-edit loops (stronger models); failure rows labeled goal-drift or state-tracking-failure
- Signs in evals: success by trajectory length, per model

## [7.c.i Reflection loop](https://mahalsystems.ai/aot/optimizations/reflection-loop.md)
Tier 3 (Tuning) · levers: Control flow · domains: General
The same model critiques and revises its own output for a few rounds (Self-Refine), or after a failed attempt writes a short lesson it carries into the next try (Reflexion).
- Signs in code: A step where the same model critiques its own output and a step that revises it (a reflect, critique or "double-check your answer" call), or verbal reflections stored between attempts, with a round cap. Flag a loop that re-checks correctness with no test, error, environment signal or explicit checklist in the critique, no cap, one that always returns the last revision, and an outer reflect step on a reasoning model with thinking on.
- Signs in traces: Rounds per item; share of rounds whose critique changes the answer; on items with ground truth, right-to-wrong and wrong-to-right flips per round; whether the critique cites a test, error or requirement or only restates the answer; tokens per round.
- Signs in evals: Accuracy per round against the single pass and against retries or best-of-N at equal tokens; share of initially right answers the loop breaks.
- Signs in judge: Rate critiques for specificity.
- Signs in probes: Correct outputs mixed with outputs carrying one detectable defect.

## [7.c.ii Self-critique evaluator loop](https://mahalsystems.ai/aot/optimizations/self-critique-evaluator.md)
Tier 3 (Tuning) · levers: Control flow · domains: General
Train or tune the system's own judge from its own outputs and synthetic judgments (contrastive pairs with one side deliberately degraded, the judge's own verdicts kept only where they match the construction label) instead of human labels, then use it to score, select or train in the loop.
- Signs in code: A pipeline that trains or prompt-tunes a judge on the system's own outputs or its own verdicts, with periodic refresh. Flag a judge from the same model or family as the generator, no human-labeled set held out from both training and selection, and a judge validated only by the agent's downstream score.
- Signs in traces: Judge pass rate and score distribution per training or refresh iteration next to accuracy on an audited sample; a widening gap, scores saturating near the top of the scale, and output length rising across iterations.
- Signs in evals: Chance-corrected agreement with a locked human-labeled set; the judge's accuracy on the generator's own outputs against equal-quality outputs from other sources; results on judge benchmarks other than the one used for model selection.
- Signs in judge: An outer judge kept outside every loop, from another family, given the task contract and checked against humans.
- Signs in probes: Outputs built to fool the judge: content-free answers, plausible-but-wrong answers and padded answers, plus the generator's outputs paired with equal-quality outputs from other sources.

## [7.c.iii Evaluator-optimizer loop](https://mahalsystems.ai/aot/optimizations/evaluator-optimizer-loop.md)
Tier 4 (Restructuring) · levers: Control flow · domains: General
A separate evaluator (an LLM judge with written criteria, a different model, or an agent that tests the running app) scores the generator's output and returns specific findings; the generator revises until the output passes or a round cap is hit.
- Signs in code: A generate, evaluate, revise loop: a separate evaluator call, model or tool-using agent scores the output against written criteria and returns feedback, with a pass threshold and a round cap. Flag an evaluator that shares the generator's dialogue history, one weaker than the generator, no cap, or a loop that returns the last version instead of the best.
- Signs in traces: Rounds per item; evaluator score per round next to an independent check; veto rate and recovery after a veto; items whose evaluator score rose while the independent check fell; revisions that rewrote parts that had passed.
- Signs in evals: Quality per round on a grader the loop never sees; share of items that end worse than their first output; evaluator agreement with human or gold labels.
- Signs in judge: The evaluator is a judge; a meta-judge checks its agreement with humans.
- Signs in probes: Items with a clear rubric and a known defect.

## [7.c.iv Graph-of-Thought](https://mahalsystems.ai/aot/optimizations/graph-of-thought.md)
Tier 4 (Restructuring) · levers: Control flow · domains: Reasoning
Graph-of-Thought runs one task as a graph of model calls: split the input into parts, generate and score several candidates per part, keep the best, then merge the partial results and refine the merged answer.
- Signs in code: A controller that runs a fixed or generated graph of model calls on one task: split the input, generate several candidates per part, score and keep the best, aggregate partial results and refine the merge (GoT's Generate, Score, KeepBestN, Aggregate). Flag independent nodes run one after another, every node on the strongest model, an LLM scorer where code could check, and no cap on depth, branching, nodes or tokens.
- Signs in traces: Per item: nodes and model calls, input and output tokens per node, aggregation and refine steps, scorer verdicts beside an independent check, budget-cap hits, wall-clock time. In a single-call agent, errors that grow with input size (dropped, duplicated or miscounted elements) mark where a split-and-merge graph could help.
- Signs in evals: Accuracy against chain-of-thought and self-consistency at matched tokens, sliced by input size; cost and latency per correct answer.
- Signs in judge: Rate whether each merge kept all information from its inputs without duplication (retention and redundancy, as in GoT's document-merging score).
- Signs in probes: Tasks whose sub-results must be merged, at growing input sizes, plus a sequential control.

## [7.c.v Language Agent Tree Search (LATS)](https://mahalsystems.ai/aot/optimizations/language-agent-tree-search.md)
Tier 4 (Restructuring) · levers: Control flow · domains: Reasoning, Planning
Language Agent Tree Search runs the agent as a Monte Carlo tree search over its own actions: sample several next actions, execute each in the environment, score the resulting states with the model, reflect on failed endings, and back the scores up the tree until a branch succeeds or the budget runs out.
- Signs in code: A search loop over agent actions: UCT or best-first selection, expansion of several sampled actions executed in the environment, a model-scored value (often with self-consistency), reflections on failed endings, backpropagation, and a way to restore earlier states (reset and replay, a commit per node, snapshots). Flag search over actions with side effects and no restore path or destructive-action gate, and search with no simpler baseline at a matched budget.
- Signs in traces: Nodes expanded and LLM calls per item; tokens and wall time per item; restore or replay time; value of the chosen trajectory against its actual outcome; destructive actions executed during exploration.
- Signs in evals: Success against ReAct with retries and best-of-N with a vote or judge at equal tokens and wall time, per model and task difficulty.
- Signs in judge: Rate reflections used as value estimates.
- Signs in probes: Multi-step tasks with recoverable mistakes.

## [7.c.vi Multi-agent debate](https://mahalsystems.ai/aot/optimizations/multi-agent-debate.md)
Tier 4 (Restructuring) · levers: Control flow · domains: Reasoning
Several model instances answer the same question, read each other's answers and revise over one or more rounds, and a vote or judge settles the final answer.
- Signs in code: Several agents answer the same question, see each other's answers and revise over rounds, with a vote, judge or synthesizer at the end. Flag it when the eval has no majority-vote arm at the same number of generations, when more than one exchange round runs, when transcripts can outgrow the context window (agents x rounds x per-turn budget, thinking included), when agents get persona prompts, or when arms parse answers differently.
- Signs in traces: Rounds per item; share of items where the final answer differs from the first-round vote and, on items with ground truth, whether the change fixed or broke it; right-to-wrong and wrong-to-right flips per round; empty or truncated final-round answers; prompt and total tokens per round.
- Signs in evals: Accuracy against a majority vote over the same number of independent generations (same model, or the same mixed roster) and against the single pass at full thinking budget, with the same answer extractor; for open-ended output, blind ranking by intended users against a single pass of equal length.
- Signs in judge: Secondary only: an external AI judge agreed with paper authors at 0.14 and would have reversed their ranking of debate against a single pass; for open-ended output rank with intended users, and balance any judge panel across vendors.
- Signs in probes: Items where the first-round majority is wrong but at least one agent is right, mixed with items where the majority is right.

## [7.c.vii Tree-of-Thought](https://mahalsystems.ai/aot/optimizations/tree-of-thought.md)
Tier 4 (Restructuring) · levers: Control flow · domains: Reasoning
Tree-of-Thought replaces one reasoning chain with a search: the model proposes several next steps, an evaluator (the model itself, a trained verifier or a rule check) scores the partial solutions, and breadth-first or depth-first search keeps the best or backtracks.
- Signs in code: A search controller: several candidate next steps per node, an evaluator that scores partial solutions (the model's value or vote, a trained verifier, or a rule or execution check), breadth-first or beam search keeping the best b, or depth-first with backtracking, and a node or token budget. Flag a one-prompt imitation ("imagine three experts") presented as ToT, a model self-evaluator where a rule or execution check exists, a search with no fallback answer when it ends empty, and no comparison against self-consistency or best-of-N at equal tokens.
- Signs in traces: Nodes expanded, depth and branching per item; pruned and backtracked branches; LLM calls, tokens and wall-clock per item; evaluator score of the chosen path next to the ground truth; share of searches that hit the budget or end without a solution.
- Signs in evals: Accuracy against the same model in one call with thinking on and against self-consistency or best-of-N at matched tokens, split by difficulty; evaluator ranking accuracy on labeled partial solutions; tokens and latency.
- Signs in judge: The partial-solution evaluator is the judge: check its ranking accuracy against ground truth before relying on it (tree search needed at least 90% to beat re-ranking).
- Signs in probes: Puzzles with dead ends.

## [7.d.i Continuous autonomous task loop](https://mahalsystems.ai/aot/optimizations/continuous-task-loop.md)
Tier 4 (Restructuring) · levers: Control flow · domains: Planning
Run the coding agent from a shell or harness loop that starts a fresh headless session every iteration with the same short prompt; each session reads the plan, progress notes and git history, does one item, runs its tests, commits and exits, until an iteration cap, a budget or a stop rule ends the run.
- Signs in code: A shell or harness loop that starts a headless agent each iteration with the same prompt file over a plan, task list or progress file, committing per iteration; whether it resumes the previous session (a stop hook or resume flag) or starts a fresh one; iteration, budget and time caps; permission prompts turned off outside a container.
- Signs in traces: Iterations; commits; rate-limit events; iterations with no tool calls; tokens since the last test or score gain; the same item claimed or built twice; previously passing tests broken; completion claims while hidden tests fail.
- Signs in evals: Throughput; defect rate of produced work.
- Signs in judge: Rate produced commits.
- Signs in probes: A queue with a poison task.

## [7.d.ii Inversion of control](https://mahalsystems.ai/aot/optimizations/inversion-of-control.md)
Tier 4 (Restructuring) · levers: Control flow · domains: General
Give the agent the goal, the tools and the acceptance checks and let it choose the sequence of steps, instead of coding a fixed workflow; the goal, permissions, budget and evaluation stay outside the agent.
- Signs in code: Minimal orchestration code; the agent decides the sequence.
- Signs in traces: Variety of action sequences across tasks of the same type: one sequence recurring across most tasks suggests code could own the path; widely varying sequences on a known procedure suggest constraints are being missed.
- Signs in evals: Success, constraint violations, LLM calls, tokens and latency per task for the agent loop against a fixed workflow, per model tier.
- Signs in judge: Rate whether autonomy was used well.
- Signs in probes: Tasks with a known procedure and hard constraints, and tasks whose next step depends on what is found.

## [7.d.iii ReAct loop](https://mahalsystems.ai/aot/optimizations/react-loop.md)
Tier 4 (Restructuring) · levers: Control flow · domains: General
Run the agent as a loop: the model reasons, calls a tool, reads the result and picks the next step, until it answers or hits a cap.
- Signs in code: A loop that alternates model call and tool dispatch, feeds results back, and has a max-iteration guard; check that the guard bounds the whole feedback path (retries, handoffs), not just the outer loop.
- Signs in traces: Thought before each action; observations used in the next thought; loops that ignore results.
- Signs in evals: Success on tool-use tasks; steps per task; tool error rate.
- Signs in judge: Do thoughts inform the next action or are they boilerplate.
- Signs in probes: Tasks needing two to four dependent tool calls.

## [8.a.i Choose the retriever by corpus size and question type](https://mahalsystems.ai/aot/optimizations/choose-retriever.md)
Tier 3 (Tuning) · levers: Knowledge access, Control flow · domains: Retrieval
On a bounded corpus (up to about 100K documents), grep and file reads over the raw text beat an embedding retriever: the agent narrows into documents it has found.
- Signs in code: the retriever in use and the corpus size; a single retrieval path for all question types
- Signs in traces: repeated searches that fail to narrow; documents found but never read in full; failure rows labeled missed-read on multi-hop questions
- Signs in evals: retrieval recall and answer accuracy split by corpus size and question type

## [8.a.ii Surface information the source holds but the agent cannot read](https://mahalsystems.ai/aot/optimizations/surface-unreadable-information.md)
Tier 3 (Tuning) · levers: Knowledge access · domains: Retrieval, Data analysis, Document extraction
Find information a connected source holds that the agent never reads, and expose it.
- Signs in code: a file tool limited to plain-text formats; a spreadsheet reader that drops formatting; the source catalog lists tables or columns the agent's prompts and tools never reference
- Signs in traces: failed opens of PDF, Excel, Word or ZIP files; the set of tables and columns queried, compared with the catalog, joined to failures; failure rows labeled domain-knowledge-deficit or missed-read
- Signs in evals: failures where the answer existed in an unread part of the source

## [8.b.i Tune the search tool: snippets, enough results, retries, reformulation, filters, source quality](https://mahalsystems.ai/aot/optimizations/tune-search-tool.md)
Tier 2 (Reliability) · levers: Knowledge access · domains: Retrieval, Web research
Make the search tool return what the model needs to choose what to open (a snippet per result, not just titles; five or more results; a full-document reader beside truncated search), retry failed searches, and reformulate when a query returns little (start broad, then narrow; use the corpus's own terms) rather than retrying it verbatim.
- Signs in code: a search tool returning titles or URLs only; a fixed small result count; no retry; no filter parameters over a corpus with metadata; results passed to the model unfiltered
- Signs in traces: many page fetches per question; queries returning zero results followed by the same query or by giving up; results from the wrong product, version or date; failure rows labeled missed-read
- Signs in evals: retrieval recall below answer accuracy needs; failures where the answer was in an unfetched result

## [8.b.ii Link large schemas by exploration: start broad, grow the linked set with probing queries, favor recall](https://mahalsystems.ai/aot/optimizations/link-large-schemas.md)
Tier 3 (Tuning) · levers: Knowledge access, Control flow · domains: Data analysis
When a schema is too large to load whole, link the tables and columns a question needs by exploration rather than by one up-front retrieval or by loading everything: start from all table names plus the top retrieved columns, give the agent a column-level vector index, metadata probes and trial queries to grow the linked set, favor recall over minimizing noise, and cap the exploration turns low.
- Signs in code: a large schema (too many tables to load whole) linked by a single up-front retrieval, or loaded whole into every prompt
- Signs in traces: queries against non-existent columns or the wrong tables on a large schema; prompt tokens dominated by the schema
- Signs in evals: schema-linking recall and execution accuracy on the large-schema subset

## [9.a.i Feed back a sound external verifier's verdict; never the model's own critique](https://mahalsystems.ai/aot/optimizations/external-verifier-feedback.md)
Tier 2 (Reliability) · levers: Control flow · domains: General
Run a check the model cannot fool on its output (compiler, tests, schema validation, execution, EXPLAIN on generated SQL, a solver) and feed the verdict back; a pass/fail verdict is enough, and resampling until the check accepts works about as well as detailed feedback.
- Signs in code: no step runs the build, tests, linter, schema check or execution on the output before it is returned; a critique or reflect step where the same model reviews its own answer with no tool output or environment state in the loop
- Signs in traces: errors a compiler, test or schema check would catch reaching the final answer; SQL errors and hallucinated table names; answers changed after a critique turn with no new external information; failure rows labeled tool-feedback-neglect or satisficing
- Signs in evals: failures a deterministic check would have flagged; accuracy before and after any critique step

## [9.a.ii Verify citations claim by claim for factual support](https://mahalsystems.ai/aot/optimizations/verify-citations.md)
Tier 3 (Tuning) · levers: Control flow, Knowledge access · domains: Web research, Retrieval
Extract claim-citation pairs deterministically, then check each: the link resolves, the source is relevant, and the source actually supports the claim.
- Signs in code: a research or RAG agent that returns citations with no check of what they support; no cap on search calls
- Signs in traces: search-call counts far above what the question needs; cited passages that do not contain the claim
- Signs in evals: share of citations that factually support their claim

## [9.b.i Make the final answer reliably extractable: state the format, extract robustly, force an answer at the limit](https://mahalsystems.ai/aot/optimizations/extractable-final-answer.md)
Tier 2 (Reliability) · levers: Control flow, Prompt · domains: Structured output
State the required answer format in the prompt, and make the harness extract it robustly: strip markers, normalize numbers, detect refusals, and force a best-effort answer when the loop ends on a budget rather than returning nothing.
- Signs in code: no answer-format statement; extraction that assumes a marker; no forced answer when the loop ends on a budget
- Signs in traces: runs ending with no extractable answer; refusals returned as answers; numbers in the wrong format; runs ending at the turn cap with a correct value in context (in one image-labeling agent, 71 items returned all-unknown)
- Signs in evals: failures where the working was right but the submitted answer was not (instruction-grader-mismatch)

## [9.b.ii Triage false success with a cheap trained detector, not an LLM judge](https://mahalsystems.ai/aot/optimizations/false-success-detector.md)
Tier 2 (Reliability) · levers: Control flow · domains: Judging
Train a lightweight text classifier (TF-IDF with logistic regression) on 50-100 labeled runs to flag likely false successes for review.
- Signs in traces: runs that end with a success claim; the closing message text and the tool-call sequence are the detector's inputs
- Signs in evals: an LLM judge currently used to decide whether a run completed; a labeled sample with a false-success rate above a few percent

## [9.b.iii Verify completion against the environment before the agent stops](https://mahalsystems.ai/aot/optimizations/verify-completion.md)
Tier 2 (Reliability) · levers: Control flow · domains: General
Never take the agent's own report that it finished.
- Signs in code: completion taken from the agent's final message or its own status flag; no stop hook; the agent can edit the tests or checklist it is judged by
- Signs in traces: final message claims success while the environment or transcript shows otherwise; runs ending right after an edit with no test or build run; edits to checklist items; failure rows labeled satisficing or specification-gaming
- Signs in evals: tasks scored failed where the agent reported success (false-success rate); features marked done that fail an end-to-end check

## [9.c.i Don't prompt agents to write their own tests; suppress it for test-heavy models](https://mahalsystems.ai/aot/optimizations/no-self-written-tests.md)
Tier 3 (Tuning) · levers: Control flow · domains: Code generation
Agent-written tests are mostly print probes and do not change resolve rate.
- Signs in code: a prompt that tells the agent to write tests
- Signs in traces: test files written with print statements and no assertions; test writing and running over about 20% of tokens
- Signs in evals: resolve rate unchanged between runs with and without agent tests

## [9.c.ii Spend extra samples before extra verification: parallel attempts with a vote or judge; verify only where it pays](https://mahalsystems.ai/aot/optimizations/samples-before-verification.md)
Tier 3 (Tuning) · levers: Control flow · domains: General
Running several independent attempts and picking by majority vote or a judge raised accuracy (Genie 73.8% to 81.4% with 8 runs, at up to 2.1x cost).
- Signs in code: a single attempt per question; or a generative verifier or best-of-N step with no comparison against a plain vote
- Signs in traces: answer disagreement across repeated runs of the same question
- Signs in evals: accuracy of vote against verifier at equal compute, split by difficulty

## [9.c.iii Audit a verifier-gated cascade against a gold sample the verifier never sees](https://mahalsystems.ai/aot/optimizations/audit-verifier-cascade.md)
Tier 4 (Restructuring) · levers: Control flow, Model · domains: General
When a cheap model's outputs are accepted or escalated by a verifier, measure true error on a small gold-labeled sample outside the verifier: metrics measured through the verifier showed about 3% error while true error was 14-32%.
- Signs in code: a cheap model whose outputs are accepted or escalated by a verifier model, with error measured only through the verifier
- Signs in traces: escalation rate against the eval's true error rate
- Signs in evals: no gold-labeled sample scored independently of the verifier

## [10.a.i Constrain what the agent can touch](https://mahalsystems.ai/aot/optimizations/constrain-agent-reach.md)
Tier 2 (Reliability) · levers: Guardrails · domains: General
Run the agent in an OS-level sandbox with a workspace path guard that resolves symlinks, scrub credentials from the command environment, run agent-written SQL read-only with timeouts and row caps, and set resource limits on generated code.
- Signs in code: shell or code execution without a sandbox; credentials in the agent's environment; write access where read is enough; no row cap or timeout on agent SQL
- Signs in traces: commands touching paths outside the workspace; queries returning very large result sets; failure rows labeled unauthorized-irreversible-action
- Signs in evals: safety evals with out-of-bounds actions

## [10.b.i Gate irreversible actions: allow, ask or deny by side effect](https://mahalsystems.ai/aot/optimizations/risky-action-policy.md)
Tier 2 (Reliability) · levers: Guardrails · domains: General
Classify every action the agent can take by its side effect (read, reversible write, irreversible or externally visible: sending email, issuing a refund, deleting a record, posting publicly, pushing code) and gate by class: allow, ask a human or a policy, or deny.
- Signs in code: tools with irreversible or externally visible effects (send, pay, delete, publish, push) with no gate and no staging step; one approval setting for every tool; a deny list that auto-approve disables
- Signs in traces: irreversible actions taken without a policy or human check; failure rows labeled unauthorized-irreversible-action or over-initiative
- Signs in evals: safety cases built from the agent's own irreversible actions, including prompt-injected requests for them

## [10.c.i Contain sensitive data and untrusted input](https://mahalsystems.ai/aot/optimizations/contain-untrusted-input.md)
Tier 2 (Reliability) · levers: Guardrails · domains: General
Tokenize PII in the client so it never reaches the model, wrap untrusted tool and web results in taint delimiters, scan context files and skills for prompt injection, and state an explicit instruction-precedence order (user, repo, defaults, external data as data).
- Signs in code: PII or untrusted content passed to the model unmarked; no instruction-precedence statement
- Signs in traces: sensitive fields in model inputs or logs; injected instructions followed; failure rows labeled indirect-prompt-injection
- Signs in evals: injection and data-leak safety evals