Mahal Systems Overview › Lever
Control flow
The harness loop: orchestration, multi-agent topology, budgets, stop rules, verify-and-retry. 31 optimizations change this part of an agent.
- 2.b.i
Make tool calls and tool results fail soft, and retry transient failures
Keep a failed call from ending the run: validate tool calls and model responses, return tool errors to the agent as results, and retry only failures that are transient and safe to repeat.
- 6.b.i
Parallel tool execution for read-only calls
Let the model put several independent read-only tool calls (searches, fetches, file reads) in one turn and have the harness run them at once, while calls that write, share state or need another call's result run one at a time in the order the model emitted them.
- 6.b.ii
LLM map-reduce
Give each piece of a large or untrusted input its own tool-less model call that sees only that piece and must answer in a fixed schema, then combine the answers in code or in a reducer that never sees the raw text.
- 6.b.iii
Use multi-agent orchestration only for breadth-first, independent sub-tasks
Spawn parallel sub-agents when the work splits into independent, breadth-first parts with little shared context; expect about 15x the tokens of a chat and up to 90% less wall-clock.
- 6.c.ii
Plan-then-execute
Have the agent fix its tool calls (an ordered list or a dependency graph) from the trusted request before it reads any tool output, and let a controller execute only those calls, so injected text cannot add an action.
- 7.a.i
Route metric questions through a deterministic semantic layer; fall back to text-to-SQL
Where a semantic layer exists, have the agent answer metric questions by choosing metrics and dimensions that the layer compiles to SQL, and write SQL only when the layer cannot answer.
- 7.a.ii
Prompt chaining with gate checks
Split a task that decomposes cleanly into fixed steps into a chain of model calls, one goal per call, with a gate between steps: a schema check, an assertion, or a run against the real system, and an LLM check only for subjective criteria.
- 7.b.i
Detect stalls and loops: warn, then stop early
Track repeated identical actions (same tool, same arguments; same search query) and turns with no new information.
- 7.b.ii
Keep errors in context
Return every failed action to the model as an error it can read (the call, the cause, a capped trace) instead of swallowing the exception, retrying silently, returning an empty result or deleting the failed attempt, and collapse each error once it is resolved.
- 7.b.iii
Size run budgets so productive runs aren't cut off: turn caps, deadlines, command timeouts
Set the turn or step cap, the wall-clock deadline and per-command timeouts from what runs that succeeded actually needed, not from framework defaults, so a run that is still making progress is not stopped short.
- 7.b.iv
Track explicit progress: a task list with a status per item, re-injected each turn
Give the agent a task-list tool (each item with a status, one in progress at a time), ask for the list on the first turn of non-trivial tasks, and re-inject the current list after the system prompt each turn instead of keeping old copies in history.
- 7.c.i
Reflection loop
The same model critiques and revises its own output for a few rounds (Self-Refine), or after a failed attempt writes a short lesson it carries into the next try (Reflexion).
- 7.c.ii
Self-critique evaluator loop
Train or tune the system's own judge from its own outputs and synthetic judgments (contrastive pairs with one side deliberately degraded, the judge's own verdicts kept only where they match the construction label) instead of human labels, then use it to score, select or train in the loop.
- 7.c.iii
Evaluator-optimizer loop
A separate evaluator (an LLM judge with written criteria, a different model, or an agent that tests the running app) scores the generator's output and returns specific findings; the generator revises until the output passes or a round cap is hit.
- 7.c.iv
Graph-of-Thought
Graph-of-Thought runs one task as a graph of model calls: split the input into parts, generate and score several candidates per part, keep the best, then merge the partial results and refine the merged answer.
- 7.c.v
Language Agent Tree Search (LATS)
Language Agent Tree Search runs the agent as a Monte Carlo tree search over its own actions: sample several next actions, execute each in the environment, score the resulting states with the model, reflect on failed endings, and back the scores up the tree until a branch succeeds or the budget runs out.
- 7.c.vi
Multi-agent debate
Several model instances answer the same question, read each other's answers and revise over one or more rounds, and a vote or judge settles the final answer.
- 7.c.vii
Tree-of-Thought
Tree-of-Thought replaces one reasoning chain with a search: the model proposes several next steps, an evaluator (the model itself, a trained verifier or a rule check) scores the partial solutions, and breadth-first or depth-first search keeps the best or backtracks.
- 7.d.i
Continuous autonomous task loop
Run the coding agent from a shell or harness loop that starts a fresh headless session every iteration with the same short prompt; each session reads the plan, progress notes and git history, does one item, runs its tests, commits and exits, until an iteration cap, a budget or a stop rule ends the run.
- 7.d.ii
Inversion of control
Give the agent the goal, the tools and the acceptance checks and let it choose the sequence of steps, instead of coding a fixed workflow; the goal, permissions, budget and evaluation stay outside the agent.
- 7.d.iii
ReAct loop
Run the agent as a loop: the model reasons, calls a tool, reads the result and picks the next step, until it answers or hits a cap.
- 8.a.i
Choose the retriever by corpus size and question type
On a bounded corpus (up to about 100K documents), grep and file reads over the raw text beat an embedding retriever: the agent narrows into documents it has found.
- 8.b.ii
Link large schemas by exploration: start broad, grow the linked set with probing queries, favor recall
When a schema is too large to load whole, link the tables and columns a question needs by exploration rather than by one up-front retrieval or by loading everything: start from all table names plus the top retrieved columns, give the agent a column-level vector index, metadata probes and trial queries to grow the linked set, favor recall over minimizing noise, and cap the exploration turns low.
- 9.a.i
Feed back a sound external verifier's verdict; never the model's own critique
Run a check the model cannot fool on its output (compiler, tests, schema validation, execution, EXPLAIN on generated SQL, a solver) and feed the verdict back; a pass/fail verdict is enough, and resampling until the check accepts works about as well as detailed feedback.
- 9.a.ii
Verify citations claim by claim for factual support
Extract claim-citation pairs deterministically, then check each: the link resolves, the source is relevant, and the source actually supports the claim.
- 9.b.i
Make the final answer reliably extractable: state the format, extract robustly, force an answer at the limit
State the required answer format in the prompt, and make the harness extract it robustly: strip markers, normalize numbers, detect refusals, and force a best-effort answer when the loop ends on a budget rather than returning nothing.
- 9.b.ii
Triage false success with a cheap trained detector, not an LLM judge
Train a lightweight text classifier (TF-IDF with logistic regression) on 50-100 labeled runs to flag likely false successes for review.
- 9.b.iii
Verify completion against the environment before the agent stops
Never take the agent's own report that it finished.
- 9.c.i
Don't prompt agents to write their own tests; suppress it for test-heavy models
Agent-written tests are mostly print probes and do not change resolve rate.
- 9.c.ii
Spend extra samples before extra verification: parallel attempts with a vote or judge; verify only where it pays
Running several independent attempts and picking by majority vote or a judge raised accuracy (Genie 73.8% to 81.4% with 8 runs, at up to 2.1x cost).
- 9.c.iii
Audit a verifier-gated cascade against a gold sample the verifier never sees
When a cheap model's outputs are accepted or escalated by a verifier, measure true error on a small gold-labeled sample outside the verifier: metrics measured through the verifier showed about 3% error while true error was 14-32%.