Mahal Systems Overview › 7Orchestration
Orchestration
16 optimizations in 4 sub-categories.
a.Routing
Route work to the right path up front
- i.
Route metric questions through a deterministic semantic layer; fall back to text-to-SQL
Where a semantic layer exists, have the agent answer metric questions by choosing metrics and dimensions that the layer compiles to SQL, and write SQL only when the layer cannot answer.
- ii.
Prompt chaining with gate checks
Split a task that decomposes cleanly into fixed steps into a chain of model calls, one goal per call, with a gate between steps: a schema check, an assertion, or a run against the real system, and an LLM check only for subjective criteria.
See also
- 8.a.i
Choose the retriever by corpus size and question type
On a bounded corpus (up to about 100K documents), grep and file reads over the raw text beat an embedding retriever: the agent narrows into documents it has found.
b.Bounding execution
Bound execution and fail gracefully
- i.
Detect stalls and loops: warn, then stop early
Track repeated identical actions (same tool, same arguments; same search query) and turns with no new information.
- ii.
Keep errors in context
Return every failed action to the model as an error it can read (the call, the cause, a capped trace) instead of swallowing the exception, retrying silently, returning an empty result or deleting the failed attempt, and collapse each error once it is resolved.
- iii.
Size run budgets so productive runs aren't cut off: turn caps, deadlines, command timeouts
Set the turn or step cap, the wall-clock deadline and per-command timeouts from what runs that succeeded actually needed, not from framework defaults, so a run that is still making progress is not stopped short.
- iv.
Track explicit progress: a task list with a status per item, re-injected each turn
Give the agent a task-list tool (each item with a status, one in progress at a time), ask for the list on the first turn of non-trivial tasks, and re-inject the current list after the system prompt each turn instead of keeping old copies in history.
See also
- 2.b.i
Make tool calls and tool results fail soft, and retry transient failures
Keep a failed call from ending the run: validate tool calls and model responses, return tool errors to the agent as results, and retry only failures that are transient and safe to repeat.
c.Scaling compute
Spend extra compute only where it pays
- i.
Reflection loop
The same model critiques and revises its own output for a few rounds (Self-Refine), or after a failed attempt writes a short lesson it carries into the next try (Reflexion).
- ii.
Self-critique evaluator loop
Train or tune the system's own judge from its own outputs and synthetic judgments (contrastive pairs with one side deliberately degraded, the judge's own verdicts kept only where they match the construction label) instead of human labels, then use it to score, select or train in the loop.
- iii.
Evaluator-optimizer loop
A separate evaluator (an LLM judge with written criteria, a different model, or an agent that tests the running app) scores the generator's output and returns specific findings; the generator revises until the output passes or a round cap is hit.
- iv.
Graph-of-Thought
Graph-of-Thought runs one task as a graph of model calls: split the input into parts, generate and score several candidates per part, keep the best, then merge the partial results and refine the merged answer.
- v.
Language Agent Tree Search (LATS)
Language Agent Tree Search runs the agent as a Monte Carlo tree search over its own actions: sample several next actions, execute each in the environment, score the resulting states with the model, reflect on failed endings, and back the scores up the tree until a branch succeeds or the budget runs out.
- vi.
Multi-agent debate
Several model instances answer the same question, read each other's answers and revise over one or more rounds, and a vote or judge settles the final answer.
- vii.
Tree-of-Thought
Tree-of-Thought replaces one reasoning chain with a search: the model proposes several next steps, an evaluator (the model itself, a trained verifier or a rule check) scores the partial solutions, and breadth-first or depth-first search keeps the best or backtracks.
See also
- 9.c.ii
Spend extra samples before extra verification: parallel attempts with a vote or judge; verify only where it pays
Running several independent attempts and picking by majority vote or a judge raised accuracy (Genie 73.8% to 81.4% with 8 runs, at up to 2.1x cost).
d.Shaping the loop
Choose the shape of the agent loop
- i.
Continuous autonomous task loop
Run the coding agent from a shell or harness loop that starts a fresh headless session every iteration with the same short prompt; each session reads the plan, progress notes and git history, does one item, runs its tests, commits and exits, until an iteration cap, a budget or a stop rule ends the run.
- ii.
Inversion of control
Give the agent the goal, the tools and the acceptance checks and let it choose the sequence of steps, instead of coding a fixed workflow; the goal, permissions, budget and evaluation stay outside the agent.
- iii.
ReAct loop
Run the agent as a loop: the model reasons, calls a tool, reads the result and picks the next step, until it answers or hits a cap.