Mahal Systems Overview › Agent type
Judging
Evaluating, grading, verifying outputs: LLM-as-judge, graders, decision models.
Specific to judging
- 9.b.ii
Triage false success with a cheap trained detector, not an LLM judge
Train a lightweight text classifier (TF-IDF with logistic regression) on 50-100 labeled runs to flag likely false successes for review.
General optimizations with evidence from judging
- 3.a.ii
Avoid few-shot ruts
An agent's own history, and any examples in its prompt, act as few-shot demonstrations: when the context fills with near-identical action-observation pairs (twenty similar items in a row, a retried call), the model tends to copy their surface form instead of judging the current item.
- 7.c.ii
Self-critique evaluator loop
Train or tune the system's own judge from its own outputs and synthetic judgments (contrastive pairs with one side deliberately degraded, the judge's own verdicts kept only where they match the construction label) instead of human labels, then use it to score, select or train in the loop.
- 7.c.iii
Evaluator-optimizer loop
A separate evaluator (an LLM judge with written criteria, a different model, or an agent that tests the running app) scores the generator's output and returns specific findings; the generator revises until the output passes or a round cap is hit.