Evaluating Multi-Agent Systems in Production Without Burning Your API Budget

The Evaluation Dilemma

When engineering teams build autonomous multi-agent pipelines (like LangGraph agent swarms), they quickly run into the evaluation trap:

• The “Vibe Check” Anti-Pattern: Engineers manually inspect 5 sample outputs, decide it “feels right,” and deploy. Silent regressions occur on edge cases.
• The “LLM-as-a-Judge” Money Pit: Calling GPT-4 to judge every intermediate step across a 200-test regression suite costs $40–$100 per PR build, discouraging developers from running tests.

The 4-Layer Evaluation Pyramid

            /\
           /  \     Layer 4: LLM-as-a-Judge (5% of tests, semantic tone & quality)
          /────\
         /      \    Layer 3: Trajectory & Tool Assertions (20%, correct tools invoked)
        /────────\
       /          \   Layer 2: Pydantic Schema Contracts (35%, strictly valid JSON)
      /────────────\
     /              \  Layer 1: Deterministic Invariants (40%, latency, token budget, zero $)

Layer 1: Deterministic Invariants ($0 Cost)

Before evaluating what the model wrote, evaluate how the system executed:

• Cycle / Step Budget: Did the LangGraph agent resolve within the designated step limit (e.g. <10 iterations)?
• Token Consumption Cap: Did the agent stay within the max token threshold (e.g. <4,000 tokens per session)?
• Latency SLA: Did the total execution pipeline complete within user tolerance (<4.5 seconds)?

Layer 2: Output Schema & Assertion Contracts ($0 Cost)

Deterministic schema testing with Pydantic:

• Validating that all required schema attributes exist.
• Asserting that numerical metrics (e.g. prices, dates, confidence scores) fall within valid ranges.
• Checking that no prohibited substrings or prompt leakage markers appear in the output.

Layer 3: Agent Trajectory & Tool Scoring

In multi-agent systems, the journey matters as much as the destination. You must verify whether the agent chose the optimal execution path:

• Tool Selection Accuracy: Did the agent call the SearchVectorStore tool for retrieval queries, or did it call the DatabaseQuery tool unnecessarily?
• Tool Argument Validity: Were the generated parameters passed to the tool syntactically and logically correct?
• State Transition Assertions: Did the supervisor agent correctly route from discovery to evaluation without looping back?

Layer 4: Targeted Semantic Evaluation (Low Token Cost)

Only run LLM-as-a-judge on the final synthesized response, and use fast, cost-efficient models:

• Use smaller models (Claude 3.5 Haiku or GPT-4o-mini) paired with strict rubric scoring (0 to 1 scales with categorical criteria).
• Faithfulness Evaluation: Asserting that claims in the output are strictly supported by the retrieved context snippets, penalizing ungrounded hallucinations.

Automated CI/CD Regression Gates

In production deployments, we wire this 4-layer pyramid directly into GitHub Actions:

1. On pull request, a synthetic dataset of 30 known failure cases executes against the staging agent graph.
2. Layers 1, 2, and 3 run deterministically in sub-30 seconds.
3. Layer 4 executes on a 10% representative sample, costing <$0.40 per build.
4. If trajectory accuracy or schema compliance drops below 98%, the pull request is blocked automatically.