Evals

arcana.evals ships with arcana-core so you can evaluate your own agents without any extra install.

from arcana.evals import EvalHarness

harness = EvalHarness(use_llm=False)   # rules-only, no cost
summary = await harness.run(suite="cards")
print(f"Pass rate: {summary.pass_rate:.1%}")

Harness

arcana.evals.harness.EvalHarness

EvalHarness(
    use_llm=True,
    judge_model=None,
    default_model="ollama/hermes-3",
    concurrency=3,
    results_dir=None,
)

Runs EvalCases against live agents and scores results.

Responsibilities

• Load cases from suites
• Run each case (skipping marked cases)
• Score with the configured judge
• Store results to ~/.arcana/evals/results/ (or results_dir if provided)
• Optionally compare against a baseline run (regression mode)
• Return a summary

Source code in packages/arcana-core/arcana/evals/harness.py

def __init__(
    self,
    use_llm: bool = True,
    judge_model: str | None = None,
    default_model: str = "ollama/hermes-3",
    concurrency: int = 3,
    results_dir: Path | None = None,
) -> None:
    self._use_llm = use_llm
    self._default_model = default_model
    self._concurrency = concurrency
    self._results_dir = results_dir or _DEFAULT_RESULTS_DIR
    self._judge = CompositeJudge(
        llm_judge=LLMJudge(model=judge_model) if use_llm else None,
        use_llm=use_llm,
    )

run async

run(suite=None, baseline_run_id=None)

Run all (or a subset of) eval cases.

Source code in packages/arcana-core/arcana/evals/harness.py

async def run(
    self,
    suite: str | None = None,
    baseline_run_id: str | None = None,
) -> EvalRunSummary:
    """Run all (or a subset of) eval cases."""
    run_id = f"run-{datetime.now(UTC).strftime('%Y%m%d-%H%M%S')}"
    cases = self._load_cases(suite)

    print("\n🌌 Arcana Eval Harness")
    print(f"   Run ID:  {run_id}")
    print(f"   Cases:   {len(cases)}")
    print(f"   Judge:   {'LLM + Rules' if self._use_llm else 'Rules only'}")
    print(f"   Suite:   {suite or 'all'}\n")

    # Token usage is recorded on the session regardless. Per-call cost is emitted only if
    # on_cost is provided at construction; without it, no CostEvent fires.
    async with ModelGateway(ConnectionStore()) as gateway:
        results = await self._run_cases(cases, run_id, gateway)
    self._save_results(run_id, results)

    regression = None
    if baseline_run_id:
        regression = self._compare_to_baseline(run_id, baseline_run_id, results)

    summary = self._build_summary(run_id, suite, cases, results, regression)
    self._print_summary(summary)
    return summary

Judges

arcana.evals.judge.CompositeJudge

CompositeJudge(
    llm_judge=None, use_llm=True, llm_weight=0.7
)

Bases: BaseJudge

Combines RuleJudge and LLMJudge.

Rule checks are always free and fast — run them regardless. LLM scoring is optional (set use_llm=False for CI fast path).

Scoring

If use_llm=True: overall = llm_weight * llm_score + (1-llm_weight) * rule_score If use_llm=False: overall = rule_score only

Source code in packages/arcana-core/arcana/evals/judge.py

def __init__(
    self,
    llm_judge: LLMJudge | None = None,
    use_llm: bool = True,
    llm_weight: float = 0.7,
) -> None:
    self._rule = RuleJudge()
    self._llm = llm_judge or LLMJudge()
    self._use_llm = use_llm
    self._llm_weight = llm_weight

arcana.evals.judge.RuleJudge

Bases: BaseJudge

Deterministic checks against EvalRubric.required_elements and EvalRubric.forbidden_elements. Fast, cheap, always runs.

Scores

required: 1.0 if present, 0.0 if absent forbidden: 1.0 if absent, 0.0 if present overall: fraction of rules passed

arcana.evals.judge.LLMJudge

LLMJudge(model=None)

Bases: BaseJudge

Uses a strong LLM to score qualitative dimensions against the rubric.

Always uses a different model than the agent being evaluated. Default: claude-opus-4 (strongest available, most reliable judge).

Prompt strategy

• Provides the original prompt, the response, and each dimension
• Asks for a score (0.0–1.0) and one-sentence reasoning per dimension
• Requests JSON output for reliable parsing
• Instructs the judge to be calibrated, not generous

Source code in packages/arcana-core/arcana/evals/judge.py

def __init__(self, model: str | None = None) -> None:
    self._model = model or self.JUDGE_MODEL

Types

arcana.evals.types.EvalCase

Bases: BaseModel

A single evaluation scenario. Fully reproducible — same inputs every run.

Suites

cards — does the card system produce meaningfully different agents? memory — does memory actually improve responses? decay — do stale entries rank below fresh ones? blending — does multi-card blending produce balanced output? coordination — multi-agent scenarios (Phase 2)

arcana.evals.types.EvalResult

Bases: BaseModel

The output of running a single EvalCase.

arcana.evals.types.EvalRubric

Bases: BaseModel

Defines what good output looks like for an EvalCase. Used by both LLMJudge (dimensions) and RuleJudge (elements).

arcana.evals.types.EvalDimension

Bases: BaseModel

A single axis of quality for a judge to score.

Examples:

EvalDimension("depth", "Explores nuance and tradeoffs thoroughly", weight=2.0) EvalDimension("tone", "Measured and precise, not rushed", weight=1.0) EvalDimension("memory_recall", "References pre-seeded memory accurately", weight=1.5)