Scoring and Metrics

​

Evaluation modes

@ze.evaluation(mode="row", outputs=["exact_match"])
def exact_match(row, answer_col, prediction_col):
    return {"exact_match": int(answer_col == prediction_col)}

@ze.evaluation(mode="column", outputs=["accuracy"])
def accuracy(exact_match_col):
    total = len(exact_match_col)
    return {"accuracy": (sum(exact_match_col) / total) if total else 0.0}

@ze.evaluation(mode="run", outputs=["accuracy_mean"])
def accuracy_mean(all_runs):
    values = [r.metrics["accuracy"] for r in all_runs if "accuracy" in r.metrics]
    return {"accuracy_mean": (sum(values) / len(values)) if values else 0.0}

​

How inputs bind

• Row evaluations receive one row plus any mapped scalar values.
• Column evaluations receive mapped column lists across the whole run.
• Run evaluations receive all_runs after repetitions.

@ze.evaluation(mode="row", outputs=["exact_match"])
def exact_match(row, gold_col, pred_col):
    return {"exact_match": int(gold_col == pred_col)}

@ze.evaluation(mode="column", outputs=["accuracy"])
def accuracy(exact_match_col):
    return {"accuracy": sum(exact_match_col) / len(exact_match_col)}

​

Column mapping

run = run.score(
    [exact_match, accuracy],
    column_map={
        "exact_match": {
            "gold_col": "answer",
            "pred_col": "prediction",
        },
        "accuracy": {"exact_match_col": "exact_match"},
    },
)

​

score() vs eval()

• run.score(...) is an alias of run.eval(...)
• In the docs, prefer run.score(...) for clarity

​

Signals vs metrics

• Emit signals during task execution for runtime facts like retrieved_doc_count, phase, or tests_failed_after.
• Compute metrics after execution with row/column/run evaluations.

​

Metric helpers

run.column_metrics([accuracy])
run.run_metrics([accuracy_mean], all_runs=repeated_runs)

​

Output locations

• Task outputs and row scores are appended to each run.rows[i]
• Aggregate metrics are placed in run.metrics
• The execution summary is available in run.health