Common Failures

Initialization and auth

SDK not initialized

Calls like Dataset.push(...) and Dataset.pull(...) require initialization.

import zeroeval as ze
ze.init(api_key="sk_ze_...")

Missing API key

Set ZEROEVAL_API_KEY in your environment or pass api_key directly to ze.init(...).

Task contract errors

If a task is decorated with @ze.task(outputs=["prediction"]), it must return:

A dictionary
Including prediction

@ze.task(outputs=["prediction"])
def bad_task(row):
    return "oops"  # invalid: must return dict

Column mapping errors

Evaluator args must map to real columns.

run.score(
    [exact_match],
    column_map={"exact_match": {"answer_col": "answer_typo"}},  # invalid column
)

The SDK validates mappings and raises clear errors for unknown columns.

Resume errors

skip_completed=True requires stable row identity.

# Good row schema for resume
{"row_id": "r_42", "question": "...", "answer": "..."}

Without row_id (or id), row-level resume tracking cannot be guaranteed. repeat(n) also works best when rows have stable identity, because repeated runs and resumed runs are easier to compare by row_id.

Reliability tuning

Use these controls for flaky providers:

execution = ze.ExecutionConfig(
    workers=8,
    timeout_s=30,
    retry=ze.RetryPolicy(
        max_attempts=4,
        base_delay_s=1.0,
        max_delay_s=20.0,
    ),
)

For production workloads, pair retries with checkpointing so partial progress is persisted during long runs.

Getting Started

Datasets

Evals

CLI

Examples

Initialization and auth

Task contract errors

Column mapping errors

Resume errors

Reliability tuning

​Initialization and auth

​Task contract errors

​Column mapping errors

​Resume errors

​Reliability tuning

Initialization and auth

Task contract errors

Column mapping errors

Resume errors

Reliability tuning