Creating Datasets

Recommended: git or UI

For most benchmarks, upload data via git or the CSV importer in the benchmark hub. See Uploading Data for step-by-step instructions. Git is best for:

large or versioned datasets
including scorer implementations and documentation alongside data
reproducible, CI-friendly workflows

The browser editor and CSV importer are best for:

quick prototypes
small manually-curated test sets
one-off imports

SDK-based creation

You can also create and push datasets programmatically from Python. This is useful for generating datasets from code or integrating with data pipelines.

Create from Python data

import zeroeval as ze

ze.init()

dataset = ze.Dataset(
    "capital-cities",
    data=[
        {"row_id": "fr", "question": "Capital of France?", "answer": "Paris"},
        {"row_id": "de", "question": "Capital of Germany?", "answer": "Berlin"},
    ],
    description="Simple geography dataset",
)

Add a stable row_id field whenever possible. It makes resume behavior and row-level tracking much more reliable.

Create from CSV

dataset = ze.Dataset("/path/to/questions.csv")
print(dataset.name)  # e.g. "questions"
print(len(dataset))

The dataset name defaults to the CSV filename without the extension.

Modify rows

dataset.add_rows(
    [{"row_id": "es", "question": "Capital of Spain?", "answer": "Madrid"}]
)

dataset.update_row(0, {"question": "Capital city of France?", "answer": "Paris"})

dataset.delete_row(1)

Row updates by index can change row ordering and row content. Keep this in mind if your downstream logic assumes fixed order.

Push to ZeroEval

dataset.push()
print(dataset.version_id, dataset.version_number)

After push, you can pin future runs to that version number:

latest = ze.Dataset.pull("capital-cities")
pinned = ze.Dataset.pull("capital-cities", version_number=dataset.version_number)

Validate shape

print(dataset.columns)  # sorted list of column names
print(len(dataset))     # number of rows
print(dataset[0])       # DotDict for first row

Getting Started

Datasets

Evals

CLI

Examples

Recommended: git or UI

SDK-based creation

Create from Python data

Create from CSV

Modify rows

Push to ZeroEval

Validate shape

​Recommended: git or UI

​SDK-based creation

​Create from Python data

​Create from CSV

​Modify rows

​Push to ZeroEval

​Validate shape

Recommended: git or UI

SDK-based creation

Create from Python data

Create from CSV

Modify rows

Push to ZeroEval

Validate shape