Skip to main content

​
Why versioning matters

Reproducible evaluation depends on running against a known dataset snapshot. The SDK exposes version metadata after push and supports pulling a specific version.
To run against a known version:

​
One compact example

​
Subset pulls

If your dataset includes named subsets, pull only the subset you want:
If subset is omitted, the SDK resolves the default subset automatically:
  1. The benchmark’s configured default subset
  2. A file named data/default.parquet
  3. If there is only one subset, that subset
  4. The first subset alphabetically
Subsets are detected from Parquet files in the data/ directory. Each file data/<name>.parquet becomes a subset named <name>. See Uploading Data for the full layout.

​
Reproducibility checklist

1

Pin dataset version when benchmarking

For benchmark comparisons over time, prefer version_number instead of always pulling latest.
2

Track subset in experiments

Store subset name in run parameters for transparent reporting.
3

Use stable row identifiers

Include deterministic row_id fields to make resume and row comparisons robust.
Attach dataset version + subset in run parameters so dashboards and downstream analysis remain self-describing.