Skip to content

Migration from the Parquet backend

Version 0.2 of research-tracker replaced the Parquet tables of 0.1.x with a single SQLite database. A store written by 0.1.x carries schema.json = {"schema_version": 1} plus one Parquet file per logical table (runs.parquet, evaluations.parquet, conditions.parquet, metrics.parquet, artifacts.parquet). The SQLite backend is schema version 2, and opening a version-1 store with ExperimentStore fails at construction:

SchemaMismatchError: Store <root> was written with schema version 1, but this
version of research-tracker expects schema version 2.

The migrate command upgrades such a store in place, preserving every record and every ID.

Install the extra first

Reading Parquet requires pyarrow, which is an optional dependency needed only for the upgrade:

pip install "research-tracker[migrate]"

Without it, the migration names that extra in its error instead of failing with a bare import error. pyarrow is deliberately not a required dependency of the package.

The module reference is at migration module.

Running the migration

research-tracker --root ./artifacts migrate --dry-run
research-tracker --root ./artifacts migrate

The command takes the usual --root/-o store path, and:

  • --dry-run / --dry_run reports the row counts and modifies nothing;
  • --no-backup / --no_backup leaves the Parquet files where they are instead of moving them to parquet-v1/.

Anything that prevents the upgrade raises MigrationError and leaves the store untouched. That includes a path that is not a directory, a schema.json that declares a version other than 1, a directory that already contains store.sqlite, a Parquet file whose columns do not match the v1 layout, and a missing migrate extra.

What it does

  1. Reads schema.json and refuses anything that is not a schema-version-1 Parquet store.
  2. Reads and validates the five Parquet tables into canonical, typed frames. A v1 store that lacks one of the files is treated as having an empty table for it.
  3. Builds the v2 store in a staging directory — so ExperimentStore never sees the v1 marker at the real root — and bulk-loads every table, preserving each record and every ID, so evaluation_id relationships survive unchanged. The staging row counts are read back and checked against the source.
  4. Moves the original Parquet files into a parquet-v1/ subdirectory of the store, unless --no-backup is given.
  5. Publishes the database and rewrites schema.json to version 2.

If anything fails during publication, the migration restores the Parquet files and the original schema.json, so a raised error never leaves a half-migrated store.

After the command completes, the directory is an ordinary v2 store and every other command works against it:

research-tracker --root ./artifacts run show

What changes

  • The five *.parquet files are replaced by store.sqlite; the record model, the API, and the CLI are otherwise unchanged.
  • condition.value was written through pyarrow type unification in 0.1.x and read back as text ('0.1', '3', 'True'). The v2 store keeps an exact str/int/float/bool value, so the migration parses each value back: JSON first, then a Python literal, and otherwise keeps it as text.
  • path_map and the runs_path/metrics_path/... file attributes are gone in 0.2; use db_path. The add_id CLI subcommand was removed, since every table now always carries its primary key.

Backup

By default the migration keeps the old files under <store>/parquet-v1/, so a store can be inspected or restored if something looks wrong. Use --no-backup only when the Parquet files are already archived elsewhere.

It refuses to run twice

The guard is the version marker. migrate reads schema.json and only proceeds when it declares version 1; a store already on version 2 raises MigrationError naming that fact, and a root that already contains a store.sqlite is refused outright rather than written over. Once schema.json says 2 the command will not touch the store again — the same rule that makes ExperimentStore refuse to open a v1 directory.

Editing schema.json by hand to bypass the version check is not supported: a v2 marker without a matching store.sqlite is an incomplete store, and forcing a v1 marker onto SQLite data would silently mix Parquet-era layout with the SQLite schema.

Rolling back

The parquet-v1/ directory is a backup of the source files, not a switch back to the old backend. Restoring it means moving the parquet-v1/*.parquet files back into the store root, removing store.sqlite, and setting schema.json back to {"schema_version": 1} so a 0.1.x installation can open the store again. Anything written through the v2 store after the migration is lost by that rollback, so archive the migrated directory first if you might need it.