Skip to content

Architecture

This page explains why the store looks the way it does. The rules are also recorded as invariants in AGENTS.md; the code is the authority.

One database, one marker

A store is a directory containing exactly two required files:

artifacts/
├── schema.json     # {"schema_version": 2}
├── store.sqlite    # runs, evaluations, conditions, metrics, artifacts
└── .sync.lock      # file lock for writers

schema.json is the version marker, not a schema definition. The real schema lives in ExperimentStore._TABLE_SCHEMAS — a tuple of (column, role) pairs per logical table — and the tables are created with SQLite column types derived from those roles.

The marker is written through a temporary file plus os.replace, so a concurrent constructor either reads the previous state or the complete file, never a half-written one. It is read and validated before the lock is acquired, so a rejected store gains neither a .sync.lock file nor anything else. ExperimentStore raises SchemaMismatchError in three cases:

  • schema.json's version differs from trackers.SCHEMA_VERSION;
  • the directory has schema.json but no store.sqlite, or the reverse (a partial layout, unless a .sync.lock shows another process is mid-create);
  • the database exists but is missing a canonical table or column.

An existing store is validated, never repaired: a dropped table surfaces as an error instead of being silently recreated.

Rollback journal, never WAL

Every connection sets:

PRAGMA busy_timeout = 30000;
PRAGMA journal_mode = DELETE;
BEGIN IMMEDIATE;

The rollback journal is what keeps a store portable as a single file: WAL mode would leave -wal and -shm sidecars holding committed data, so a directory copy could silently lose records. Concurrency between separate jobs is handled by the .sync.lock FileLock rather than by SQLite's own writer model.

A rollback journal is not a snapshot guarantee on its own: copying while a transaction is open can capture a half-applied state. That is why copies are only valid from a flushed, quiescent source (see sync).

BEGIN IMMEDIATE takes the write lock at the start rather than on first write, so two writers cannot both begin and then deadlock on upgrade. One logical operation that touches several tables commits in one transaction, so an interrupted cascade delete or sync install cannot leave half its rows behind.

Explicit per-column codecs

Nothing reads a stored record through pandas type inference. Each column has a role (text, id, int, float, bool, timestamp, json) and a matching codec in utils.py:

Role SQLite type Python value Read as
text, id TEXT str object dtype, no inference
int INTEGER int Int64
float REAL float float64
bool INTEGER 0/1 boolean
timestamp TEXT ISO 8601, UTC datetime64[ns, UTC]
json TEXT json.dumps(value) decoded Python object

On the way in, encode_value maps missing values (None, NaT, NaN) to NULL, unwraps numpy scalars before JSON encoding, localizes naive timestamps to UTC, and converts bool to int. On the way out, read_sqlite_frame builds each column from a Series with its declared dtype.

Pandas inference was rejected because it is lossy exactly where research data is subtle:

  • pd.read_sql_query infers a whole column at once. A nullable 64-bit integer column — a seed, say — comes back as float64, and large values round silently.
  • Missing strings come back as NaN, which is not a str and compares unequal to itself; the store keeps None.
  • 1, 1.0, and True are different values. Inference collapses them, and the store needs the distinction both to preserve condition.value's type and to compare records during sync.

Because the codecs are explicit, a value that cannot be encoded for its column is caught by _preflight_table before anything is written, with the column name and the offending value in the message.

Transaction boundaries

Writes take the store lock, then a single BEGIN IMMEDIATE transaction:

  • Record appends (create_run, create_evaluation, log_artifact, and a metric/condition batch) — one transaction per append or flush batch.
  • Run field updates (set_status, add_checkpoint) — SELECT COUNT(*) to assert exactly one matching run, then one UPDATE that also refreshes updated_at. A wrong run ID raises ValueError instead of creating a row.
  • Evaluation cascade (remove_evaluation_cascade) — counts and deletes metric, condition, artifact, then evaluation, all in one transaction; a dry_run issues no writes and explicitly ROLLBACKs the read transaction.
  • Sync install — all five merged tables replaced in one transaction, after validation of every table and after file copying.
  • Whole-table replace (update_table) — columns, uniqueness, and stored paths validated first, then one transaction.

The lock is held across more than the transaction where correctness needs it. In the evaluation cascade, the transaction commits while the lock is still held, so the survivor read and the artifact unlink below cannot race a concurrent log_artifact that would add a reference back.

Buffered metrics and conditions are the exception to "one call, one transaction": _buffer_record accumulates until the threshold, then flushes the whole batch in one transaction. flush() drains both buffers; it is registered with atexit and runs at the end of every store.evaluation(...) block.

Artifact purge semantics

Deleting an evaluation deletes metadata; it does not blindly delete files:

  1. Before deleting the rows, the cascade reads the path of every artifact of the evaluations being removed — the candidates.
  2. The transaction deletes the child rows and the evaluation rows.
  3. Still under the lock, it re-reads the surviving artifact paths and unlinks only candidates that no survivor references. References are compared as normalized absolute paths, so result and sub/../result collapse to one file and a still-referenced file is never unlinked.

Metadata commits before the unlink, deliberately: a crash in between leaves the file on disk (harmless) rather than deleting a file whose metadata still exists (destructive). The same rule applies to evaluation rollback, which removes rows and leaves artifact files alone.

Paths

config, checkpoint, and artifact path are stored relative to the store root. ensure_path resolves and relativizes on the way in, and every public write path checks containment, not just the sync copy step. This is what makes a copy of the directory self-contained and mergeable.

Reads

Reads open a short-lived connection with the busy timeout set, SELECT the canonical columns ORDER BY rowid, and build the frame with the declared dtypes. There is no query pushdown: load_table reads the whole table and filtering happens in pandas (the CLI's query is DataFrame.query on the loaded frame). For the sizes this store targets that is the right trade — simple and type-exact.