Runs and evaluations¶

Every stored record is a dataclass from research_tracker.trackers, and every record is written through ExperimentStore. The hierarchy is:

graph TD
    R[Run] --> E[Evaluation]
    E --> C[Condition]
    E --> M[Metric]
    E --> A[Artifact]

A run is one training or inference invocation. An evaluation is one named assessment of that run ("test", "val", "ood", ...). Conditions, metrics, and artifacts all hang off an evaluation through evaluation_id, so deleting an evaluation can cascade to exactly its children.

Records and IDs¶

Record	Table	Primary key	Required fields
`Run`	`runs`	`run_id`	`model`, `model_class`, `dataset`, `config`, `status`
`Evaluation`	`evaluations`	`evaluation_id`	`run_id`, `name`
`Condition`	`conditions`	`condition_id`	`evaluation_id`, `name`, `value`
`Metric`	`metrics`	`metric_id`	`evaluation_id`, `metric`, `value`
`Artifact`	`artifacts`	`artifact_id`	`evaluation_id`, `kind`, `path`, `sample_id`

IDs are generated by make_id(prefix) as <prefix>_<uuid4 hex>, e.g. run_2f3a.... The prefixes are run, eval, condition, metric, and artifact. Run.run_id and Evaluation.evaluation_id are created by the store itself; you normally never construct one by hand.

A Run also carries created_at and updated_at timestamps. updated_at is the field sync reconciles on, and every mutation through set_status, add_checkpoint, or import_wandb_run refreshes it. The remaining run fields (git_commit, git_dirty, tracking_backend, tracking_id, tracking_project, loader, dataset_version, datamodule_class, parent_run_id, seed, checkpoint) are optional and stored as null when unset. Run.external_id returns (tracking_backend, tracking_project, tracking_id) or None when there is no external reference.

Pick a record back out by ID:

run = store.get_run(run_id)
evaluations = store.get_evaluations(run_id)
metrics = store.get_metrics(evaluation_id)
conditions = store.get_conditions(evaluation_id)
artifacts = store.get_artifacts(evaluation_id)

get_run returns a Run; the others return lists of the corresponding dataclass. has_run(run_id) is the cheap existence check.

Buffering and `flush()`¶

Runs, evaluations, and artifacts are written immediately. Metrics and conditions are buffered in memory and written in batches:

when their buffer reaches the threshold — metrics_write_every (default 50) or conditions_write_every (default 50), set on ExperimentStore;
at the end of each store.evaluation(...) block;
when the process exits normally (atexit);
when you call store.flush().

flush() is what makes buffered records durable and what makes a store consistent before it is copied or handed to another process. Call it explicitly before copying a store:

store.log_metric(evaluation_id, metric="mean_mse", value=0.012)
# not in the database yet
store.flush()
# now it is

Because the metric and condition buffers are per store instance, two ExperimentStore objects on the same directory have independent buffers.

The evaluation context manager¶

store.evaluation(run_id, name) creates the evaluation row and yields an EvaluationSession:

with store.evaluation(run.run_id, "test") as evaluation:
    evaluation.add_condition("split", "out-of-distribution")
    evaluation.log_metric(metric="mse", value=0.012, sample_id="mouse-01")

The session is a thin wrapper that binds the evaluation_id for you; each method forwards to the matching ExperimentStore writer. It also exposes evaluation_id.

Passing unit to add_condition and comparison, region, or unit to log_metric stores free-text qualifiers alongside the value.

Rollback semantics¶

On a successful exit the context flushes buffered metrics and conditions. If the body or the flush raises, the store rolls back that evaluation:

records of that evaluation still sitting in the metric/condition buffers are discarded;
the evaluation's metric, condition, and artifact rows are deleted, then the evaluation row itself, in one SQLite transaction.

So an evaluation that failed mid-block does not leave partial metrics behind. Two things the rollback deliberately does not do:

It does not delete artifact files. The metadata rows go away; the files on disk stay, because they may be shared with another evaluation or pre-exist the run.
It does not survive abrupt process termination. This is exception cleanup, not a crash-proof transaction: a process killed mid-evaluation keeps whatever was already committed. See limitations.

Rollback is scoped: unrelated runs, evaluations, and buffered records are untouched.

Removing an evaluation outside the context manager uses remove_evaluation_cascade(evaluation_ids, dry_run=False, remove_files=True), which returns a per-table count of removed rows. The CLI's evaluation query ... --remove-all-associated uses the same path.

Relative paths¶

The config, checkpoint (on a run) and path (on an artifact) columns hold paths relative to the store root, never absolute ones. ExperimentStore converts on the way in:

store.create_run(..., config=Path("artifacts/config.yaml"))
run.config  # PosixPath('config.yaml') -- stored relative to artifacts/

Every public write path enforces the containment rule, not just the sync copy step: an absolute path, or one that escapes the root via .., raises ValueError. This is what makes a store directory movable and copyable — a copied store's references still resolve, which is what sync relies on.

add_checkpoint(path, run_id) works the same way: it converts to a store-relative path before writing. Reading a run's checkpoint back through load_checkpoint(run_id) uses Run.loader if set (a "module:callable" path resolved with import_object), otherwise Run.model_class.load_from_checkpoint.