Limitations

The store is deliberately small, and its edges are worth knowing before you rely on it.

Copying a store requires a quiescent source

The database uses a rollback journal, which keeps a store portable as a single file, but that is not a snapshot guarantee on its own. Copying while a transaction is open can capture a half-applied state, and copying while a writer still holds unflushed metrics or conditions captures less than the whole store. Copy only from a store that no process is mid-writing and that has been flushed. See sync.

Sync is not fully atomic

Sync validates every merged table before copying a single file, and installs all five tables in one transaction — a rejected merge never reaches the install step. That is not a full rollback, though:

• Sync first flushes the destination's own buffered metrics and conditions under the store lock. Those writes are already committed by the time the merge is validated, so a later failure does not undo them.
• Files copied before a later step fails are not removed.

A failed sync leaves the local store's previously buffered records written and any already-copied files in place. Only the table install is all-or-nothing.

The post-commit unlink is not crash-atomic

Deleting an evaluation commits its metadata deletion first and unlinks orphaned artifact files afterwards. A crash between the two leaves the file on disk with no metadata pointing at it. This ordering is intentional — a stray file is harmless, while deleting a file whose metadata still exists is destructive — but it does mean the store can accumulate files nothing references.

atexit flush is best-effort

Buffered metrics and conditions are flushed at the end of an evaluation block, at their batch threshold, on flush(), and on normal process exit via atexit. atexit does not run on SIGKILL, an abort, a segfault, or a power loss, and it does not run when a worker process is hard-killed by a scheduler. Anything written but not yet flushed in those cases is lost. Call flush() explicitly before copying a store, before handing it to another process, or at any point where the process might not exit cleanly.

Evaluation rollback is exception cleanup, not a crash-proof transaction: a process killed mid-evaluation keeps whatever was already committed.

update_table is an unconditional whole-table replace

update_table(kind, df) deletes every row in the table and inserts the frame you give it, in one transaction. It validates the canonical columns, rejects duplicate primary keys, and rejects absolute or escaping stored paths — but it will not help you merge. Passing a frame that is missing rows silently drops them. It exists for bulk loads such as the migration, not for incremental edits.

Loads read whole tables

There is no query pushdown. load_table and its per-table wrappers read the entire table into a pandas frame with explicit dtypes, and the CLI's query runs DataFrame.query over that frame in memory. Memory use grows with the table, and a query's cost is a full scan. This is fine for the sizes the store targets; it is not built for millions of sample-level metrics in one store.

Buffers are per instance

The metric and condition buffers live on the ExperimentStore object. Two ExperimentStore objects over the same directory have independent buffers, and neither's flush() drains the other's. Create one store per process (the Lightning callback does) and explicitly flush before sharing a directory.

condition.value has a fixed type set

add_condition accepts str, int, float, or bool; the value is stored as JSON and keeps its exact Python type on read, with 1, 1.0, and True staying distinct. Other types (a datetime, a list, a dict) are not part of the supported set.

Paths must stay inside the store

Config, checkpoint, and artifact paths are stored relative to the store root and must remain contained. An absolute path, or one that escapes the root with .., raises ValueError on every public write path. A file that must live outside the store has to be copied in first, or referenced by your own convention in a non-path column.