Caching

Xorq’s caching system stores intermediate results so iterative ML pipelines don’t recompute work they’ve already done. A cache can invalidate automatically when upstream data changes, or freeze a result so it never recomputes—and it can live inside a database backend or on disk. This page explains the model behind those choices. For step-by-step usage see the Explore caching tutorial and the Cache results by backend how-to; for the full class matrix and how to build each cache see Cache API overview.

Lazy caching

Xorq operations are lazy—nothing runs until you call .execute(). Attaching .cache(...) to an expression adds a CachedNode to the expression graph but computes nothing on its own.

Note

The lazy .cache is a deviation from Ibis cache, where the method eagerly executes the expression.

On .execute(), Xorq walks the graph bottom-up looking for CachedNodes. For each one it computes a cache key, checks storage for that key, and then either:

  • Cache miss: materializes the result, writes it to storage, and proceeds as if it were a hit.
  • Cache hit: returns a reference to the stored table (backend storage) or a deferred read of the stored file (Parquet storage).

The CachedNode is replaced with the storage result and execution continues on the transformed graph. Hits and misses emit OpenTelemetry trace events keyed by the cache key, so a pipeline run shows exactly what was reused.

Why multiple cache classes? Strategy versus storage

Every cache is the combination of two independent choices:

  • A strategy decides how the cache key is computed—which in turn decides when the cache invalidates.
  • A storage decides where the cached data lives.

The classes you import from xorq.caching (SourceCache, ParquetCache, SourceSnapshotCache, …) are named combinations of these two axes. Picking a class is really picking one strategy and one storage.

Strategy: when the cache invalidates

A cache key is a hex string that identifies a cached result, computed by hashing the expression. The two strategies differ in what goes into that hash:

  • ModificationTimeStrategy folds backend-specific change metadata (a row count estimate, a last-altered timestamp, file mtime, a snapshot id) into the key. When the source data changes, the metadata changes, the key changes, the old key is no longer found, and the expression re-executes. This is what “automatic invalidation” means in Xorq—not a timer or a watcher, but a key that changes when its inputs change.
  • SnapshotStrategy keys on the expression’s structure only—table name, path, schema. The key is stable even when the underlying data changes, so the first cached result is served until you delete it. Use it when you’d rather serve a fixed result than pay for recomputation, like model-training inputs or an expensive backfill.

The exact change signal per backend (and its caveats) is tabulated in Cache API overview.

Storage: where the data lives

  • SourceStorage stores the result as a table inside whatever backend you pass as source. If that backend is in-memory (DuckDB, DataFusion, Xorq) the cache is an in-memory table; if it’s a remote database (Postgres, Snowflake, Trino) the cache is a table there.
  • ParquetStorage writes the result as a .parquet file on local disk (default ~/.cache/xorq/parquet/, configurable via XORQ_CACHE_DIR). For Parquet storage the source backend isn’t where the cache lives—it only writes the file on a miss and reads it back on a hit—so the source can be any backend.
  • ParquetTTLStorage extends Parquet storage with a time-to-live: a file older than the TTL is treated as expired.
  • GCStorage stores the Parquet files in a GCS bucket instead of local disk.

Calling .cache() with no arguments builds a SourceCache whose source is the expression’s own backend.

Record batches and streaming

Data moves through the cache as Arrow record batches—chunks of rows with a fixed schema, streamed one at a time by a RecordBatchReader rather than loaded whole into memory. Writing to Parquet storage streams batches from the source expression into the file; cross-engine transfer via into_backend() moves batches over Arrow Flight; ingestion into a database streams batches in over ADBC. The pattern is always the same: produce batches from one backend, consume them in another, without temp files or whole-table loads.

Chaining caches across engines

Because storage and source are decoupled, you can cache at each engine boundary in a multi-engine pipeline—read and filter in one backend, move the result to another with into_backend(), and cache the transferred result so the transfer doesn’t repeat on the next run. Each .cache() is an independent strategy/storage choice, so a single pipeline can mix backend caches and on-disk Parquet caches.

TipWhere to go next