# xorq

> Executable memory for tabular data: portable, multi-engine pipelines with caching, lineage, and a git-backed catalog agents and humans can reuse.

----------------------------------------------------------------------
This is the API documentation for the xorq library.
----------------------------------------------------------------------

## Core Operations

APIs for reading and returning data

### connect(session_config: 'SessionConfig | None' = None) -> 'Backend'

Create a xorq backend.

### execute(expr: 'ir.Expr', **kwargs: 'Any')

Execute an expression against its backend if one exists.

Parameters
----------
kwargs
    Keyword arguments

Examples
--------
>>> import xorq.api as xo
>>> t = xo.examples.penguins.fetch()
>>> t.execute()
       species     island  bill_length_mm  ...  body_mass_g     sex  year
0       Adelie  Torgersen            39.1  ...       3750.0    male  2007
1       Adelie  Torgersen            39.5  ...       3800.0  female  2007
2       Adelie  Torgersen            40.3  ...       3250.0  female  2007
3       Adelie  Torgersen             NaN  ...          NaN    None  2007
4       Adelie  Torgersen            36.7  ...       3450.0  female  2007
..         ...        ...             ...  ...          ...     ...   ...
339  Chinstrap      Dream            55.8  ...       4000.0    male  2009
340  Chinstrap      Dream            43.5  ...       3400.0  female  2009
341  Chinstrap      Dream            49.6  ...       3775.0    male  2009
342  Chinstrap      Dream            50.8  ...       4100.0    male  2009
343  Chinstrap      Dream            50.2  ...       3775.0  female  2009
[344 rows x 8 columns]

Scalar parameters can be supplied dynamically during execution.
>>> species = xo.param("string")
>>> expr = t.filter(t.species == species).order_by(t.bill_length_mm)
>>> expr.execute(limit=3, params={species: "Gentoo"})
  species  island  bill_length_mm  ...  body_mass_g     sex  year
0  Gentoo  Biscoe            40.9  ...         4650  female  2007
1  Gentoo  Biscoe            41.7  ...         4700  female  2009
2  Gentoo  Biscoe            42.0  ...         4150  female  2007
<BLANKLINE>
[3 rows x 8 columns]

### memtable(data, *, columns: 'Iterable[str] | None' = None, schema: 'SchemaLike | None' = None, name: 'str | None' = None) -> 'Table'

Construct an ibis table expression from in-memory data.

Parameters
----------
data
    A table-like object (`pandas.DataFrame`, `pyarrow.Table`, or
    `polars.DataFrame`), or any data accepted by the `pandas.DataFrame`
    constructor (e.g. a list of dicts).

    Note that ibis objects (e.g. `MapValue`) may not be passed in as part
    of `data` and will result in an error.

    Do not depend on the underlying storage type (e.g., pyarrow.Table),
    it's subject to change across non-major releases.
columns
    Optional [](`typing.Iterable`) of [](`str`) column names. If provided,
    must match the number of columns in `data`.
schema
    Optional [`Schema`](./schemas.qmd#ibis.expr.schema.Schema).
    The functions use `data` to infer a schema if not passed.
name
    Optional name of the table.

Returns
-------
Table
    A table expression backed by in-memory data.

Examples
--------
>>> import ibis
>>> t = ibis.memtable([{"a": 1}, {"a": 2}])
>>> t
InMemoryTable
  data:
    PandasDataFrameProxy:
         a
      0  1
      1  2

>>> t = ibis.memtable([{"a": 1, "b": "foo"}, {"a": 2, "b": "baz"}])
>>> t
InMemoryTable
  data:
    PandasDataFrameProxy:
         a    b
      0  1  foo
      1  2  baz

Create a table literal without column names embedded in the data and pass
`columns`

>>> t = ibis.memtable([(1, "foo"), (2, "baz")], columns=["a", "b"])
>>> t
InMemoryTable
  data:
    PandasDataFrameProxy:
         a    b
      0  1  foo
      1  2  baz

Create a table literal without column names embedded in the data. Ibis
generates column names if none are provided.

>>> t = ibis.memtable([(1, "foo"), (2, "baz")])
>>> t
InMemoryTable
  data:
    PandasDataFrameProxy:
         col0 col1
      0     1  foo
      1     2  baz

### deferred_read_csv(path: 'str | Path', con: 'Backend | None' = None, table_name: 'str | None' = None, schema: 'Schema | None' = None, normalize_method: 'Callable' = <function normalize_read_path_stat>, relocatable: 'bool' = False, **kwargs: 'Any') -> 'ir.Table'

Create a deferred read operation for CSV files that will execute only when needed.

This function creates a representation of a read operation that doesn't immediately
load data into memory. Instead, it registers the operation to be performed when
the resulting expression is executed.

The function works with different backend engines (pandas, duckdb, postgres, etc.)
and adapts the read parameters accordingly.


Parameters
----------
path : str or Path
    The path to the CSV file to be read. This can be a local file path or a URL.

con : Backend, optional
    The connection object representing the backend where the CSV will be read.
    This can be any backend that supports reading CSV files (pandas, duckdb,
    postgres, etc.).

table_name : str, optional
    The name to give to the resulting table in the backend. If not provided,
    a unique name will be generated automatically.

schema : Schema, optional
    The schema definition for the CSV data. If not provided, the schema will
    be inferred from the data by sampling the CSV file.

relocatable : bool, optional
    When True, ``xorq build`` will copy the backing file into the build
    artifact and rewrite the path so the archive is self-contained.

kwargs : Any
    Additional keyword arguments that will be passed to the backend's read_csv
    method.

Returns
-------
Expr
    An expression representing the deferred read operation.

### deferred_read_parquet(path: 'str | Path', con: 'Backend | None' = None, table_name: 'str | None' = None, schema: 'Schema | None' = None, normalize_method: 'Callable' = <function normalize_read_path_stat>, relocatable: 'bool' = False, **kwargs: 'Any') -> 'ir.Table'

 Create a deferred read operation for Parquet files that will execute only when needed.

This function creates a representation of a read operation that doesn't immediately
load data into memory. Instead, it registers the operation to be performed when
the resulting expression is executed.

Parameters
----------
path : str or Path
    The path to the Parquet file or directory to be read.

con : Backend, optional
    The connection object representing the backend where the Parquet data will be read.

table_name : str, optional
    The name to give to the resulting table in the backend. If not provided,
    a unique name will be generated automatically.

normalize_method : Callable, optional
    The method that returns the values to be used in the hashing of the Read operation.

relocatable : bool, optional
    When True, ``xorq build`` will copy the backing file into the build
    artifact and rewrite the path so the archive is self-contained.

**kwargs : dict
    Additional keyword arguments passed to the backend's read_parquet method.

Returns
-------
Expr
    An expression representing the deferred read operation.

### to_csv(expr: 'ir.Expr', path: 'str | Path', params: 'Mapping[ir.Scalar, Any] | None' = None, **kwargs: 'Any')

Write the results of executing the given expression to a CSV file.

This method is eager and will execute the associated expression
immediately.

Parameters
----------
path
    The data source. A string or Path to the CSV file.
params
    Mapping of scalar parameter expressions to value.
**kwargs
    Additional keyword arguments passed to pyarrow.csv.CSVWriter

https://arrow.apache.org/docs/python/generated/pyarrow.csv.CSVWriter.htmlditional keyword arguments passed to pyarrow.csv.CSVWriter

### to_json(expr: 'ir.Expr', path: 'str | Path | TextIOWrapper', params: 'Mapping[ir.Scalar, Any] | None' = None)

Write the results of `expr` to a NDJSON file.

This method is eager and will execute the associated expression
immediately.

Parameters
----------
path
    The data source. A string or Path to the Delta Lake table.
**kwargs
    Additional, backend-specific keyword arguments.

https://github.com/ndjson/ndjson-spec

### to_parquet(expr: 'ir.Expr', path: 'str | Path', params: 'Mapping[ir.Scalar, Any] | None' = None, **kwargs: 'Any')

Write the results of executing the given expression to a parquet file.

This method is eager and will execute the associated expression
immediately.

See https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetWriter.html for details.

Parameters
----------
path
    A string or Path where the Parquet file will be written.
params
    Mapping of scalar parameter expressions to value.
**kwargs
    Additional keyword arguments passed to pyarrow.parquet.ParquetWriter

Examples
--------
Write out an expression to a single parquet file.

>>> import ibis
>>> import tempfile
>>> penguins = ibis.examples.penguins.fetch()
>>> penguins.to_parquet(tempfile.mktemp())

### to_pyarrow(expr: 'ir.Expr', **kwargs: 'Any')

Execute expression and return results in as a pyarrow table.

This method is eager and will execute the associated expression
immediately.

Parameters
----------
kwargs
    Keyword arguments

Returns
-------
Table
    A pyarrow table holding the results of the executed expression.

### to_pyarrow_batches(expr: 'ir.Expr', *, chunk_size: 'int' = 1000000, **kwargs: 'Any')

Execute expression and return a RecordBatchReader.

This method is eager and will execute the associated expression
immediately.

The returned reader must be **fully consumed**: drain threads are joined
and temp tables are dropped only after the last batch is read. Consuming
partially (an early ``break``) or discarding the reader leaks those
resources. Drain failures are surfaced when the reader is exhausted.

Parameters
----------
chunk_size
    Maximum number of rows in each returned record batch.
kwargs
    Keyword arguments

Returns
-------
results
    RecordBatchReader

### to_sql(expr: 'ir.Expr', compiler=None, pretty: 'bool' = True) -> 'SQLString'

Return the formatted SQL string for an expression.

Parameters
----------
expr
    Ibis expression.
compiler
    The target compiler to use to translate the Ibis expr
pretty
    Whether to use pretty formatting.

Returns
-------
str
    Formatted SQL string

### register(source: 'str | Path | pa.Table | pa.RecordBatch | pa.Dataset | pd.DataFrame', table_name: 'str | None' = None, **kwargs: 'Any')

### get_plans(expr: 'ir.Expr') -> 'dict'

## Data Operations

### Table(*args, **kwargs)

An immutable and lazy dataframe.

Table.agg(self, metrics: 'Sequence[ir.Scalar] | None' = (), by: 'Sequence[ir.Value] | None' = (), having: 'Sequence[ir.BooleanValue] | None' = (), **kwargs: 'ir.Value') -> 'Table'

Aggregate a table with a given set of reductions grouping by `by`.

Table.aggregate(self, metrics: 'Sequence[ir.Scalar] | None' = (), by: 'Sequence[ir.Value] | None' = (), having: 'Sequence[ir.BooleanValue] | None' = (), **kwargs: 'ir.Value') -> 'Table'

Aggregate a table with a given set of reductions grouping by `by`.

Table.alias(self, alias: 'str') -> 'ir.Table'

Create a table expression with a specific name `alias`.

Table.anti_join(self: 'ir.Table', right: 'ir.Table', predicates: 'str | Sequence[str | tuple[str | ir.Column, str | ir.Column] | ir.BooleanValue]' = (), *, lname: 'str' = '', rname: 'str' = '{name}_right') -> 'ir.Table'

Perform a join between two tables.

Table.any_inner_join(self: 'ir.Table', right: 'ir.Table', predicates: 'str | Sequence[str | tuple[str | ir.Column, str | ir.Column] | ir.BooleanValue]' = (), *, lname: 'str' = '', rname: 'str' = '{name}_right') -> 'ir.Table'

Perform a join between two tables.

Table.any_left_join(self: 'ir.Table', right: 'ir.Table', predicates: 'str | Sequence[str | tuple[str | ir.Column, str | ir.Column] | ir.BooleanValue]' = (), *, lname: 'str' = '', rname: 'str' = '{name}_right') -> 'ir.Table'

Perform a join between two tables.

Table.as_scalar(self) -> 'ir.ScalarExpr'

Inform ibis that the table expression should be treated as a scalar.

Table.as_table(self) -> 'Table'

Promote the expression to a table.

Table.asof_join(left: 'Table', right: 'Table', on: 'str | ir.BooleanColumn', predicates: 'str | ir.Column | Sequence[str | ir.Column]' = (), tolerance: 'str | ir.IntervalScalar | None' = None, *, lname: 'str' = '', rname: 'str' = '{name}_right') -> 'Table'

Perform an "as-of" join between `left` and `right`.

Table.bind(self, *args: 'Any', **kwargs: 'Any') -> 'tuple[Value, ...]'

Bind column values to a table expression.

Table.cache(self, cache=None) -> 'Table'

Cache the results of a computation to improve performance on subsequent executions.
This method allows you to cache the results of a computation either in memory, on disk
using Parquet files, or in a database table. The caching strategy and cache location
are determined by the cache parameter.

Table.cast(self, schema: 'SchemaLike') -> 'Table'

Cast the columns of a table.

Table.columns (property)

The list of column names in this table.

Table.compile(self, limit: 'int | None' = None, params: 'Mapping[ir.Value, Any] | None' = None, pretty: 'bool' = False)

Compile to an execution target.

Table.count(self, where: 'ir.BooleanValue | None' = None) -> 'ir.IntegerScalar'

Compute the number of rows in the table.

Table.cross_join(left: 'Table', right: 'Table', *rest: 'Table', lname: 'str' = '', rname: 'str' = '{name}_right') -> 'Table'

Compute the cross join of a sequence of tables.

Table.describe(self, quantile: 'Sequence[ir.NumericValue | float]' = (0.25, 0.5, 0.75)) -> 'Table'

Return summary information about a table.

Table.difference(self, table: 'Table', *rest: 'Table', distinct: 'bool' = True) -> 'Table'

Compute the set difference of multiple table expressions.

Table.distinct(self, *, on: 'str | Iterable[str] | s.Selector | None' = None, keep: "Literal['first', 'last'] | None" = 'first') -> 'Table'

Return a Table with duplicate rows removed.

Table.drop(self, *fields: 'str | Selector') -> 'Table'

Remove fields from a table.

Table.drop_null(self, subset: 'Sequence[str] | str | None' = None, how: "Literal['any', 'all']" = 'any') -> 'Table'

Remove rows with null values from the table.

Table.dropna(self, subset: 'Sequence[str] | str | None' = None, how: "Literal['any', 'all']" = 'any') -> 'Table'

Deprecated - use `drop_null` instead.

Table.equals(self, other)

Return whether this expression is _structurally_ equivalent to `other`.

Table.execute(self: 'ir.Expr', **kwargs: 'Any')

Execute an expression against its backend if one exists.

Table.fill_null(self, replacements: 'ir.Scalar | Mapping[str, ir.Scalar]') -> 'Table'

Fill null values in a table expression.

Table.fillna(self, replacements: 'ir.Scalar | Mapping[str, ir.Scalar]') -> 'Table'

Deprecated - use `fill_null` instead.

Table.filter(self, *predicates: 'ir.BooleanValue | Sequence[ir.BooleanValue] | IfAnyAll') -> 'Table'

Select rows from `table` based on `predicates`.

Table.get_name(self) -> 'str'

Return the fully qualified name of the table.

Table.group_by(self, *by: 'str | ir.Value | Iterable[str] | Iterable[ir.Value] | None', **key_exprs: 'str | ir.Value | Iterable[str] | Iterable[ir.Value]') -> 'GroupedTable'

Create a grouped table expression.

Table.has_name(self)

Check whether this expression has an explicit name.

Table.hashing_tag(self, tag: 'Hashable', **kwargs: 'Hashable') -> 'Table'

Wrap this table in a `HashingTag` node carrying arbitrary metadata.

Table.head(self, n: 'int' = 5) -> 'Table'

Select the first `n` rows of a table.

Table.info(self) -> 'Table'

Return summary information about a table.

Table.inner_join(self: 'ir.Table', right: 'ir.Table', predicates: 'str | Sequence[str | tuple[str | ir.Column, str | ir.Column] | ir.BooleanValue]' = (), *, lname: 'str' = '', rname: 'str' = '{name}_right') -> 'ir.Table'

Perform a join between two tables.

Table.intersect(self, table: 'Table', *rest: 'Table', distinct: 'bool' = True) -> 'Table'

Compute the set intersection of multiple table expressions.

Table.into_backend(self, con: 'BaseBackend', name: 'str | None' = None) -> 'ir.Table'

Converts the Expr to a table in the given backend `con` with an optional table name `name`.

Table.join(left: 'Table', right: 'Table', predicates: 'str | Sequence[str | ir.BooleanColumn | Literal[True] | Literal[False] | tuple[str | ir.Column | ir.Deferred, str | ir.Column | ir.Deferred]]' = (), how: 'JoinKind' = 'inner', *, lname: 'str' = '', rname: 'str' = '{name}_right') -> 'Table'

Perform a join between two tables.

Table.left_join(self: 'ir.Table', right: 'ir.Table', predicates: 'str | Sequence[str | tuple[str | ir.Column, str | ir.Column] | ir.BooleanValue]' = (), *, lname: 'str' = '', rname: 'str' = '{name}_right') -> 'ir.Table'

Perform a join between two tables.

Table.limit(self, n: 'int | None', offset: 'int' = 0) -> 'Table'

Select `n` rows from `self` starting at `offset`.

Table.ls (property)

xorq (LETSQL) accessor for inspecting kind, backends, and cache state.

Table.mutate(self, *exprs: 'Sequence[ir.Expr] | None', **mutations: 'ir.Value') -> 'Table'

Add columns to a table expression.

Table.nunique(self, where: 'ir.BooleanValue | None' = None) -> 'ir.IntegerScalar'

Compute the number of unique rows in the table.

Table.op(self) -> 'ops.Node'

Table.order_by(self, *by: 'str | ir.Column | s.Selector | Sequence[str] | Sequence[ir.Column] | Sequence[s.Selector] | None') -> 'Table'

Sort a table by one or more expressions.

Table.outer_join(self: 'ir.Table', right: 'ir.Table', predicates: 'str | Sequence[str | tuple[str | ir.Column, str | ir.Column] | ir.BooleanValue]' = (), *, lname: 'str' = '', rname: 'str' = '{name}_right') -> 'ir.Table'

Perform a join between two tables.

Table.pipe(self, f, *args: 'Any', **kwargs: 'Any') -> 'Expr'

Compose `f` with `self`.

Table.pivot_longer(self, col: 'str | s.Selector', *, names_to: 'str | Iterable[str]' = 'name', names_pattern: 'str | re.Pattern' = '(.+)', names_transform: 'Callable[[str], ir.Value] | Mapping[str, Callable[[str], ir.Value]] | None' = None, values_to: 'str' = 'value', values_transform: 'Callable[[ir.Value], ir.Value] | Deferred | None' = None) -> 'Table'

Transform a table from wider to longer.

Table.pivot_wider(self, *, id_cols: 's.Selector | None' = None, names_from: 'str | Iterable[str] | s.Selector' = 'name', names_prefix: 'str' = '', names_sep: 'str' = '_', names_sort: 'bool' = False, names: 'Iterable[str] | None' = None, values_from: 'str | Iterable[str] | s.Selector' = 'value', values_fill: 'int | float | str | ir.Scalar | None' = None, values_agg: 'str | Callable[[ir.Value], ir.Scalar] | Deferred' = 'arbitrary') -> 'Table'

Pivot a table to a wider format.

Table.preview(self, *, max_rows: 'int | None' = None, max_columns: 'int | None' = None, max_length: 'int | None' = None, max_string: 'int | None' = None, max_depth: 'int | None' = None, console_width: 'int | float | None' = None) -> 'RichTable'

Return a subset as a Rich Table.

Table.projection(self, *exprs: 'ir.Value | str | Iterable[ir.Value | str]', **named_exprs: 'ir.Value | str') -> 'Table'

Compute a new table expression using `exprs` and `named_exprs`.

Table.relabel(self, substitutions: "Mapping[str, str] | Callable[[str], str | None] | str | Literal['snake_case', 'ALL_CAPS']") -> 'Table'

Deprecated in favor of `Table.rename`.

Table.relocate(self, *columns: 'str | s.Selector', before: 'str | s.Selector | None' = None, after: 'str | s.Selector | None' = None, **kwargs: 'str') -> 'Table'

Relocate `columns` before or after other specified columns.

Table.rename(self, method: "str | Callable[[str], str | None] | Literal['snake_case', 'ALL_CAPS'] | Mapping[str, str] | None" = None, /, **substitutions: 'str') -> 'Table'

Rename columns in the table.

Table.right_join(self: 'ir.Table', right: 'ir.Table', predicates: 'str | Sequence[str | tuple[str | ir.Column, str | ir.Column] | ir.BooleanValue]' = (), *, lname: 'str' = '', rname: 'str' = '{name}_right') -> 'ir.Table'

Perform a join between two tables.

Table.rowid(self) -> 'ir.IntegerValue'

A unique integer per row.

Table.sample(self, fraction: 'float', *, method: "Literal['row', 'block']" = 'row', seed: 'int | None' = None) -> 'Table'

Sample a fraction of rows from a table.

Table.schema(self) -> 'sch.Schema'

Return the [Schema](./schemas.qmd#ibis.expr.schema.Schema) for this table.

Table.select(self, *exprs: 'ir.Value | str | Iterable[ir.Value | str]', **named_exprs: 'ir.Value | str') -> 'Table'

Compute a new table expression using `exprs` and `named_exprs`.

Table.semi_join(self: 'ir.Table', right: 'ir.Table', predicates: 'str | Sequence[str | tuple[str | ir.Column, str | ir.Column] | ir.BooleanValue]' = (), *, lname: 'str' = '', rname: 'str' = '{name}_right') -> 'ir.Table'

Perform a join between two tables.

Table.sql(self, query: 'str', dialect: 'str | None' = None) -> 'ir.Table'

Run a SQL query against a table expression.

Table.tag(self, tag: 'Hashable', **kwargs: 'Hashable') -> 'Table'

Wrap this table in a `Tag` node carrying arbitrary metadata.

Table.tee(self, target: 'WriteThrough | BaseBackend | str | os.PathLike', *, table_name: 'str | None' = None, drain: 'bool' = True, **kwargs: 'Any') -> 'Table'

Pass rows through while writing them as a side effect (ADR-0014).

Table.to_array(self) -> 'ir.Column'

View a single column table as an array.

Table.to_csv(self, path: 'str | Path', *, params: 'Mapping[ir.Scalar, Any] | None' = None, **kwargs: 'Any') -> 'None'

Write the results of executing the given expression to a CSV file.

Table.to_json(self, path: 'str | Path', *, params: 'Mapping[ir.Scalar, Any] | None' = None, **kwargs: 'Any') -> 'None'

Write the results of `expr` to a NDJSON file.

Table.to_pandas(self, **kwargs) -> 'pd.DataFrame'

Convert a table expression to a pandas DataFrame.

Table.to_parquet(self: 'ir.Expr', path: 'str | Path', params: 'Mapping[ir.Scalar, Any] | None' = None, **kwargs: 'Any')

Write the results of executing the given expression to a parquet file.

Table.to_pyarrow(self: 'ir.Expr', **kwargs: 'Any')

Execute expression and return results in as a pyarrow table.

Table.to_pyarrow_batches(self: 'ir.Expr', *, chunk_size: 'int' = 1000000, **kwargs: 'Any')

Execute expression and return a RecordBatchReader.

Table.try_cast(self, schema: 'SchemaLike') -> 'Table'

Cast the columns of a table.

Table.unbind(self) -> 'ir.Table'

Return an expression built on `UnboundTable` instead of backend-specific objects.

Table.union(self, table: 'Table', *rest: 'Table', distinct: 'bool' = False) -> 'Table'

Compute the set union of multiple table expressions.

Table.unnest(self, column, offset: 'str | None' = None, keep_empty: 'bool' = False) -> 'Table'

Unnest an array `column` from a table.

Table.unpack(self, *columns: 'str') -> 'Table'

Project the struct fields of each of `columns` into `self`.

Table.value_counts(self) -> 'ir.Table'

Compute a frequency table of this table's values.

Table.view(self) -> 'Table'

Create a new table expression distinct from the current one.

Table.visualize(self, format: 'str' = 'svg', *, label_edges: 'bool' = False, verbose: 'bool' = False, node_attr: 'Mapping[str, str] | None' = None, node_attr_getter: 'NodeAttributeGetter | None' = None, edge_attr: 'Mapping[str, str] | None' = None, edge_attr_getter: 'EdgeAttributeGetter | None' = None) -> 'None'

Visualize an expression as a GraphViz graph in the browser.

Table.window_by(self, time_col: 'str | ir.Value') -> 'WindowedTable'

### GroupedTable(table: Relation, groupings: Annotated[tuple[Value, ...], Length(at_least=1, at_most=None)], orderings: tuple[SortKey, ...] = (), havings: tuple[Value[Boolean, DataShape], ...] = ())

An intermediate table expression to hold grouping information.

GroupedTable.agg(self, *metrics, **kwds) -> 'ir.Table'

Compute aggregates over a group by.

GroupedTable.aggregate(self, *metrics, **kwds) -> 'ir.Table'

Compute aggregates over a group by.

GroupedTable.count(self) -> 'ir.Table'

Computing the number of rows per group.

GroupedTable.having(self, *predicates: 'ir.BooleanScalar') -> 'GroupedTable'

Add a post-aggregation result filter `expr`.

GroupedTable.mutate(self, *exprs: 'ir.Value | Sequence[ir.Value]', **kwexprs: 'ir.Value') -> 'ir.Table'

Return a table projection with window functions applied.

GroupedTable.order_by(self, *by: 'ir.Value') -> 'GroupedTable'

Sort a grouped table expression by `expr`.

GroupedTable.over(self, window=None, *, rows=None, range=None, group_by=None, order_by=None) -> 'GroupedTable'

Apply a window over the input expressions.

GroupedTable.projection(self, *exprs, **kwexprs) -> 'ir.Table'

Project new columns out of the grouped table.

GroupedTable.select(self, *exprs, **kwexprs) -> 'ir.Table'

Project new columns out of the grouped table.

GroupedTable.size(self) -> 'ir.Table'

Computing the number of rows per group.

### Value(*args, **kwargs)

Base class for a data generating expression having a known type.

Value.asc(self, nulls_first: 'bool' = False) -> 'ir.Value'

Sort an expression ascending.

Value.between(self, lower: 'Value', upper: 'Value') -> 'ir.BooleanValue'

Check if this expression is between `lower` and `upper`, inclusive.

Value.case(self) -> 'bl.SimpleCaseBuilder'

Create a SimpleCaseBuilder to chain multiple if-else statements.

Value.cases(self, branch: 'tuple[Value, Value]', /, *branches: 'tuple[Value, Value]', else_: 'Value | None' = None) -> 'Value'

Create a multi-branch if-else expression.

Value.cast(self, target_type: 'Any') -> 'Value'

Cast expression to indicated data type.

Value.coalesce(self, *args: 'Value') -> 'Value'

Return the first non-null value from `args`.

Value.collect(self, where: 'ir.BooleanValue | None' = None, order_by: 'Any' = None, include_null: 'bool' = False) -> 'ir.ArrayScalar'

Aggregate this expression's elements into an array.

Value.desc(self, nulls_first: 'bool' = False) -> 'ir.Value'

Sort an expression descending.

Value.fill_null(self, fill_value: 'Scalar') -> 'Value'

Replace any null values with the indicated fill value.

Value.fillna(self, fill_value: 'Scalar') -> 'Value'

Deprecated - use `fill_null` instead.

Value.greatest(self, *args: 'ir.Value') -> 'ir.Value'

::: {.callout-warning}
## DEPRECATED: `Value.greatest` is deprecated as of v8.0.0; use ibis.greatest(self, rest...) instead
:::

Value.group_concat(self, sep: 'str' = ',', where: 'ir.BooleanValue | None' = None, order_by: 'Any' = None) -> 'ir.StringScalar'

Concatenate values using the indicated separator to produce a string.

Value.hash(self) -> 'ir.IntegerValue'

Compute an integer hash value.

Value.identical_to(self, other: 'Value') -> 'ir.BooleanValue'

Return whether this expression is identical to other.

Value.isin(self, values: 'Value | Sequence[Value]') -> 'ir.BooleanValue'

Check whether this expression's values are in `values`.

Value.isnull(self) -> 'ir.BooleanValue'

Return whether this expression is NULL.

Value.least(self, *args: 'ir.Value') -> 'ir.Value'

::: {.callout-warning}
## DEPRECATED: `Value.least` is deprecated as of v8.0.0; use ibis.least(self, rest...) instead
:::

Value.name(self, name)

Rename an expression to `name`.

Value.notin(self, values: 'Value | Sequence[Value]') -> 'ir.BooleanValue'

Check whether this expression's values are not in `values`.

Value.notnull(self) -> 'ir.BooleanValue'

Return whether this expression is not NULL.

Value.nullif(self, null_if_expr: 'Value') -> 'Value'

Set values to null if they equal the values `null_if_expr`.

Value.over(self, window=None, *, rows=None, range=None, group_by=None, order_by=None) -> 'Value'

Construct a window expression.

Value.substitute(self, value: 'Value | dict', replacement: 'Value | None' = None, else_: 'Value | None' = None)

Replace values given in `values` with `replacement`.

Value.to_pandas(self, **kwargs) -> 'pd.Series'

Convert a column expression to a pandas Series or scalar object.

Value.try_cast(self, target_type: 'Any') -> 'Value'

Try cast expression to indicated data type.

Value.type(self) -> 'dt.DataType'

Return the [DataType](./datatypes.qmd) of `self`.

Value.typeof(self) -> 'ir.StringValue'

Return the string name of the datatype of self.

### Scalar(*args, **kwargs)

Base class for a data generating expression having a known type.

Scalar.as_scalar(self)

Inform ibis that the expression should be treated as a scalar.

Scalar.as_table(self) -> 'ir.Table'

Promote the scalar expression to a table.

### Column(*args, **kwargs)

Base class for a data generating expression having a known type.

Column.approx_median(self, where: 'ir.BooleanValue | None' = None) -> 'Scalar'

Return an approximate of the median of `self`.

Column.approx_nunique(self, where: 'ir.BooleanValue | None' = None) -> 'ir.IntegerScalar'

Return the approximate number of distinct elements in `self`.

Column.arbitrary(self, where: 'ir.BooleanValue | None' = None, how: 'Any' = None) -> 'Scalar'

Select an arbitrary value in a column.

Column.argmax(self, key: 'ir.Value', where: 'ir.BooleanValue | None' = None) -> 'Scalar'

Return the value of `self` that maximizes `key`.

Column.argmin(self, key: 'ir.Value', where: 'ir.BooleanValue | None' = None) -> 'Scalar'

Return the value of `self` that minimizes `key`.

Column.as_scalar(self) -> 'Scalar'

Inform ibis that the expression should be treated as a scalar.

Column.as_table(self) -> 'ir.Table'

Promote the expression to a Table.

Column.count(self, where: 'ir.BooleanValue | None' = None) -> 'ir.IntegerScalar'

Compute the number of rows in an expression.

Column.cume_dist(self) -> 'Column'

Return the cumulative distribution over a window.

Column.cummax(self, *, where=None, group_by=None, order_by=None) -> 'Column'

Return the cumulative max over a window.

Column.cummin(self, *, where=None, group_by=None, order_by=None) -> 'Column'

Return the cumulative min over a window.

Column.dense_rank(self) -> 'ir.IntegerColumn'

Position of first element within each group of equal values.

Column.first(self, where: 'ir.BooleanValue | None' = None, order_by: 'Any' = None, include_null: 'bool' = False) -> 'Value'

Return the first value of a column.

Column.lag(self, offset: 'int | ir.IntegerValue | None' = None, default: 'Value | None' = None) -> 'Column'

Return the row located at `offset` rows **before** the current row.

Column.last(self, where: 'ir.BooleanValue | None' = None, order_by: 'Any' = None, include_null: 'bool' = False) -> 'Value'

Return the last value of a column.

Column.lead(self, offset: 'int | ir.IntegerValue | None' = None, default: 'Value | None' = None) -> 'Column'

Return the row located at `offset` rows **after** the current row.

Column.max(self, where: 'ir.BooleanValue | None' = None) -> 'Scalar'

Return the maximum of a column.

Column.median(self, where: 'ir.BooleanValue | None' = None) -> 'Scalar'

Return the median of the column.

Column.min(self, where: 'ir.BooleanValue | None' = None) -> 'Scalar'

Return the minimum of a column.

Column.mode(self, where: 'ir.BooleanValue | None' = None) -> 'Scalar'

Return the mode of a column.

Column.nth(self, n: 'int | ir.IntegerValue') -> 'Column'

Return the `n`th value (0-indexed) over a window.

Column.ntile(self, buckets: 'int | ir.IntegerValue') -> 'ir.IntegerColumn'

Return the integer number of a partitioning of the column values.

Column.nunique(self, where: 'ir.BooleanValue | None' = None) -> 'ir.IntegerScalar'

Compute the number of distinct rows in an expression.

Column.percent_rank(self) -> 'Column'

Return the relative rank of the values in the column.

Column.preview(self, *, max_rows: 'int | None' = None, max_length: 'int | None' = None, max_string: 'int | None' = None, max_depth: 'int | None' = None, console_width: 'int | float | None' = None) -> 'rich.table.Table'

Print a subset as a single-column Rich Table.

Column.quantile(self, quantile: 'float | ir.NumericValue | Sequence[ir.NumericValue | float]', where: 'ir.BooleanValue | None' = None) -> 'Scalar'

Return value at the given quantile.

Column.rank(self) -> 'ir.IntegerColumn'

Compute position of first element within each equal-value group in sorted order.

Column.topk(self, k: 'int', by: 'ir.Value | None' = None, *, name: 'str | None' = None) -> 'ir.Table'

Return a "top k" expression.

Column.value_counts(self, *, name: 'str | None' = None) -> 'ir.Table'

Compute a frequency table.

### NumericColumn(*args, **kwargs)

Base class for a data generating expression having a known type.

NumericColumn.approx_quantile(self, quantile: 'float | ir.NumericValue | Sequence[ir.NumericValue | float]', where: 'ir.BooleanValue | None' = None) -> 'NumericScalar'

Compute one or more approximate quantiles of a column.

NumericColumn.bucket(self, buckets: 'Sequence[int]', closed: "Literal['left', 'right']" = 'left', close_extreme: 'bool' = True, include_under: 'bool' = False, include_over: 'bool' = False) -> 'ir.IntegerColumn'

Compute a discrete binning of a numeric array.

NumericColumn.corr(self, right: 'NumericColumn', where: 'ir.BooleanValue | None' = None, how: "Literal['sample', 'pop']" = 'sample') -> 'NumericScalar'

Return the correlation of two numeric columns.

NumericColumn.cov(self, right: 'NumericColumn', where: 'ir.BooleanValue | None' = None, how: "Literal['sample', 'pop']" = 'sample') -> 'NumericScalar'

Return the covariance of two numeric columns.

NumericColumn.cummean(self, *, where=None, group_by=None, order_by=None) -> 'NumericColumn'

Return the cumulative mean of the input.

NumericColumn.cumsum(self, *, where=None, group_by=None, order_by=None) -> 'NumericColumn'

Return the cumulative sum of the input.

NumericColumn.histogram(self, nbins: 'int | None' = None, binwidth: 'float | None' = None, base: 'float | None' = None, eps: 'float' = 1e-13)

Compute a histogram with fixed width bins.

NumericColumn.mean(self, where: 'ir.BooleanValue | None' = None) -> 'NumericScalar'

Return the mean of a numeric column.

NumericColumn.std(self, where: 'ir.BooleanValue | None' = None, how: "Literal['sample', 'pop']" = 'sample') -> 'NumericScalar'

Return the standard deviation of a numeric column.

NumericColumn.sum(self, where: 'ir.BooleanValue | None' = None) -> 'NumericScalar'

Return the sum of a numeric column.

NumericColumn.var(self, where: 'ir.BooleanValue | None' = None, how: "Literal['sample', 'pop']" = 'sample') -> 'NumericScalar'

Return the variance of a numeric column.

### IntegerColumn(*args, **kwargs)

Base class for a data generating expression having a known type.

IntegerColumn.bit_and(self, where: 'ir.BooleanValue | None' = None) -> 'IntegerScalar'

Aggregate the column using the bitwise and operator.

IntegerColumn.bit_or(self, where: 'ir.BooleanValue | None' = None) -> 'IntegerScalar'

Aggregate the column using the bitwise or operator.

IntegerColumn.bit_xor(self, where: 'ir.BooleanValue | None' = None) -> 'IntegerScalar'

Aggregate the column using the bitwise exclusive or operator.

### FloatingColumn(*args, **kwargs)

Base class for a data generating expression having a known type.

### StringValue(*args, **kwargs)

Base class for a data generating expression having a known type.

StringValue.as_date(self, format_str: 'str') -> 'ir.DateValue'

Parse a string and return a date.

StringValue.as_timestamp(self, format_str: 'str') -> 'ir.TimestampValue'

Parse a string and return a timestamp.

StringValue.ascii_str(self) -> 'ir.IntegerValue'

Return the numeric ASCII code of the first character of a string.

StringValue.authority(self)

Parse a URL and extract authority.

StringValue.capitalize(self) -> 'StringValue'

Uppercase the first letter, lowercase the rest.

StringValue.concat(self, other: 'str | StringValue', *args: 'str | StringValue') -> 'StringValue'

Concatenate strings.

StringValue.contains(self, substr: 'str | StringValue') -> 'ir.BooleanValue'

Return whether the expression contains `substr`.

StringValue.convert_base(self, from_base: 'int | ir.IntegerValue', to_base: 'int | ir.IntegerValue') -> 'ir.IntegerValue'

Convert a string representing an integer from one base to another.

StringValue.endswith(self, end: 'str | StringValue') -> 'ir.BooleanValue'

Determine if `self` ends with `end`.

StringValue.file(self)

Parse a URL and extract file.

StringValue.find(self, substr: 'str | StringValue', start: 'int | ir.IntegerValue | None' = None, end: 'int | ir.IntegerValue | None' = None) -> 'ir.IntegerValue'

Return the position of the first occurrence of substring.

StringValue.find_in_set(self, str_list: 'Sequence[str]') -> 'ir.IntegerValue'

Find the first occurrence of `str_list` within a list of strings.

StringValue.fragment(self)

Parse a URL and extract fragment identifier.

StringValue.hashbytes(self, how: "Literal['md5', 'sha1', 'sha256', 'sha512']" = 'sha256') -> 'ir.BinaryValue'

Compute the binary hash value of the input.

StringValue.hexdigest(self, how: "Literal['md5', 'sha1', 'sha256', 'sha512']" = 'sha256') -> 'ir.StringValue'

Return the hash digest of the input as a hex encoded string.

StringValue.host(self)

Parse a URL and extract host.

StringValue.ilike(self, patterns: 'str | StringValue | Iterable[str | StringValue]') -> 'ir.BooleanValue'

Match `patterns` against `self`, case-insensitive.

StringValue.initcap(self) -> 'StringValue'

Deprecated. Use `capitalize` instead.

StringValue.join(self, strings: 'Sequence[str | StringValue] | ir.ArrayValue') -> 'StringValue'

Join a list of strings using `self` as the separator.

StringValue.left(self, nchars: 'int | ir.IntegerValue') -> 'StringValue'

Return the `nchars` left-most characters.

StringValue.length(self) -> 'ir.IntegerValue'

Compute the length of a string.

StringValue.levenshtein(self, other: 'StringValue') -> 'ir.IntegerValue'

Return the Levenshtein distance between two strings.

StringValue.like(self, patterns: 'str | StringValue | Iterable[str | StringValue]') -> 'ir.BooleanValue'

Match `patterns` against `self`, case-sensitive.

StringValue.lower(self) -> 'StringValue'

Convert string to all lowercase.

StringValue.lpad(self, length: 'int | ir.IntegerValue', pad: 'str | StringValue' = ' ') -> 'StringValue'

Pad `arg` by truncating on the right or padding on the left.

StringValue.lstrip(self) -> 'StringValue'

Remove whitespace from the left side of string.

StringValue.path(self)

Parse a URL and extract path.

StringValue.protocol(self)

Parse a URL and extract protocol.

StringValue.query(self, key: 'str | StringValue | None' = None)

Parse a URL and returns query string or query string parameter.

StringValue.re_extract(self, pattern: 'str | StringValue', index: 'int | ir.IntegerValue') -> 'StringValue'

Return the specified match at `index` from a regex `pattern`.

StringValue.re_replace(self, pattern: 'str | StringValue', replacement: 'str | StringValue') -> 'StringValue'

Replace all matches found by regex `pattern` with `replacement`.

StringValue.re_search(self, pattern: 'str | StringValue') -> 'ir.BooleanValue'

Return whether the values match `pattern`.

StringValue.re_split(self, pattern: 'str | StringValue') -> 'ir.ArrayValue'

Split a string by a regular expression `pattern`.

StringValue.repeat(self, n: 'int | ir.IntegerValue') -> 'StringValue'

Repeat a string `n` times.

StringValue.replace(self, pattern: 'StringValue', replacement: 'StringValue') -> 'StringValue'

Replace each exact match of `pattern` with `replacement`.

StringValue.reverse(self) -> 'StringValue'

Reverse the characters of a string.

StringValue.right(self, nchars: 'int | ir.IntegerValue') -> 'StringValue'

Return up to `nchars` from the end of each string.

StringValue.rlike(self, pattern: 'str | StringValue') -> 'ir.BooleanValue'

Return whether the values match `pattern`.

StringValue.rpad(self, length: 'int | ir.IntegerValue', pad: 'str | StringValue' = ' ') -> 'StringValue'

Pad `self` by truncating or padding on the right.

StringValue.rstrip(self) -> 'StringValue'

Remove whitespace from the right side of string.

StringValue.split(self, delimiter: 'str | StringValue') -> 'ir.ArrayValue'

Split as string on `delimiter`.

StringValue.startswith(self, start: 'str | StringValue') -> 'ir.BooleanValue'

Determine whether `self` starts with `start`.

StringValue.strip(self) -> 'StringValue'

Remove whitespace from left and right sides of a string.

StringValue.substr(self, start: 'int | ir.IntegerValue', length: 'int | ir.IntegerValue | None' = None) -> 'StringValue'

Extract a substring.

StringValue.to_date(self, format_str: 'str') -> 'ir.DateValue'

::: {.callout-warning}
## DEPRECATED: `StringValue.to_date` is deprecated as of v10.0; use as_date() instead
:::

StringValue.to_timestamp(self, format_str: 'str') -> 'ir.TimestampValue'

::: {.callout-warning}
## DEPRECATED: `StringValue.to_timestamp` is deprecated as of v10.0; use as_timestamp() instead
:::

StringValue.translate(self, from_str: 'StringValue', to_str: 'StringValue') -> 'StringValue'

Replace `from_str` characters in `self` characters in `to_str`.

StringValue.upper(self) -> 'StringValue'

Convert string to all uppercase.

StringValue.userinfo(self)

Parse a URL and extract user info.

### TimeValue(*args, **kwargs)

Temporal expressions that have a time component.

TimeValue.add(self, other: 'datetime.timedelta | pd.Timedelta | IntervalValue') -> 'TimeValue'

Add an interval to a time expression.

TimeValue.delta(self, other: 'datetime.time | Value[dt.Time]', part: "Literal['hour', 'minute', 'second', 'millisecond', 'microsecond', 'nanosecond'] | Value[dt.String]") -> 'ir.IntegerValue'

Compute the number of `part`s between two times.

TimeValue.radd(self, other: 'datetime.timedelta | pd.Timedelta | IntervalValue') -> 'TimeValue'

Add an interval to a time expression.

TimeValue.rsub(self: None, other: Value[Union[Interval, Time], DataShape])

Subtract a time or an interval from a time expression.

TimeValue.strftime(self, format_str: 'str') -> 'ir.StringValue'

Format a time according to `format_str`.

TimeValue.sub(self: None, other: Value[Union[Interval, Time], DataShape])

Subtract a time or an interval from a time expression.

TimeValue.truncate(self, unit: "Literal['h', 'm', 's', 'ms', 'us', 'ns']") -> 'TimeValue'

Truncate the expression to a time expression in units of `unit`.

### DateValue(*args, **kwargs)

Base class for a data generating expression having a known type.

DateValue.add(self, other: 'datetime.timedelta | pd.Timedelta | IntervalValue') -> 'DateValue'

Add an interval to a date.

DateValue.delta(self, other: 'datetime.date | Value[dt.Date]', part: "Literal['year', 'quarter', 'month', 'week', 'day'] | Value[dt.String]") -> 'ir.IntegerValue'

Compute the number of `part`s between two dates.

DateValue.epoch_days(self) -> 'ir.IntegerValue'

Return the number of days since the UNIX epoch date.

DateValue.radd(self, other: 'datetime.timedelta | pd.Timedelta | IntervalValue') -> 'DateValue'

Add an interval to a date.

DateValue.rsub(self: None, other: Value[Union[Date, Interval], DataShape])

Subtract a date or an interval from a date.

DateValue.strftime(self, format_str: 'str') -> 'ir.StringValue'

Format a date according to `format_str`.

DateValue.sub(self: None, other: Value[Union[Date, Interval], DataShape])

Subtract a date or an interval from a date.

DateValue.truncate(self, unit: "Literal['Y', 'Q', 'M', 'W', 'D']") -> 'DateValue'

Truncate date expression to units of `unit`.

### DayOfWeek(expr)

A namespace of methods for extracting day of week information.

DayOfWeek.full_name(self)

Get the name of the day of the week.

DayOfWeek.index(self)

Get the index of the day of the week.

### TimestampValue(*args, **kwargs)

Temporal expressions that have a date component.

TimestampValue.add(self, other: 'datetime.timedelta | pd.Timedelta | IntervalValue') -> 'TimestampValue'

Add an interval to a timestamp.

TimestampValue.bucket(self, interval: 'Any' = None, *, years: 'int | None' = None, quarters: 'int | None' = None, months: 'int | None' = None, weeks: 'int | None' = None, days: 'int | None' = None, hours: 'int | None' = None, minutes: 'int | None' = None, seconds: 'int | None' = None, milliseconds: 'int | None' = None, microseconds: 'int | None' = None, nanoseconds: 'int | None' = None, offset: 'Any' = None) -> 'TimestampValue'

Truncate the timestamp to buckets of a specified interval.

TimestampValue.date(self) -> 'DateValue'

Return the date component of the expression.

TimestampValue.delta(self, other: 'datetime.datetime | Value[dt.Timestamp]', part: "Literal['year', 'quarter', 'month', 'week', 'day', 'hour', 'minute', 'second', 'millisecond', 'microsecond', 'nanosecond'] | Value[dt.String]") -> 'ir.IntegerValue'

Compute the number of `part`s between two timestamps.

TimestampValue.radd(self, other: 'datetime.timedelta | pd.Timedelta | IntervalValue') -> 'TimestampValue'

Add an interval to a timestamp.

TimestampValue.rsub(self: None, other: Value[Union[Timestamp, Interval], DataShape])

Subtract a timestamp or an interval from a timestamp.

TimestampValue.strftime(self, format_str: 'str') -> 'ir.StringValue'

Format a timestamp according to `format_str`.

TimestampValue.sub(self: None, other: Value[Union[Timestamp, Interval], DataShape])

Subtract a timestamp or an interval from a timestamp.

TimestampValue.truncate(self, unit: "Literal['Y', 'Q', 'M', 'W', 'D', 'h', 'm', 's', 'ms', 'us', 'ns']") -> 'TimestampValue'

Truncate timestamp expression to units of `unit`.

### IntervalValue(*args, **kwargs)

Base class for a data generating expression having a known type.

IntervalValue.add(self, other: 'datetime.timedelta | pd.Timedelta | IntervalValue') -> 'IntervalValue'

Add this interval to `other`.

IntervalValue.as_unit(self, target_unit: 'str') -> 'IntervalValue'

Convert this interval to units of `target_unit`.

IntervalValue.days (property)

The number of days (IntegerValue).

IntervalValue.floordiv(self, other: 'ir.IntegerValue') -> 'IntervalValue'

Floor-divide this interval by `other`.

IntervalValue.hours (property)

The number of hours (IntegerValue).

IntervalValue.microseconds (property)

The number of microseconds (IntegerValue).

IntervalValue.milliseconds (property)

The number of milliseconds (IntegerValue).

IntervalValue.minutes (property)

The number of minutes (IntegerValue).

IntervalValue.months (property)

The number of months (IntegerValue).

IntervalValue.mul(self, other: 'int | ir.IntegerValue') -> 'IntervalValue'

Multiply this interval by `other`.

IntervalValue.nanoseconds (property)

The number of nanoseconds (IntegerValue).

IntervalValue.negate(self) -> 'ir.IntervalValue'

Negate an interval expression.

IntervalValue.quarters (property)

The number of quarters (IntegerValue).

IntervalValue.radd(self, other: 'datetime.timedelta | pd.Timedelta | IntervalValue') -> 'IntervalValue'

Add this interval to `other`.

IntervalValue.rmul(self, other: 'int | ir.IntegerValue') -> 'IntervalValue'

Multiply this interval by `other`.

IntervalValue.rsub(self, other: 'datetime.timedelta | pd.Timedelta | IntervalValue') -> 'IntervalValue'

Subtract `other` from this interval.

IntervalValue.seconds (property)

The number of seconds (IntegerValue).

IntervalValue.sub(self, other: 'datetime.timedelta | pd.Timedelta | IntervalValue') -> 'IntervalValue'

Subtract `other` from this interval.

IntervalValue.to_unit(self, target_unit: 'str') -> 'IntervalValue'

::: {.callout-warning}
## DEPRECATED: `IntervalValue.to_unit` is deprecated as of v10.0; use as_unit() instead
:::

IntervalValue.weeks (property)

The number of weeks (IntegerValue).

IntervalValue.years (property)

The number of years (IntegerValue).

## Caching

Caching

### ParquetCache(strategy, storage) -> None

Cache expression results as Parquet files, re-hashing when source data changes.

Pairs ``ModificationTimeStrategy`` with ``ParquetStorage``: results are
written as Parquet files on local disk, and the cache key folds in
source-data metadata so the cache invalidates automatically when the
upstream data changes. Build it with :meth:`from_kwargs`.

Parameters
----------
source : ibis.backends.BaseBackend, optional
    Backend used to write the file on a miss and read it back on a hit.
    Defaults to xorq's default backend.
relative_path : Path, optional
    Subdirectory under the cache root. Defaults to
    ``xorq.config.options.cache.default_relative_path`` (``parquet``).
base_path : Path, optional
    Cache root. Defaults to ``None``, which resolves to ``XORQ_CACHE_DIR``.

ParquetCache.storage_typ(source=NOTHING, relative_path=NOTHING, base_path=None) -> None

ParquetCache.strategy_typ(key_prefix=NOTHING) -> None

### ParquetSnapshotCache(strategy, storage) -> None

Cache expression results as Parquet files with a stable, snapshot key.

Unlike :class:`ParquetCache` (which uses ``ModificationTimeStrategy`` and
re-hashes when source data changes), this class pairs ``SnapshotStrategy``
with ``ParquetStorage``: the cache key is computed from the expression
structure only, so source-data changes do not invalidate cached results.
Build it with :meth:`from_kwargs`.

Parameters
----------
source : ibis.backends.BaseBackend, optional
    Backend used to write the file on a miss and read it back on a hit.
    Defaults to xorq's default backend.
relative_path : Path, optional
    Subdirectory under the cache root. Defaults to
    ``xorq.config.options.cache.default_relative_path`` (``parquet``).
base_path : Path, optional
    Cache root. Defaults to ``None``, which resolves to ``XORQ_CACHE_DIR``.

ParquetSnapshotCache.storage_typ(source=NOTHING, relative_path=NOTHING, base_path=None) -> None

ParquetSnapshotCache.strategy_typ(key_prefix=NOTHING) -> None

### SourceCache(strategy, storage) -> None

Cache expression results as a table in a source backend, with automatic invalidation.

Pairs ``ModificationTimeStrategy`` with ``SourceStorage``: the result is
stored as a table in the ``source`` backend, and the cache key folds in
source-data metadata so the cache invalidates automatically when the
upstream data changes. Build it with :meth:`from_kwargs`.

Parameters
----------
source : ibis.backends.BaseBackend, optional
    Backend the cache table lives in. Defaults to xorq's default backend.

SourceCache.storage_typ(source=NOTHING) -> None

SourceCache.strategy_typ(key_prefix=NOTHING) -> None

### SourceSnapshotCache(strategy, storage) -> None

Cache expression results as a table in a source backend, with a stable key.

Pairs ``SnapshotStrategy`` with ``SourceStorage``: the result is stored as
a table in the ``source`` backend, and the cache key is computed from the
expression structure only, so source-data changes do not invalidate cached
results. Build it with :meth:`from_kwargs`.

Parameters
----------
source : ibis.backends.BaseBackend, optional
    Backend the cache table lives in. Defaults to xorq's default backend.

SourceSnapshotCache.storage_typ(source=NOTHING) -> None

SourceSnapshotCache.strategy_typ(key_prefix=NOTHING) -> None

## Window and Selectors

Window functions and column selectors

### window(preceding=None, following=None, order_by=None, group_by=None, *, rows=None, range=None, between=None)

Create a window clause for use with window functions.

The `ROWS` window clause includes peer rows based on differences in row
**number** whereas `RANGE` includes rows based on the differences in row
**value** of a single `order_by` expression.

All window frame bounds are inclusive.

Parameters
----------
preceding
    Number of preceding rows in the window
following
    Number of following rows in the window
group_by
    Grouping key
order_by
    Ordering key
rows
    Whether to use the `ROWS` window clause
range
    Whether to use the `RANGE` window clause
between
    Automatically infer the window kind based on the boundaries

Returns
-------
Window
    A window frame

### selectors (module)

Convenient column selectors.

::: {.callout-tip}
## Check out the [blog post on selectors](../posts/selectors) for examples!
:::

## Rationale

Column selectors are convenience functions for selecting columns that share some property.

## Discussion

For example, a common task is to be able to select all numeric columns for a
subsequent computation.

Without selectors this becomes quite verbose and tedious to write:

>>> import ibis
>>> t = ibis.table(dict(a="int", b="string", c="array<int>", abcd="float"))
>>> expr = t.select([t[c] for c in t.columns if t[c].type().is_numeric()])
>>> expr.columns
['a', 'abcd']

Compare that to the [`numeric`](#ibis.selectors.numeric) selector:

>>> import ibis.selectors as s
>>> expr = t.select(s.numeric())
>>> expr.columns
['a', 'abcd']

When there are multiple properties to check it gets worse:

>>> expr = t.select(
...     [
...         t[c]
...         for c in t.columns
...         if t[c].type().is_numeric() or t[c].type().is_string()
...         if ("a" in c or "b" in c or "cd" in c)
...     ]
... )
>>> expr.columns
['a', 'b', 'abcd']

Using a composition of selectors this is much less tiresome:

>>> expr = t.select((s.numeric() | s.of_type("string")) & s.contains(("a", "b", "cd")))
>>> expr.columns
['a', 'b', 'abcd']

- Across: Opinionated base class for immutable data classes
- All: A column selector
- AllColumns: Cache instances of the class based on instantiation arguments
- Any: A column selector
- Callable
- Cols: A column selector
- ColumnIndex: A column selector
- Concrete: Opinionated base class for immutable data classes
- Contains: A column selector
- Deferred: The user facing wrapper object providing syntactic sugar for deferreds
- EndsWith: A column selector
- Expandable: Base class for many of the ibis core classes, see `AbstractMeta`
- First: Cache instances of the class based on instantiation arguments
- IfAnyAll: Opinionated base class for immutable data classes
- Indexable: Cache instances of the class based on instantiation arguments
- Iterable
- Last: Cache instances of the class based on instantiation arguments
- Mapping: A Mapping is a generic container for associating key/value
- Matches: A column selector
- NoColumns: Cache instances of the class based on instantiation arguments
- OfType: A column selector
- Optional: Optional[X] is equivalent to Union[X, None]
- Resolver: Specification about constructing a value given a context
- Selector: A column selector
- Sequence: All the operations on a read-only sequence
- Singleton: Cache instances of the class based on instantiation arguments
- Slice: Hashable and smaller-scoped slice object versus the builtin one
- StartsWith: A column selector
- Union: Union type; Union[X, Y] means either X or Y
- VarTuple: Built-in immutable sequence
- Where: A column selector
- across: Apply data transformations across multiple columns
- all: Return every column from a table
- all_of: Include columns satisfying all of `predicates`
- annotations
- any_of: Include columns satisfying any of `predicates`
- cols: Select specific column names
- contains: Return columns whose name contains `needles`
- endswith: Select columns whose name ends with one of `suffixes`
- first: Return the first column of a table
- frozendict: dict() -> new empty dictionary
- if_all: Return the **conjunction** of `predicate` applied on all `selector` columns
- if_any: Return the **disjunction** of `predicate` applied on all `selector` columns
- index
- last: Return the last column of a table
- matches: Return columns whose name matches the regular expression `regex`
- none: Return no columns
- numeric: Return numeric columns
- of_type: Select columns of type `dtype`
- public: Add a name or names to __all__
- reduce: reduce(function, iterable[, initial]) -> value
- startswith: Select columns whose name starts with one of `prefixes`
- where: Select columns that satisfy `predicate`

## Machine Learning Operations

Machine Learning Functions and Helpers

### train_test_splits(table: xorq.vendor.ibis.expr.types.relations.Table, test_sizes: Union[Iterable[float], float], unique_key: str | tuple[str] | list[str] | xorq.vendor.ibis.common.selectors.Selector = AllColumns(), num_buckets: int = 10000, random_seed: int | None = None) -> Iterator[xorq.vendor.ibis.expr.types.relations.Table]

Generates multiple train/test splits of an Ibis table for different test sizes.

This function splits an Ibis table into multiple subsets based on a unique key
or combination of keys and a list of test sizes. It uses a hashing function to
convert the unique key into an integer, then applies a modulo operation to split
the data into buckets. Each subset of data is defined by a range of
buckets determined by the cumulative sum of the test sizes.

Parameters
----------
table : ir.Table
    The input Ibis table to be split.
unique_key : str | tuple[str] | list[str] | Selector, optional
    The column name(s) that uniquely identify each row in the table. This
    unique_key is used to create a deterministic split of the dataset
    through a hashing process.
test_sizes : Iterable[float] | float
    An iterable of floats representing the desired proportions for data splits.
    Each value should be between 0 and 1, and their sum must equal 1. The
    order of test sizes determines the order of the generated subsets. If float is passed
    it assumes that the value is for the test size and that a tradition tain test split of (1-test_size, test_size) is returned.
num_buckets : int, optional
    The number of buckets into which the data can be binned after being
    hashed (default is 10000). It controls how finely the data is divided
    during the split process. Adjusting num_buckets can affect the
    granularity and efficiency of the splitting operation, balancing
    between accuracy and computational efficiency.
random_seed : int | None, optional
    Seed for the random number generator. If provided, ensures
    reproducibility of the split (default is None).

Returns
-------
Iterator[ir.Table]
    An iterator yielding Ibis table expressions, each representing a mutually exclusive
    subset of the original table based on the specified test sizes.

Raises
------
ValueError
    If any value in `test_sizes` is not between 0 and 1.
    If `test_sizes` does not sum to 1.
    If `num_buckets` is not an integer greater than 1.

Examples
--------
>>> import xorq.api as xo
>>> table = xo.memtable({"key": range(100), "value": range(100,200)})
>>> unique_key = "key"
>>> test_sizes = [0.2, 0.3, 0.5]
>>> splits = xo.train_test_splits(table, test_sizes, unique_key=unique_key, num_buckets=10, random_seed=42)
>>> for i, split_table in enumerate(splits):
...     print(f"Split {i+1} size: {split_table.count().execute()}")
...     print(split_table.execute())
Split 1 size: 20
Split 2 size: 30
Split 3 size: 50

### Step(typ, name=None, params_tuple=NOTHING) -> None

A single step in a machine learning pipeline that wraps a scikit-learn estimator.

This class represents an individual processing step that can either transform data
(transformers like StandardScaler, SelectKBest) or make predictions (classifiers
like KNeighborsClassifier, LinearSVC). Steps can be combined into Pipeline objects
to create complex ML workflows.

Parameters
----------
typ : type
    The scikit-learn estimator class (must inherit from BaseEstimator).
name : str, optional
    A unique name for this step. If None, generates a name from the class name and ID.
params_tuple : tuple, optional
    Tuple of (parameter_name, parameter_value) pairs for the estimator.
    Parameters are automatically sorted for consistency.

Attributes
----------
typ : type
    The scikit-learn estimator class.
name : str
    The unique name for this step in the pipeline.
params_tuple : tuple
    Sorted tuple of parameter key-value pairs.

Examples
--------
Create a scaler step:

>>> from xorq.ml import Step
>>> from sklearn.preprocessing import StandardScaler
>>> scaler_step = Step(typ=StandardScaler, name="scaler")
>>> scaler_step.instance
StandardScaler()

Create a classifier step with parameters:

>>> from sklearn.neighbors import KNeighborsClassifier
>>> knn_step = Step(
...     typ=KNeighborsClassifier,
...     name="knn",
...     params_tuple=(("n_neighbors", 5), ("weights", "uniform"))
... )
>>> knn_step.instance
KNeighborsClassifier(n_neighbors=5)

Notes
-----
- The Step class is frozen (immutable) using attrs.
- All estimators must inherit from sklearn.base.BaseEstimator.
- Parameter tuples are automatically sorted for hash consistency.
- Steps can be fitted to data using the fit() method which returns a FittedStep.

Step.fit(self, expr, features=None, target=None, cache=None, dest_col=None)

Fit this step to the given expression data.

Parameters
----------
expr : Expr
    The xorq expression containing the training data.
features : tuple of str, optional
    Column names to use as features. If None, infers from expr.columns.
target : str, optional
    Target column name. Required for prediction steps.
cache : Cache, optional
    Storage backend for caching fitted models.
dest_col : str, optional
    Destination column name for transformed output.

Returns
-------
FittedStep
    A fitted step that can transform or predict on new data.

Step.from_fit_predict(cls, fit, predict, return_type, klass_name=None, name=None)

Create a Step from custom fit and predict functions.

Parameters
----------
fit : callable
    Function to fit the model.
predict : callable
    Function to make predictions.
return_type : DataType
    The return type for predictions.
klass_name : str, optional
    Name for the generated estimator class.
name : str, optional
    Name for the step.

Returns
-------
Step
    A new Step with a dynamically created estimator type.

Step.from_fit_transform(cls, fit, transform, return_type, klass_name=None, name=None)

Create a Step from custom fit and transform functions.

Parameters
----------
fit : callable
    Function to fit the model.
transform : callable
    Function to transform with.
return_type : DataType
    The return type for the transformation.
klass_name : str, optional
    Name for the generated estimator class.
name : str, optional
    Name for the step.

Returns
-------
Step
    A new Step with a dynamically created transform type.

Step.from_instance_name(cls, instance, name=None, deep=False)

Create a Step from an existing scikit-learn estimator instance.

Parameters
----------
instance : object
    A scikit-learn estimator instance.
name : str, optional
    Name for the step. If None, generates from instance class name.

Returns
-------
Step
    A new Step wrapping the estimator instance.

Step.from_name_instance(cls, name, instance, deep=False)

Create a Step from a name and estimator instance.

Parameters
----------
name : str
    Name for the step.
instance : object
    A scikit-learn estimator instance.

Returns
-------
Step
    A new Step wrapping the estimator instance.

Step.instance (property)

Create an instance of the estimator with the configured parameters.

Step.set_params(self, **kwargs)

Create a new Step with updated parameters.

Parameters
----------
**kwargs
    Parameter names and values to update.

Returns
-------
Step
    A new Step instance with updated parameters.

Examples
--------
>>> knn_step = Step(typ=KNeighborsClassifier, name="knn")
>>> updated_step = knn_step.set_params(n_neighbors=10, weights="distance")

Step.tag_kwargs (property)

### Pipeline(steps) -> None

A machine learning pipeline that chains multiple processing steps together.

This class provides a xorq-native implementation that wraps scikit-learn pipelines,
enabling deferred execution and integration with xorq expressions. The pipeline
can contain both transform steps (data preprocessing) and a final prediction step.

Parameters
----------
steps : tuple of Step
    Sequence of Step objects that make up the pipeline.

Attributes
----------
steps : tuple of Step
    The sequence of processing steps.
instance : sklearn.pipeline.Pipeline
    The equivalent scikit-learn Pipeline instance.
transform_steps : tuple of Step
    All steps except the final prediction step (if any).
predict_step : Step or None
    The final step if it has a predict method, otherwise None.

Examples
--------
Create a pipeline from scikit-learn estimators:

>>> from xorq.ml import Pipeline
>>> from sklearn.preprocessing import StandardScaler
>>> from sklearn.neighbors import KNeighborsClassifier
>>> import sklearn.pipeline
>>>
>>> sklearn_pipeline = sklearn.pipeline.Pipeline([
...     ("scaler", StandardScaler()),
...     ("knn", KNeighborsClassifier(n_neighbors=5))
... ])
>>> xorq_pipeline = Pipeline.from_instance(sklearn_pipeline)

Fit and predict with xorq expressions:

>>> # Assuming train and test are xorq expressions
>>> fitted = xorq_pipeline.fit(train, features=("feature1", "feature2"), target="target")  # quartodoc: +SKIP
>>> predictions = fitted.predict(test)  # quartodoc: +SKIP

Update pipeline parameters:

>>> updated_pipeline = xorq_pipeline.set_params(knn__n_neighbors=10)  # quartodoc: +SKIP

Notes
-----
- The Pipeline class is frozen (immutable) using attrs.
- Pipelines automatically detect transform vs predict steps based on method availability.
- The fit() method returns a FittedPipeline that can transform and predict on new data.
- Parameter updates use sklearn's parameter naming convention (step__parameter).

Pipeline.fit(self, expr, features=None, target=None, cache=None)

Fit the pipeline to training data.

This method sequentially fits each step in the pipeline, using the output
of each transform step as input to the next step.

Parameters
----------
expr : Expr
    The xorq expression containing training data.
features : tuple of str, optional
    Column names to use as features. If None, infers from expr columns
    excluding the target.
target : str, optional
    Target column name. Required if pipeline has a prediction step.
cache : Cache, optional
    Storage backend for caching fitted models.

Returns
-------
FittedPipeline
    A fitted pipeline that can transform and predict on new data.

Raises
------
ValueError
    If target is not provided but pipeline has a prediction step.

Examples
--------
>>> fitted = pipeline.fit(
...     train_data,
...     features=("sepal_length", "sepal_width"),
...     target="species"
... )  # quartodoc: +SKIP

Pipeline.from_instance(cls, instance, deep=False)

Create a Pipeline from an existing scikit-learn Pipeline.

Parameters
----------
instance : sklearn.pipeline.Pipeline
    A fitted or unfitted scikit-learn pipeline.

Returns
-------
Pipeline
    A new xorq Pipeline wrapping the scikit-learn pipeline.

Examples
--------
>>> import sklearn.pipeline
>>> from sklearn.preprocessing import StandardScaler
>>> from sklearn.svm import SVC
>>>
>>> sklearn_pipe = sklearn.pipeline.Pipeline([
...     ("scaler", StandardScaler()),
...     ("svc", SVC())
... ])
>>> xorq_pipe = Pipeline.from_instance(sklearn_pipe)

Pipeline.instance (property)

Create an equivalent scikit-learn Pipeline instance.

Pipeline.predict_step (property)

Get the final prediction step if it exists.

Pipeline.remap_columns(self, column_map, *, strict=False, **kwargs)

Return a new Pipeline with ColumnTransformer column references remapped.

column_map: dict of {slash-qualified-path: new_col_list},
    e.g. {"preprocessor/num": ["distance", "flight_time"]}.
strict: if True, raise if any pipeline slot path is absent from column_map.
**kwargs: passed to ColumnRemapper (e.g. extra_registry).

Pipeline.remap_params(self, param_map, *, strict=False)

Return a new Pipeline with arbitrary sklearn params remapped.

param_map: dict of {sklearn-double-underscore-path: value},
    e.g. {"classifier__C": 0.01} or {"classifier": Ridge()}.
strict: if True, raise if any param_map key is absent from
    pipeline.get_params(deep=True).

Pipeline.set_params(self, **kwargs)

Pipeline.transform_steps (property)

Get all transformation steps (excluding final prediction step).

### FittedPipeline(fitted_steps, expr, training_hash: str | None = None) -> None

FittedPipeline.decision_function(self, expr, name=None)

FittedPipeline.feature_importances(self, expr, name=None)

FittedPipeline.from_expr(cls, expr)

Recover the outermost FittedPipeline from a tagged expression.

FittedPipeline.from_tag_node(cls, tag_node)

Recover a FittedPipeline from a specific pipeline tag node.

Reads features/target from ALL_STEPS metadata on the tag and finds the
training source by graph structure (innermost step tag's parent).

FittedPipeline.get_tag_kwargs(self, which)

FittedPipeline.invoke_predict_method(self='__no__default__', expr='__no__default__', tag_name='__no__default__', *, methodname='__no__default__', name=None)

FittedPipeline.is_predict (property)

FittedPipeline.pipeline (property)

FittedPipeline.predict(self, expr, name=None)

FittedPipeline.predict_proba(self, expr, name=None)

FittedPipeline.predict_step (property)

FittedPipeline.reemit(self, tag_node, rebuild_subexpr)

Re-emit *tag_node*'s subtree under current code.

Refits the pipeline on the rebuilt training subtree so the
outer tag's ``training_hash`` and step kwargs refresh, rebuilds
catalog subtrees inside the tag's parent (so inner Reads
resolve to the target catalog's files), and re-stamps the outer
pipeline tag on the rebuilt parent with fresh kwargs from the
refitted pipeline.

The internal transform/predict expression structure is
preserved; we do not re-invoke the response method, since the
original predict/transform input is not recoverable from the
tag subtree alone.

FittedPipeline.score(self, X, y, scorer=None, **kwargs)

Compute model score on test data.

Parameters
----------
X : array-like
    Test features
y : array-like
    Test targets
scorer : str or callable, optional
    Scorer name from sklearn.metrics.get_scorer_names() or a callable metric function.
    If None, uses model's default (accuracy for classifiers, r2 for regressors)
**kwargs : dict
    Additional arguments passed to the scorer function

Returns
-------
float
    The computed score

FittedPipeline.score_expr(self, expr, scorer=None, **kwargs)

Compute metrics using deferred execution.

Parameters
----------
expr : ibis.Expr
    Expression containing test data
scorer : str, callable, _BaseScorer, Scorer, or None
    Scorer specification. If None, uses model's default.
    Automatically detects whether scorer needs predict, predict_proba,
    or decision_function.
**kwargs : dict
    Additional arguments passed to the metric function

Returns
-------
ibis.Expr
    Deferred metric expression

FittedPipeline.transform(self, expr, tag=True)

FittedPipeline.transform_steps (property)

### deferred_fit_predict(expr='__no__default__', target='__no__default__', features='__no__default__', cls='__no__default__', return_type='__no__default__', params=(), name_infix=<ResponseMethod.PREDICT: 'predict'>, cache=None)

### deferred_fit_transform(expr='__no__default__', features='__no__default__', fit='__no__default__', other='__no__default__', return_type='__no__default__', target=None, name_infix='transform', cache=None)

### calc_split_column(table: xorq.vendor.ibis.expr.types.relations.Table, unique_key: str | tuple[str] | list[str] | xorq.vendor.ibis.common.selectors.Selector, test_sizes: Iterable[float], num_buckets: int = 10000, random_seed: int | None = None, name: str = 'split') -> xorq.vendor.ibis.expr.types.numeric.IntegerColumn

Parameters
----------
table : ir.Table
    The input Ibis table to be split.
unique_key : str | tuple[str] | list[str] | Selector
    The column name(s) that uniquely identify each row in the table. This
    unique_key is used to create a deterministic split of the dataset
    through a hashing process.
test_sizes : Iterable[float]
    An iterable of floats representing the desired proportions for data splits.
    Each value should be between 0 and 1, and their sum must equal 1. The
    order of test sizes determines the order of the generated subsets. If float is passed
    it assumes that the value is for the test size and that a tradition tain test split of (1-test_size, test_size) is returned.
num_buckets : int, optional
    The number of buckets into which the data can be binned after being
    hashed (default is 10000). It controls how finely the data is divided
    during the split process. Adjusting num_buckets can affect the
    granularity and efficiency of the splitting operation, balancing
    between accuracy and computational efficiency.
random_seed : int | None, optional
    Seed for the random number generator. If provided, ensures
    reproducibility of the split (default is None).
name : str, optional
    Name for the returned IntegerColumn (default is "split").

Returns
-------
ibis.IntergerColumn
    A column with split indices representing mutually exclusive subsets of the original table based on the specified test sizes.

Raises
------
ValueError
    If any value in `test_sizes` is not between 0 and 1.
    If `test_sizes` does not sum to 1.
    If `num_buckets` is not an integer greater than 1.

Examples
--------
>>> import xorq.api as xo
>>> unique_key = "key"
>>> table = xo.memtable({unique_key: range(100), "value": range(100, 200)})
>>> test_sizes = [0.2, 0.3, 0.5]
>>> col = xo.expr.ml.calc_split_column(table, unique_key, test_sizes, num_buckets=10, random_seed=42, name="my-split")

## Lineage

Data lineage tracking utilities

### build_column_trees(expr: 'Any') -> 'dict[str, GenericNode]'

Builds a lineage tree for each column in the expression.

### build_tree(node: 'GenericNode', *, dedup: 'bool' = True, max_depth: 'int | None' = None) -> 'TextTree'

## Flight Operations

Apache Arrow Flight server and client operations

### FlightServer(flight_url=None, tls_certificates=(), verify_client=False, root_certificates=None, auth: xorq.flight.BasicAuth = None, make_connection=<function connect>, exchangers=())

FlightServer.auth_kwargs (property)

FlightServer.client (property)

FlightServer.close(self, *args)

FlightServer.from_udxf(cls, expr, host=None, port=None, make_connection=None, **kwargs)

FlightServer.serve(self, block=False)

FlightServer.wait(self)

### FlightUrl(scheme: str = 'grpc', host: str = 'localhost', username: Optional[str] = None, password: Optional[str] = None, port: Optional[int] = None, path: Optional[str] = '', query: Optional[str] = '', fragment: Optional[str] = '') -> None

FlightUrl.bind_socket(self)

FlightUrl.client_kwargs (property)

FlightUrl.find_and_bind_socket(self)

FlightUrl.port_in_use(port, host='localhost')

FlightUrl.to_location(self)

FlightUrl.unbind_socket(self)

### make_udxf(process_df, maybe_schema_in, maybe_schema_out, name=None, description=None, command=None, do_wraps=True)

## Catalog Operations

Compute catalog management

### Catalog(backend) -> None

A git-backed registry for versioned build artifacts.

A catalog is a git repository containing serialized xorq expressions
as content-addressed zip archives.  When backed by git-annex, cloning
downloads only metadata and artifact content is fetched on demand.
A plain-git backend stores archives as regular blobs.

Construct via the classmethods ``from_name``, ``from_repo_path``,
``from_default``, ``clone_from``, or the dispatch helper ``from_kwargs``.

Catalog.add(self, obj: 'Expr | Path', sync: 'bool' = True, aliases: 'tuple[str, ...]' = (), exist_ok: 'bool' = False, project_path: 'Path | None' = None, relocate_reads: 'bool | None' = None) -> 'CatalogEntry'

Add a build to the catalog.

*obj* may be a ``Path`` to a zip archive, a ``Path`` to a build
directory, or an xorq ``Expr``.  Returns the created ``CatalogEntry``.

*project_path* is the directory containing the ``pyproject.toml`` used
to build the wheel and requirements sidecars.  If omitted, the packager
walks upward from the current working directory to find one.  Passing
it explicitly is required when the caller's cwd is not inside the
project (e.g. Jupyter kernels started from ``/tmp``).  Ignored for zip
inputs, which are already complete build archives.

*relocate_reads* controls how the build is produced and so only applies
to an ``Expr`` input; ``Path`` inputs are already-built artifacts whose
reads were settled at build time. It defaults (``None``) to ``True`` for
an ``Expr`` -- matching the CLI: local-file reads are bundled so the entry
is self-contained and resolves on both the load and fuse/bind paths
(#2133) -- and is a no-op for a ``Path``. Passing ``relocate_reads=True``
explicitly with a ``Path`` is a misuse and raises.

Catalog.add_alias(self, name, alias, sync=True)

Create an alias pointing at entry *name*.  Overwrites if the alias already exists.

Catalog.add_as_submodule(self, root_repo)

Catalog.assert_consistency(self)

Verify that catalog.yaml, entries, metadata, and aliases are all in agreement.

Catalog.bind(self, source_entry, *transforms, con=None)

Bind a source entry through one or more transform entries.

Catalog.catalog_aliases (property)

Catalog.catalog_entries (property)

Catalog.catalog_yaml (property)

Catalog.clone_from(cls, url, repo_path=None, check_consistency=True, annex=None, content_store_config: 'ContentStoreConfig | None' = None, git_config=None, **remote_kwargs)

Clone a catalog repo and detect the backend automatically.

*content_store_config* and *annex* are mutually exclusive.
If the cloned repo contains a ``content_store.yaml``, the pointer
backend is used.  Otherwise *annex* controls the backend:

- ``None`` (default) — auto-detect.  If the cloned repo has a
  ``git-annex`` branch, git-annex is initialised and the remote
  is enabled when credentials are available (embedded, env vars,
  or *remote_kwargs*).  Otherwise falls back to plain git.
- ``False`` — force plain git, even if the repo has a
  ``git-annex`` branch.
- Any ``AnnexConfig`` instance — git-annex is initialised and
  the remote is enabled if remote.log has a special remote
  configured.

Content is **not** fetched eagerly; it is retrieved on demand
when ``entry.expr`` is accessed (via ``fetch_content``).
For S3 remotes without embedded credentials, the caller can
supply credentials via *remote_kwargs* or environment variables
(``XORQ_CATALOG_S3_*`` / ``XORQ_CONTENT_STORE_S3_*``).

Catalog.clone_from_as_submodule(cls, root_repo, url, check_consistency=True, annex=None, content_store_config: 'ContentStoreConfig | None' = None)

Catalog.commit_context(self, message)

Catalog.contains(self, name)

Return True if an entry with *name* exists in the catalog.

Catalog.embed_readonly(self, readonly_config)

Embed read-only credentials into the git-annex branch.

Verifies that *readonly_config* cannot write to the bucket, then
sets ``embedcreds=yes`` and writes the config to remote.log.

Raises ``ValueError`` if the credentials have write access.

Catalog.fetch(self)

Fetch from the configured git remote (no-op if no remote is configured).

Catalog.fetch_entries(self, *entries)

Fetch annex content for the given entries in a single operation.

Each element can be a ``CatalogEntry`` or a string (entry name).
No-op for plain-git backends.

Catalog.from_default(cls, init=None, check_consistency=True, annex=None, content_store_config: 'ContentStoreConfig | None' = None, **remote_kwargs)

Catalog.from_kwargs(cls, name=None, path=None, url=None, root_repo=None, init=None, check_consistency=True, annex=None, content_store_config: 'ContentStoreConfig | None' = None)

Catalog.from_name(cls, name, init=None, check_consistency=True, annex=None, content_store_config: 'ContentStoreConfig | None' = None, **remote_kwargs)

Catalog.from_name_as_submodule(cls, root_repo, name, init=None, check_consistency=True, annex=None, content_store_config: 'ContentStoreConfig | None' = None)

Catalog.from_repo_path(cls, repo_path, init=None, check_consistency=True, annex=None, content_store_config: 'ContentStoreConfig | None' = None, **remote_kwargs)

Open or initialise a catalog; *content_store_config* and *annex* are mutually exclusive.

Catalog.get_catalog_entry(self, name, maybe_alias: 'bool' = False)

Look up a ``CatalogEntry`` by name.  Raises if not found.

Catalog.get_zip(self, name, dir_path=None)

Export an entry's archive to *dir_path* (default: cwd).  Returns the output path.

Catalog.init_repo_path(repo_path: 'str | Path', bare: 'bool' = False, annex: 'AnnexConfig | None' = None, content_store_config: 'ContentStoreConfig | None' = None) -> 'Repo'

Catalog.list(self)

Return the list of entry names in the catalog.

Catalog.list_aliases(self)

Return the list of alias names in the catalog.

Catalog.load(self, name_or_alias, con=None)

Return a tagged RemoteTable expression for a catalog entry (by hash or alias).

Catalog.maybe_synchronizing(self, sync)

Catalog.name_to_repo_path(cls, name)

Catalog.pull(self)

Fetch and merge from the catalog's git remote; raise on unmerged paths.

Replaces ``git pull`` (which inherits the user's ``pull.rebase``
config and bails on divergent branches by default) with explicit
``git fetch`` + ``git merge``.  When the merge leaves
``catalog.yaml`` conflicted (typical when both sides appended
to the entries or aliases lists), a Python 3-way list-merge
resolves it: items present in the merge base and removed by
one side are propagated as removals; items added by either
side survive; duplicates are collapsed.  Anything still
unmerged after that — typically alias symlinks at the same
path with diverging targets — surfaces as
``CatalogMergeConflict`` with the conflicted paths and the
remote name; the merge is left in-progress so the user can
resolve it (see ``CatalogMergeConflict`` for recovery recipes).

Pre-flights:

- HEAD must be on a branch (the catalog API never detaches HEAD
  on its own — this only fails if the repo was put in detached
  state outside xorq).  Raises ``CatalogPullError``.
- ``catalog.yaml`` in both ours (HEAD) and the remote tip must
  exist, parse, and have the expected dict-or-list shape.  The
  resolver assumes well-formed input on both sides; without
  this check, a ``catalog.yaml`` deleted on the remote tip
  would be silently treated as "theirs removed every entry"
  and the 3-way list merge would drop every prior entry, while
  a malformed or scalar-shaped yaml would leak a bare
  ``ValueError`` / ``AttributeError`` from inside the
  resolver.  Raises ``CatalogPullError`` naming the corrupt
  side.
- A non-conflict ``git merge`` failure (e.g. the remote ref
  doesn't exist, the working tree is dirty, a hook rejected the
  merge commit) re-raises the original ``GitCommandError``
  rather than swallowing it and falling through to a misleading
  ``git commit --no-edit``.

A catalog has at most one git remote (see ADR on single-remote
catalogs).  No remote → no-op.

Catalog.push(self)

Push to the configured git remote after verifying consistency.

Pushes ``main``, then ``git-annex`` (if present). Both pushes are
always attempted — raises a single ``CatalogPushError`` listing
every rejection or transport failure across both. No-op when no
git remote is configured.

Returns ``()``, ``(main_result,)``, or ``(main_result, annex_result)``.

Catalog.remote_config (property)

The resolved remote config, or None.

Catalog.remove(self, name, sync=True)

Remove an entry (and its aliases) from the catalog by name.

Catalog.remove_alias(self, alias: 'str', sync: 'bool' = True) -> "'CatalogAlias'"

Remove *alias*, leaving its target entry untouched.

Symmetric with ``add_alias``/``remove``: git sync is managed via
*sync*.  Raises ``ValueError`` if *alias* is not registered.

Catalog.repo (property)

Catalog.repo_path (property)

Catalog.set_remote(self, name, url, force=False)

Configure the catalog's git remote.

The catalog supports at most one git remote (ADR-0011). When the
repo has no git remote, ``set_remote`` creates one with the given
*name* and *url* and returns it.

When a git remote is already configured, ``set_remote`` raises
``CatalogConfigurationError`` unless ``force=True`` is passed. The
guard exists because silent replacement turns a typo in the
remote name into the deletion of the existing remote with no
signal — failing by default forces explicit opt-in. With
``force=True``, every existing git remote is deleted and replaced.

Catalog.set_remote_config(self, remote_config)

Update the git-annex special remote configuration.

Calls ``enableremote`` to write the config to remote.log on the
git-annex branch.  Use ``catalog.remote_config`` to read it back.

Catalog.sync(self)

Pull then push — shorthand for a full round-trip synchronization.

Catalog.synchronizing(self)

## Type System

Data types and schemas

### Data types

Scalar and column data types

#### dtype(value: 'IntoDtype', nullable: 'bool' = True) -> 'DataType'

Create a DataType object.

#### DataType(nullable: bool = True)

Base class for all data types.

DataType.cast(self, other, **kwargs)

DataType.castable(self, to, **kwargs) -> 'bool'

Check whether this type is castable to another.

DataType.column (property)

DataType.equals(self, other)

DataType.from_numpy(cls, numpy_type, nullable=True) -> 'Self'

Return the equivalent ibis datatype.

DataType.from_pandas(cls, pandas_type, nullable=True) -> 'Self'

Return the equivalent ibis datatype.

DataType.from_polars(cls, polars_type, nullable=True) -> 'Self'

Return the equivalent ibis datatype.

DataType.from_pyarrow(cls, arrow_type, nullable=True) -> 'Self'

Return the equivalent ibis datatype.

DataType.from_string(cls, value) -> 'Self'

DataType.from_typehint(cls, typ, nullable=True) -> 'Self'

DataType.is_array(self) -> 'bool'

Return True if an instance of an Array type.

DataType.is_binary(self) -> 'bool'

Return True if an instance of a Binary type.

DataType.is_boolean(self) -> 'bool'

Return True if an instance of a Boolean type.

DataType.is_date(self) -> 'bool'

Return True if an instance of a Date type.

DataType.is_decimal(self) -> 'bool'

Return True if an instance of a Decimal type.

DataType.is_enum(self) -> 'bool'

Return True if an instance of an Enum type.

DataType.is_float16(self) -> 'bool'

Return True if an instance of a Float16 type.

DataType.is_float32(self) -> 'bool'

Return True if an instance of a Float32 type.

DataType.is_float64(self) -> 'bool'

Return True if an instance of a Float64 type.

DataType.is_floating(self) -> 'bool'

Return True if an instance of any Floating type.

DataType.is_geospatial(self) -> 'bool'

Return True if an instance of a Geospatial type.

DataType.is_inet(self) -> 'bool'

Return True if an instance of an Inet type.

DataType.is_int16(self) -> 'bool'

Return True if an instance of an Int16 type.

DataType.is_int32(self) -> 'bool'

Return True if an instance of an Int32 type.

DataType.is_int64(self) -> 'bool'

Return True if an instance of an Int64 type.

DataType.is_int8(self) -> 'bool'

Return True if an instance of an Int8 type.

DataType.is_integer(self) -> 'bool'

Return True if an instance of any Integer type.

DataType.is_interval(self) -> 'bool'

Return True if an instance of an Interval type.

DataType.is_json(self) -> 'bool'

Return True if an instance of a JSON type.

DataType.is_linestring(self) -> 'bool'

Return True if an instance of a LineString type.

DataType.is_macaddr(self) -> 'bool'

Return True if an instance of a MACADDR type.

DataType.is_map(self) -> 'bool'

Return True if an instance of a Map type.

DataType.is_multilinestring(self) -> 'bool'

Return True if an instance of a MultiLineString type.

DataType.is_multipoint(self) -> 'bool'

Return True if an instance of a MultiPoint type.

DataType.is_multipolygon(self) -> 'bool'

Return True if an instance of a MultiPolygon type.

DataType.is_nested(self) -> 'bool'

Return true if an instance of any nested (Array/Map/Struct) type.

DataType.is_null(self) -> 'bool'

Return true if an instance of a Null type.

DataType.is_numeric(self) -> 'bool'

Return true if an instance of a Numeric type.

DataType.is_point(self) -> 'bool'

Return true if an instance of a Point type.

DataType.is_polygon(self) -> 'bool'

Return true if an instance of a Polygon type.

DataType.is_primitive(self) -> 'bool'

Return true if an instance of a Primitive type.

DataType.is_signed_integer(self) -> 'bool'

Return true if an instance of a SignedInteger type.

DataType.is_string(self) -> 'bool'

Return true if an instance of a String type.

DataType.is_struct(self) -> 'bool'

Return true if an instance of a Struct type.

DataType.is_temporal(self) -> 'bool'

Return true if an instance of a Temporal type.

DataType.is_time(self) -> 'bool'

Return true if an instance of a Time type.

DataType.is_timestamp(self) -> 'bool'

Return true if an instance of a Timestamp type.

DataType.is_uint16(self) -> 'bool'

Return true if an instance of a UInt16 type.

DataType.is_uint32(self) -> 'bool'

Return true if an instance of a UInt32 type.

DataType.is_uint64(self) -> 'bool'

Return true if an instance of a UInt64 type.

DataType.is_uint8(self) -> 'bool'

Return true if an instance of a UInt8 type.

DataType.is_unknown(self) -> 'bool'

Return true if an instance of an Unknown type.

DataType.is_unsigned_integer(self) -> 'bool'

Return true if an instance of an UnsignedInteger type.

DataType.is_uuid(self) -> 'bool'

Return true if an instance of a UUID type.

DataType.is_variadic(self) -> 'bool'

Return true if an instance of a Variadic type.

DataType.name (property)

Return the name of the data type.

DataType.scalar (property)

DataType.to_numpy(self)

Return the equivalent numpy datatype.

DataType.to_pandas(self)

Return the equivalent pandas datatype.

DataType.to_polars(self)

Return the equivalent polars datatype.

DataType.to_pyarrow(self)

Return the equivalent pyarrow datatype.

#### Array(value_type: DataType, nullable: bool = True)

Array values.

#### Binary(nullable: bool = True)

A type representing a sequence of bytes.

#### Boolean(nullable: bool = True)

[](`True`) or [](`False`) values.

#### Date(nullable: bool = True)

Date values.

#### Decimal(precision: Optional[int] = None, scale: Optional[int] = None, nullable: bool = True)

Fixed-precision decimal values.

#### Float16(nullable: bool = True)

16-bit floating point numbers.

#### Float32(nullable: bool = True)

32-bit floating point numbers.

#### Float64(nullable: bool = True)

64-bit floating point numbers.

#### INET(nullable: bool = True)

IP addresses.

#### Int16(nullable: bool = True)

Signed 16-bit integers.

#### Int32(nullable: bool = True)

Signed 32-bit integers.

#### Int64(nullable: bool = True)

Signed 64-bit integers.

#### Int8(nullable: bool = True)

Signed 8-bit integers.

#### Interval(unit: IntervalUnit, nullable: bool = True)

Interval values.

Interval.resolution (property)

The interval unit's name.

#### JSON(binary: bool = False, nullable: bool = True)

JSON values.

#### LineString(geotype: Literal['geography', 'geometry'] = 'geometry', srid: Optional[int] = None, nullable: bool = True)

A sequence of 2 or more points.

#### MACADDR(nullable: bool = True)

Media Access Control (MAC) address of a network interface.

#### Map(key_type: DataType, value_type: DataType, nullable: bool = True)

Associative array values.

#### MultiLineString(geotype: Literal['geography', 'geometry'] = 'geometry', srid: Optional[int] = None, nullable: bool = True)

A set of one or more line strings.

#### MultiPoint(geotype: Literal['geography', 'geometry'] = 'geometry', srid: Optional[int] = None, nullable: bool = True)

A set of one or more points.

#### MultiPolygon(geotype: Literal['geography', 'geometry'] = 'geometry', srid: Optional[int] = None, nullable: bool = True)

A set of one or more polygons.

#### Null(nullable: bool = True)

Null values.

#### Point(geotype: Literal['geography', 'geometry'] = 'geometry', srid: Optional[int] = None, nullable: bool = True)

A point described by two coordinates.

#### Polygon(geotype: Literal['geography', 'geometry'] = 'geometry', srid: Optional[int] = None, nullable: bool = True)

A set of one or more closed line strings.

#### String(nullable: bool = True)

A type representing a string.

#### Struct(fields: FrozenOrderedDict[str, DataType], nullable: bool = True)

Structured values.

Struct.from_tuples(cls, pairs: 'Iterable[tuple[str, str | DataType]]', nullable: 'bool' = True) -> 'Struct'

Construct a `Struct` type from pairs.

#### Time(nullable: bool = True)

Time values.

#### Timestamp(timezone: Optional[str] = None, scale: Optional[Literal[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]] = None, nullable: bool = True)

Timestamp values.

Timestamp.from_unit(cls, unit, timezone=None, nullable=True)

Return a timestamp type with the given unit and timezone.

Timestamp.unit (property)

Return the unit of the timestamp.

#### UInt16(nullable: bool = True)

Unsigned 16-bit integers.

#### UInt32(nullable: bool = True)

Unsigned 32-bit integers.

#### UInt64(nullable: bool = True)

Unsigned 64-bit integers.

#### UInt8(nullable: bool = True)

Unsigned 8-bit integers.

#### UUID(nullable: bool = True)

A 128-bit number used to identify information in computer systems.

### Schemas

Table Schemas

#### schema(pairs: 'SchemaLike | None' = None, names: 'Iterable[str] | None' = None, types: 'Iterable[str | dt.DataType] | None' = None) -> 'sch.Schema'

Validate and return a [`Schema`](./schemas.qmd#ibis.expr.schema.Schema) object.

Parameters
----------
pairs
    List or dictionary of name, type pairs. Mutually exclusive with `names`
    and `types` arguments.
names
    Field names. Mutually exclusive with `pairs`.
types
    Field types. Mutually exclusive with `pairs`.

Returns
-------
Schema
    An ibis schema

Examples
--------
>>> from xorq.vendor.ibis import schema, Schema
>>> sc = schema([("foo", "string"), ("bar", "int64"), ("baz", "boolean")])
>>> sc = schema(names=["foo", "bar", "baz"], types=["string", "int64", "boolean"])
>>> sc = schema(dict(foo="string"))
>>> sc = schema(Schema(dict(foo="string")))  # no-op

#### Schema(fields: FrozenOrderedDict[str, DataType])

An ordered mapping of str -> `DataType`, used to hold a Table's schema.

Schema.as_struct(self) -> 'dt.Struct'

Schema.equals(self, other: 'Schema') -> 'bool'

Return whether `other` is equal to `self`.

Schema.from_numpy(cls, numpy_schema)

Return the equivalent ibis schema.

Schema.from_pandas(cls, pandas_schema)

Return the equivalent ibis schema.

Schema.from_polars(cls, polars_schema)

Return the equivalent ibis schema.

Schema.from_pyarrow(cls, pyarrow_schema)

Return the equivalent ibis schema.

Schema.from_tuples(cls, values: 'Iterable[tuple[str, str | dt.DataType]]') -> 'Schema'

Construct a `Schema` from an iterable of pairs.

Schema.name_at_position(self, i: 'int') -> 'str'

Return the name of a schema column at position `i`.

Schema.to_numpy(self)

Return the equivalent numpy dtypes.

Schema.to_pandas(self)

Return the equivalent pandas datatypes.

Schema.to_polars(self)

Return the equivalent polars schema.

Schema.to_pyarrow(self)

Return the equivalent pyarrow schema.

Schema.to_sqlglot(self, dialect: 'str | sg.Dialect') -> 'list[sge.ColumnDef]'

Convert the schema to a list of SQL column definitions.

## UDF System

The functions for creating UDF

### make_pandas_udf(fn='__no__default__', schema='__no__default__', return_type='__no__default__', database=None, catalog=None, name=None, **kwargs)

Create a scalar User-Defined Function (UDF) that operates on pandas DataFrames.

This function creates a scalar UDF that processes data row-by-row, converting
PyArrow arrays to pandas DataFrames for processing. It's ideal for operations
that benefit from pandas' rich functionality and are easier to express with
DataFrame operations.

Parameters
----------
fn : callable
    The function to be executed. Should accept a pandas DataFrame and return
    a pandas Series or scalar value.
schema : Schema
    The input schema defining column names and their data types.
return_type : DataType
    The return data type of the UDF.
database : str, optional
    Database name for the UDF namespace.
catalog : str, optional
    Catalog name for the UDF namespace.
name : str, optional
    Name of the UDF. If None, generates a name from the function.
**kwargs
    Additional configuration parameters (e.g., volatility settings).

Returns
-------
callable
    A UDF constructor that can be used in expressions with `.on_expr()` method.

Examples
--------
Creating a UDF that calculates penguin bill ratio:

>>> import pandas as pd
>>> from xorq.expr.udf import make_pandas_udf
>>> import xorq.expr.datatypes as dt
>>> import xorq.api as xo
>>>
>>> # Load penguins dataset
>>> penguins = xo.examples.penguins.fetch(backend=xo.connect())
>>>
>>> # Define the function
>>> def bill_ratio(df):
...     return df['bill_length_mm'] / df['bill_depth_mm']
>>>
>>> # Create UDF
>>> schema = penguins.select(['bill_length_mm', 'bill_depth_mm']).schema()
>>> bill_ratio_udf = make_pandas_udf(
...     fn=bill_ratio,
...     schema=schema,
...     return_type=dt.float64,
...     name="bill_ratio"
>>> )
>>>
>>> # Apply to table
>>> result = penguins.mutate(
...     bill_ratio=bill_ratio_udf.on_expr(penguins)
>>> ).execute()

Creating a UDF for penguin size classification:

>>> def classify_penguin_size(df):
...     def size_category(row):
...         mass = row['body_mass_g']
...         flipper = row['flipper_length_mm']
...
...         if pd.isna(mass) or pd.isna(flipper):
...             return 'Unknown'
...
...         # Simple size classification based on body mass and flipper length
...         if mass > 4500 and flipper > 210:
...             return 'Large'
...         elif mass < 3500 and flipper < 190:
...             return 'Small'
...         else:
...             return 'Medium'
...
...     return df.apply(size_category, axis=1)
>>>
>>> size_schema = penguins.select(['body_mass_g', 'flipper_length_mm']).schema()
>>> size_udf = make_pandas_udf(
...     fn=classify_penguin_size,
...     schema=size_schema,
...     return_type=dt.string,
...     name="classify_size"
>>> )
>>>
>>> # Apply size classification
>>> result = penguins.mutate(
...     size_category=size_udf.on_expr(penguins)
>>> ).execute()

Creating a UDF for complex penguin feature engineering:

>>> def penguin_features(df):
...     # Create multiple derived features
...     features = pd.DataFrame(index=df.index)
...
...     # Bill area
...     features['bill_area'] = df['bill_length_mm'] * df['bill_depth_mm']
...
...     # Body condition index
...     features['body_condition'] = df['body_mass_g'] / (df['flipper_length_mm'] ** 2)
...
...     # Aspect ratio of bill
...     features['bill_aspect_ratio'] = df['bill_length_mm'] / df['bill_depth_mm']
...
...     # Return as concatenated string for this example
...     return features.apply(lambda row: f"area:{row['bill_area']:.1f}_bci:{row['body_condition']:.4f}_ratio:{row['bill_aspect_ratio']:.2f}", axis=1)
>>>
>>> all_measurements = ['bill_length_mm', 'bill_depth_mm', 'flipper_length_mm', 'body_mass_g']
>>> features_schema = penguins.select(all_measurements).schema()
>>> features_udf = make_pandas_udf(
...     fn=penguin_features,
...     schema=features_schema,
...     return_type=dt.string,
...     name="penguin_features"
>>> )
>>>
>>> # Apply feature engineering
>>> result = penguins.mutate(
...     derived_features=features_udf.on_expr(penguins)
>>> ).execute()

Notes
-----
- The function receives a pandas DataFrame where columns correspond to the schema keys
- The function should return a pandas Series or scalar value compatible with return_type
- PyArrow arrays are automatically converted to pandas and back for seamless integration
- Use this when you need pandas-specific functionality like string operations,
  datetime handling, or complex data manipulations

See Also
--------
scalar : For PyArrow-based scalar UDFs with potentially better performance
make_pandas_expr_udf : For UDFs that need pre-computed values
agg : For aggregation functions

### make_pandas_expr_udf(computed_kwargs_expr='__no__default__', fn='__no__default__', schema='__no__default__', return_type=Binary(nullable=True), database=None, catalog=None, name=None, *, post_process_fn=<built-in function loads>, **kwargs)

Create an expression-based scalar UDF that incorporates pre-computed values.

This function creates a special type of scalar UDF that can access pre-computed
values (like trained machine learning models) during execution. The pre-computed
value is generated from a separate expression and passed to the UDF function,
enabling complex workflows like model training and inference within the same
query pipeline.

Parameters
----------
computed_kwargs_expr : Expression
    An expression that computes a value to be passed to the UDF function.
    This is typically an aggregation that produces a model or other computed value.
fn : callable
    The function to be executed. Should accept (computed_arg, df, **kwargs)
    where computed_arg is the result of computed_kwargs_expr and df is a
    pandas DataFrame containing the input columns.
schema : Schema
    The input schema defining column names and their data types.
return_type : DataType, default dt.binary
    The return data type of the UDF.
database : str, optional
    Database name for the UDF namespace.
catalog : str, optional
    Catalog name for the UDF namespace.
name : str, optional
    Name of the UDF. If None, uses the function name.
post_process_fn : callable, default unwrap_model
    Function to post-process the computed_kwargs_expr result before passing
    to the main function.
**kwargs
    Additional configuration parameters.

Returns
-------
callable
    A UDF constructor that can be used in expressions.

Examples
--------
Machine learning workflow with penguin species classification:

>>> import pickle
>>> import pandas as pd
>>> from sklearn.neighbors import KNeighborsClassifier
>>> from xorq.expr.udf import make_pandas_expr_udf, agg
>>> import xorq.expr.datatypes as dt
>>> import xorq.api as xo
>>>
>>> # Load penguins dataset
>>> penguins = xo.examples.penguins.fetch(backend=xo.connect())
>>> features = ['bill_length_mm', 'bill_depth_mm', 'flipper_length_mm', 'body_mass_g']
>>>
>>> # Split data
>>> train_data = penguins.filter(penguins.year < 2009)
>>> test_data = penguins.filter(penguins.year >= 2009)
>>>
>>> # Define training function
>>> def train_penguin_model(df):
...     df_clean = df.dropna(subset=features + ['species'])
...     X = df_clean[features]
...     y = df_clean['species']
...
...     model = KNeighborsClassifier(n_neighbors=5)
...     model.fit(X, y)
...     return pickle.dumps(model)
>>>
>>> # Define prediction function
>>> def predict_penguin_species(model, df):
...     df_clean = df.dropna(subset=features)
...     X = df_clean[features]
...     predictions = model.predict(X)
...     # Return predictions for all rows (fill NaN for missing data)
...     result = pd.Series(index=df.index, dtype='object')
...     result.loc[df_clean.index] = predictions
...     return result.fillna('Unknown')
>>>
>>> # Create schemas for training and prediction
>>> train_schema = train_data.select(features + ['species']).schema()
>>> test_schema = test_data.select(features).schema()
>>>
>>> # Create model training UDAF
>>> model_udaf = agg.pandas_df(
...     fn=train_penguin_model,
...     schema=train_schema,
...     return_type=dt.binary,
...     name="train_penguin_model"
>>> )
>>>
>>> # Create prediction UDF that uses trained model
>>> predict_udf = make_pandas_expr_udf(
...     computed_kwargs_expr=model_udaf.on_expr(train_data),
...     fn=predict_penguin_species,
...     schema=test_schema,
...     return_type=dt.string,
...     name="predict_species"
>>> )
>>>
>>> # Apply predictions to test data
>>> result = test_data.mutate(
...     predicted_species=predict_udf.on_expr(test_data)
>>> ).execute()

Penguin size classification with pre-computed thresholds:

>>> def compute_size_thresholds(df):
...     df_clean = df.dropna(subset=['body_mass_g'])
...     return {
...         'small_threshold': df_clean['body_mass_g'].quantile(0.33),
...         'large_threshold': df_clean['body_mass_g'].quantile(0.67)
...     }
>>>
>>> def classify_penguin_size(thresholds, df):
...     def classify_size(mass):
...         if pd.isna(mass):
...             return 'Unknown'
...         elif mass < thresholds['small_threshold']:
...             return 'Small'
...         elif mass > thresholds['large_threshold']:
...             return 'Large'
...         else:
...             return 'Medium'
...
...     return df['body_mass_g'].apply(classify_size)
>>>
>>> # Create threshold computation UDAF
>>> threshold_udaf = agg.pandas_df(
...     fn=compute_size_thresholds,
...     schema=penguins.select(['body_mass_g']).schema(),
...     return_type=dt.Struct({
...         'small_threshold': dt.float64,
...         'large_threshold': dt.float64
...     }),
...     name="compute_thresholds"
>>> )
>>>
>>> # Create size classification UDF
>>> size_classify_udf = make_pandas_expr_udf(
...     computed_kwargs_expr=threshold_udaf.on_expr(penguins),
...     fn=classify_penguin_size,
...     schema=penguins.select(['body_mass_g']).schema(),
...     return_type=dt.string,
...     name="classify_size",
...     post_process_fn=lambda x: x  # thresholds are already a dict
>>> )
>>>
>>> # Apply size classification
>>> result = penguins.mutate(
...     size_category=size_classify_udf.on_expr(penguins)
>>> ).execute()

Notes
-----
This UDF type is particularly powerful for ML workflows where you need to:
1. Train a model on aggregated data
2. Serialize the trained model
3. Use the model for predictions on new data

The computed_kwargs_expr is evaluated once and its result is passed to every
invocation of the main function, enabling efficient model reuse.

See Also
--------
make_pandas_udf : For standard pandas-based scalar UDFs
agg : For aggregation functions

### pyarrow_udwf(fn='__no__default__', schema='__no__default__', return_type='__no__default__', name=None, namespace=Namespace(catalog=None, database=None), base=<class 'xorq.vendor.ibis.expr.operations.udf.AggUDF'>, **config_kwargs)

Create a User-Defined Window Function (UDWF) using PyArrow.

This decorator creates window functions that can process partitions of data with
support for ordering, framing, and ranking. UDWFs are powerful for implementing
custom analytics functions that need to operate over ordered sets of data within
partitions.

Parameters
----------
fn : callable
    The window function implementation. The signature depends on config_kwargs:

    - Basic window function: `fn(self, values: list[pa.Array], num_rows: int) -> pa.Array`
    - With window frame: `fn(self, values: list[pa.Array], eval_range: tuple[int, int]) -> pa.Scalar`
    - With ranking: `fn(self, num_rows: int, ranks_in_partition: list[tuple[int, int]]) -> pa.Array`

schema : Schema
    Input schema defining column names and data types.
return_type : DataType
    The return data type of the window function.
name : str, optional
    Name of the UDWF. If None, uses the function name.
namespace : Namespace, optional
    Database and catalog namespace for the function.
base : class, default AggUDF
    Base class for the UDWF (typically AggUDF).
**config_kwargs
    Configuration options:

    - `uses_window_frame` (bool): Whether function uses window framing
    - `supports_bounded_execution` (bool): Whether function supports bounded execution
    - `include_rank` (bool): Whether function uses ranking information
    - Custom parameters: Additional parameters accessible via `self` in the function

Returns
-------
callable
    A UDWF constructor that can be used in window expressions.

Examples
--------
Exponential smoothing for penguin body mass by species:

>>> from xorq.expr.udf import pyarrow_udwf
>>> import pyarrow as pa
>>> import xorq.api as xo
>>> import xorq.expr.datatypes as dt
>>> from xorq.vendor import ibis
>>>
>>> # Load penguins dataset
>>> penguins = xo.examples.penguins.fetch(backend=xo.connect())
>>>
>>> @pyarrow_udwf(
...     schema=ibis.schema({"body_mass_g": dt.float64}),
...     return_type=dt.float64,
...     alpha=0.8  # Custom smoothing parameter
... )
>>> def smooth_body_mass(self, values: list[pa.Array], num_rows: int) -> pa.Array:
...     results = []
...     curr_value = 0.0
...     mass_values = values[0]  # body_mass_g column
...
...     for idx in range(num_rows):
...         if idx == 0:
...             curr_value = float(mass_values[idx].as_py() or 0)
...         else:
...             new_val = float(mass_values[idx].as_py() or curr_value)
...             curr_value = new_val * self.alpha + curr_value * (1.0 - self.alpha)
...         results.append(curr_value)
...
...     return pa.array(results)
>>>
>>> # Apply smoothing within each species, ordered by year
>>> result = penguins.mutate(
...     smooth_mass=smooth_body_mass.on_expr(penguins).over(
...         ibis.window(group_by="species", order_by="year")
...     )
>>> ).execute()

Running difference in penguin bill measurements:

>>> @pyarrow_udwf(
...     schema=ibis.schema({"bill_length_mm": dt.float64}),
...     return_type=dt.float64,
...     uses_window_frame=True
... )
>>> def bill_length_diff(self, values: list[pa.Array], eval_range: tuple[int, int]) -> pa.Scalar:
...     start, stop = eval_range
...     bill_values = values[0]
...
...     if start == stop - 1:  # Single row
...         return pa.scalar(0.0)
...
...     current_val = bill_values[stop - 1].as_py() or 0
...     previous_val = bill_values[start].as_py() or 0
...     return pa.scalar(float(current_val - previous_val))
>>>
>>> # Calculate difference from previous measurement within species
>>> result = penguins.mutate(
...     bill_diff=bill_length_diff.on_expr(penguins).over(
...         ibis.window(
...             group_by="species",
...             order_by="year",
...             preceding=1,
...             following=0
...         )
...     )
>>> ).execute()

Penguin ranking within species by body mass:

>>> @pyarrow_udwf(
...     schema=ibis.schema({"body_mass_g": dt.float64}),
...     return_type=dt.float64,
...     include_rank=True
... )
>>> def mass_rank_score(self, num_rows: int, ranks_in_partition: list[tuple[int, int]]) -> pa.Array:
...     results = []
...     for idx in range(num_rows):
...         # Find rank for current row
...         rank = next(
...             i + 1 for i, (start, end) in enumerate(ranks_in_partition)
...             if start <= idx < end
...         )
...         # Convert rank to score (higher rank = higher score)
...         score = 1.0 - (rank - 1) / len(ranks_in_partition)
...         results.append(score)
...     return pa.array(results)
>>>
>>> # Calculate mass rank score within each species
>>> result = penguins.mutate(
...     mass_rank_score=mass_rank_score.on_expr(penguins).over(
...         ibis.window(group_by="species", order_by="body_mass_g")
...     )
>>> ).execute()

Complex penguin feature calculation across measurements:

>>> @pyarrow_udwf(
...     schema=ibis.schema({
...         "bill_length_mm": dt.float64,
...         "bill_depth_mm": dt.float64,
...         "flipper_length_mm": dt.float64
...     }),
...     return_type=dt.float64,
...     window_size=3  # Custom parameter for moving average
... )
>>> def penguin_size_trend(self, values: list[pa.Array], num_rows: int) -> pa.Array:
...     bill_length = values[0]
...     bill_depth = values[1]
...     flipper_length = values[2]
...
...     results = []
...     window_size = self.window_size
...
...     for idx in range(num_rows):
...         # Calculate size metric for current and surrounding rows
...         start_idx = max(0, idx - window_size // 2)
...         end_idx = min(num_rows, idx + window_size // 2 + 1)
...
...         size_metrics = []
...         for i in range(start_idx, end_idx):
...             if (bill_length[i].is_valid and bill_depth[i].is_valid and
...                 flipper_length[i].is_valid):
...                 # Composite size metric
...                 bill_area = bill_length[i].as_py() * bill_depth[i].as_py()
...                 size_metric = bill_area * flipper_length[i].as_py()
...                 size_metrics.append(size_metric)
...
...         # Average size metric in window
...         avg_size = sum(size_metrics) / len(size_metrics) if size_metrics else 0.0
...         results.append(avg_size)
...
...     return pa.array(results)
>>>
>>> # Apply size trend calculation within each species
>>> result = penguins.mutate(
...     size_trend=penguin_size_trend.on_expr(penguins).over(
...         ibis.window(group_by="species", order_by="year")
...     )
>>> ).execute()

Notes
-----
The function signature and behavior changes based on configuration:

- **Standard window function**: Processes all rows in partition, returns array
- **Frame-based**: Processes specific row ranges, returns scalar per invocation
- **Rank-aware**: Has access to ranking information within partition

Custom parameters passed in config_kwargs are accessible as `self.parameter_name`
in the function implementation.

See Also
--------
scalar : For row-by-row processing
agg : For aggregation across groups
make_pandas_udf : For pandas-based scalar operations

### agg.pyarrow(fn=None, name=None, signature=None, **kwargs)

Decorator for creating PyArrow-based aggregation functions.

This method creates high-performance aggregation UDFs that operate directly
on PyArrow arrays using PyArrow compute functions. It's ideal for numerical
computations and operations that benefit from vectorized processing.

Parameters
----------
fn : callable, optional
    The aggregation function. Should accept PyArrow arrays and return a scalar
    or array result using PyArrow compute functions.
name : str, optional
    Name of the UDF. If None, uses the function name.
signature : Signature, optional
    Function signature specification for type checking.
**kwargs
    Additional configuration parameters like volatility settings.

Returns
-------
callable
    A UDF decorator that can be applied to functions, or if fn is provided,
    the wrapped UDF function.

Examples
--------
Creating a PyArrow aggregation for penguin bill measurements:

>>> import pyarrow.compute as pc
>>> from xorq.expr.udf import agg
>>> import xorq.expr.datatypes as dt
>>> import xorq.api as xo
>>>
>>> # Load penguins dataset
>>> penguins = xo.examples.penguins.fetch(backend=xo.connect())
>>>
>>> @agg.pyarrow
>>> def bill_length_range(arr: dt.float64) -> dt.float64:
...     return pc.subtract(pc.max(arr), pc.min(arr))
>>>
>>> # Calculate bill length range by species
>>> result = penguins.group_by("species").agg(
...     length_range=bill_length_range(penguins.bill_length_mm)
>>> ).execute()

Creating a weighted average for penguin measurements:

>>> @agg.pyarrow
>>> def weighted_avg_flipper(lengths: dt.float64, weights: dt.float64) -> dt.float64:
...     return pc.divide(
...         pc.sum(pc.multiply(lengths, weights)),
...         pc.sum(weights)
...     )
>>>
>>> # Weighted average flipper length by body mass
>>> result = penguins.group_by("species").agg(
...     weighted_flipper=weighted_avg_flipper(
...         penguins.flipper_length_mm,
...         penguins.body_mass_g
...     )
>>> ).execute()

Notes
-----
- PyArrow UDAFs typically offer the best performance for numerical operations
- Functions receive PyArrow arrays and should use PyArrow compute functions
- The return type should be compatible with PyArrow scalar types
- Use this for high-performance aggregations on large datasets

See Also
--------
agg.pandas_df : For pandas DataFrame-based aggregations
agg.builtin : For database-native aggregate functions

### agg.pandas_df(fn='__no__default__', schema='__no__default__', return_type='__no__default__', database=None, catalog=None, name=None, **kwargs)

Create a pandas DataFrame-based aggregation function.

This method creates aggregation UDFs that operate on pandas DataFrames,
providing access to the full pandas ecosystem for complex aggregations.
It's particularly useful for statistical operations, machine learning
model training, and complex data transformations that are easier to
express with pandas.

Parameters
----------
fn : callable
    The aggregation function. Should accept a pandas DataFrame and return
    a value compatible with the return_type. The DataFrame contains all
    columns specified in the schema for each group.
schema : Schema or dict
    Input schema defining column names and their data types.
return_type : DataType
    The return data type of the aggregation.
database : str, optional
    Database name for the UDF namespace.
catalog : str, optional
    Catalog name for the UDF namespace.
name : str, optional
    Name of the UDF. If None, generates a name from the function.
**kwargs
    Additional configuration parameters (e.g., volatility settings).

Returns
-------
callable
    A UDF constructor that can be used in aggregation expressions.

Examples
--------
Training a KNN classifier on penguin data as an aggregation:

>>> import pickle
>>> from sklearn.neighbors import KNeighborsClassifier
>>> from xorq.expr.udf import agg
>>> import xorq.expr.datatypes as dt
>>> import xorq.api as xo
>>>
>>> # Load penguins dataset
>>> penguins = xo.examples.penguins.fetch(backend=xo.connect())
>>> features = ['bill_length_mm', 'bill_depth_mm', 'flipper_length_mm', 'body_mass_g']
>>>
>>> def train_penguin_classifier(df):
...     # Remove rows with missing values
...     df_clean = df.dropna(subset=features + ['species'])
...     X = df_clean[features]
...     y = df_clean['species']
...
...     model = KNeighborsClassifier(n_neighbors=3)
...     model.fit(X, y)
...     return pickle.dumps(model)
>>>
>>> # Create the aggregation UDF
>>> penguin_schema = penguins.select(features + ['species']).schema()
>>> train_model_udf = agg.pandas_df(
...     fn=train_penguin_classifier,
...     schema=penguin_schema,
...     return_type=dt.binary,
...     name="train_penguin_classifier"
>>> )
>>>
>>> # Train one model per island
>>> trained_models = penguins.group_by("island").agg(
...     model=train_model_udf.on_expr(penguins)
>>> ).execute()

Complex statistical aggregation for penguin measurements:

>>> def penguin_stats(df):
...     return {
...         'bill_ratio_mean': (df['bill_length_mm'] / df['bill_depth_mm']).mean(),
...         'mass_flipper_corr': df['body_mass_g'].corr(df['flipper_length_mm']),
...         'count': len(df),
...         'size_score': (df['body_mass_g'] * df['flipper_length_mm']).mean()
...     }
>>>
>>> stats_schema = penguins.select([
...     'bill_length_mm', 'bill_depth_mm', 'body_mass_g', 'flipper_length_mm'
>>> ]).schema()
>>>
>>> stats_udf = agg.pandas_df(
...     fn=penguin_stats,
...     schema=stats_schema,
...     return_type=dt.Struct({
...         'bill_ratio_mean': dt.float64,
...         'mass_flipper_corr': dt.float64,
...         'count': dt.int64,
...         'size_score': dt.float64
...     }),
...     name="penguin_stats"
>>> )
>>>
>>> # Calculate statistics by species
>>> result = penguins.group_by("species").agg(
...     stats=stats_udf.on_expr(penguins)
>>> ).execute()

Feature selection for penguin classification:

>>> def select_best_penguin_features(df, n_features=2):
...     from sklearn.feature_selection import mutual_info_classif
...     import pandas as pd
...
...     df_clean = df.dropna()
...     features = ['bill_length_mm', 'bill_depth_mm', 'flipper_length_mm', 'body_mass_g']
...     X = df_clean[features]
...     y = df_clean['species']
...
...     scores = mutual_info_classif(X, y)
...     return list(pd.Series(scores, index=features).nlargest(n_features).index)
>>>
>>> feature_selector = agg.pandas_df(
...     fn=select_best_penguin_features,
...     schema=penguins.schema(),
...     return_type=dt.Array(dt.string),
...     name="select_penguin_features"
>>> )
>>>
>>> # Find best features by island
>>> best_features = penguins.group_by("island").agg(
...     top_features=feature_selector.on_expr(penguins)
>>> ).execute()

Notes
-----
- The function receives a pandas DataFrame containing all rows in each group
- PyArrow arrays are automatically converted to pandas for processing
- The function can return complex data structures (dicts, lists) if return_type supports it
- Use this when you need pandas-specific functionality or ML libraries
- Performance may be lower than PyArrow UDAFs for simple numerical operations
- Particularly powerful for ML model training workflows

See Also
--------
agg.pyarrow : For high-performance PyArrow-based aggregations
make_pandas_expr_udf : For using trained models in prediction UDFs
make_pandas_udf : For scalar pandas operations

### flight_udxf(expr='__no__default__', process_df='__no__default__', maybe_schema_in='__no__default__', maybe_schema_out='__no__default__', name=None, make_server=None, make_connection=None, con=None, inner_name=None, make_udxf_kwargs=(), **kwargs)

Create a User-Defined Exchange Function (UDXF) that executes a pandas DataFrame
transformation via Apache Arrow Flight protocol.

This function wraps a pandas-based data processing function in an Arrow Flight
ephemeral server, enabling distributed execution of custom user-defined functions.
The function creates a FlightUDXF operation that can be integrated into xorq
expression pipelines for scalable data processing.

Parameters
----------
expr : Expr
    The input Ibis expression that provides data to the UDXF. This expression's
    output will be streamed to the Flight server for processing.

process_df : callable
    A function that takes a pandas DataFrame as input and returns a transformed
    pandas DataFrame. This function defines the core transformation logic that
    will be executed on the Flight server. The function signature should be:
    `process_df(df: pd.DataFrame) -> pd.DataFrame`

maybe_schema_in : Schema or callable
    Input schema specification. Can be either:
    - A pyarrow Schema object defining the expected input schema
    - A callable that validates the input schema and returns True/False
    Used to validate that the input expression's schema matches expectations.

maybe_schema_out : Schema or callable
    Output schema specification. Can be either:
    - A pyarrow Schema object defining the expected output schema
    - A callable that computes the output schema from the input schema
    Used to determine the schema of the transformed data.

name : str, optional
    Name for the resulting table in the target backend. If not provided,
    a unique name will be generated automatically.

make_server : callable, optional
    Factory function for creating the Arrow Flight server. Defaults to
    creating an mTLS-enabled FlightServer with client verification.
    The function should return a FlightServer instance.

make_connection : callable, optional
    Factory function for creating connections to backends. Defaults to
    `xo.connect`. Used for establishing connections during Flight operations.

con : Backend, optional
    Target backend connection where the result will be materialized.
    If not provided, uses the backend from the input expression.

inner_name : str, optional
    Internal name for the FlightUDXF operation. If not provided,
    a unique name will be generated.

make_udxf_kwargs : tuple, optional
    Additional keyword arguments to pass to the UDXF creation process.
    Should be a tuple of (key, value) pairs that will be converted to
    a dictionary and passed to `make_udxf`.

**kwargs : dict
    Additional keyword arguments passed to the FlightUDXF constructor.

Returns
-------
Expr
    A Xorq expression representing the transformed data. This expression
    can be further chained with other operations or executed to materialize
    the results.

Examples
--------
Basic sentiment analysis:

>>> import pandas as pd
>>> import xorq.api as xo
>>> from xorq.common.utils.toolz_utils import curry
>>>
>>> @curry
>>> def add_sentiment(df: pd.DataFrame, input_col, output_col):
...     # Simplified sentiment analysis
...     sentiments = df[input_col].apply(lambda x: "POSITIVE" if "good" in x.lower() else "NEGATIVE")
...     return df.assign(**{output_col: sentiments})
>>>
>>> # Define schemas
>>> schema_in = xo.schema({"text": "string"})
>>> schema_out = xo.schema({"text": "string", "sentiment": "string"})
>>>
>>> # Create the UDXF
>>> sentiment_udxf = xo.expr.relations.flight_udxf(
...     process_df=add_sentiment(input_col="text", output_col="sentiment"),
...     maybe_schema_in=schema_in,
...     maybe_schema_out=schema_out,
...     name="SentimentAnalyzer"
... )
>>>
>>> # Apply to data
>>> data = xo.memtable({"text": ["This is good", "This is bad"]})
>>> result = data.pipe(sentiment_udxf).execute()

Data fetching and processing:

>>> @curry
>>> def fetch_external_data(df, api_endpoint):
...     # Fetch additional data for each row
...     results = []
...     for _, row in df.iterrows():
...         # Simulate API call
...         enriched_data = {"id": row["id"], "enriched": f"data_for_{row['id']}"}
...         results.append(enriched_data)
...     return pd.DataFrame(results)
>>>
>>> fetch_udxf = xo.expr.relations.flight_udxf(
...     process_df=fetch_external_data(api_endpoint="https://api.example.com"),
...     maybe_schema_in=xo.schema({"id": "int64"}).to_pyarrow(),
...     maybe_schema_out=xo.schema({"id": "int64", "enriched": "string"}).to_pyarrow(),
...     name="DataEnricher"
... ) # quartodoc: +SKIP

Notes
-----
- The function uses Apache Arrow Flight for efficient data transfer between
  the client and server processes
- By default, the Flight server uses mTLS (mutual TLS) for secure communication
- The process_df function is executed in a separate process/server, enabling
  distributed processing and isolation
- Schema validation ensures type safety and prevents runtime errors
- The function is curried using toolz.curry, allowing partial application

----------------------------------------------------------------------
This is the CLI documentation for the xorq command-line tool.
----------------------------------------------------------------------

## xorq init

```
Usage: xorq init [OPTIONS]

  Scaffold a new Xorq project from a template.

  Each template has a pinned default branch; pass `--branch` to check out a
  different branch or commit of the template repo.

  Templates:
  - `cached-fetcher`—cached data-fetching workflows (the default)
  - `sklearn`—ML workflows with scikit-learn
  - `penguins`—penguins dataset example pipeline

  Examples:
    # Scaffold the default template
    xorq init
    # Scaffold the sklearn template in a custom directory
    xorq init --template sklearn --path ./ml-project

Options:
  -p, --path TEXT                 Path to initialize the template.  [default:
                                  ./xorq-template]
  -t, --template [cached-fetcher|sklearn|penguins]
                                  Template to use.  [default: cached-fetcher]
  -b, --branch TEXT               Branch to use for the template.
  --help                          Show this message and exit.
```

## xorq completion

```
Usage: xorq completion [OPTIONS] [SHELL]

  Print a shell-completion script to stdout.

  Pipe or `eval` the output to enable tab completion in your current shell.
  For a one-shot install to the standard location, use `xorq install-
  completion` instead.

  Arguments:
    SHELL  One of bash, zsh, fish. Defaults to detecting `$SHELL`.

  Examples:
    # bash (add to ~/.bashrc)
    eval "$(xorq completion bash)"
    # zsh (add to ~/.zshrc)
    eval "$(xorq completion zsh)"
    # fish
    xorq completion fish | source

Options:
  --help  Show this message and exit.
```

## xorq install-completion

```
Usage: xorq install-completion [OPTIONS] [SHELL]

  Install the shell-completion script to the standard location.

  After installation, restart your shell or source the generated file to
  activate completion. The command prints the path it wrote and the source
  command that activates completion in the current session.

  Install paths:
  - bash: `~/.local/share/bash-completion/completions/xorq`
  - zsh: `~/.zfunc/_xorq` (requires `~/.zfunc` in `fpath`)
  - fish: `~/.config/fish/completions/xorq.fish`

  Arguments:
    SHELL  One of bash, zsh, fish. Defaults to detecting `$SHELL`.

  Examples:
    # Detect the shell from $SHELL and install
    xorq install-completion
    # Pin a specific shell
    xorq install-completion zsh

Options:
  --help  Show this message and exit.
```

## xorq build

```
Usage: xorq build [OPTIONS] SCRIPT_PATH

  Compile a Xorq expression into a reusable build artifact.

  Loads the script, finds the expression variable, and writes serialized
  artifacts (expression YAML, backend profiles, deferred reads, and metadata)
  to the builds directory. Execute the artifact later with `xorq run`, or add
  it to a catalog with `xorq catalog add`.

  Arguments:
    SCRIPT_PATH  Path to the Python script that defines the expression.

  Examples:
    # Build the expression named `expr` (the default)
    xorq build pipeline.py
    # Build a specific expression into a custom directory
    xorq build pipeline.py -e daily_metrics --builds-dir artifacts

Options:
  -e, --expr-name TEXT            Name of the expression variable in the
                                  Python script.  [default: expr]
  --builds-dir TEXT               Directory for all generated artifacts.
                                  [default: builds]
  --cache-dir TEXT                Directory for parquet cache files.
                                  [default: (`$XORQ_CACHE_DIR` or
                                  `~/.cache/xorq`)]
  --debug                         Output SQL files and other debug artifacts.
  --relocate-reads / --no-relocate-reads
                                  Bundle local-file Read nodes into the build
                                  so it is self-contained and runnable from
                                  anywhere. Remote reads (s3://, gs://, ...)
                                  are already location-independent and left in
                                  place. Pass --no-relocate-reads for a lean,
                                  machine-local build; this only affects reads
                                  not already bundled -- relocation discards a
                                  read's original path, so it cannot be undone
                                  by a later --no-relocate-reads on an
                                  already-relocated input.  [default:
                                  relocate-reads]
  --emit-build-path-to PATH       Write the resulting build directory path to
                                  this file. Use when stdout may be polluted
                                  (for example by OTel console fallback) and a
                                  subprocess consumer needs the path
                                  unambiguously.
  --help                          Show this message and exit.
```

## xorq uv build

```
Usage: xorq uv build [OPTIONS] SCRIPT_PATH

  Build an expression inside a uv-managed isolated environment.

  Mirrors `xorq build`, but runs inside a uv-managed environment seeded from
  the script's `pyproject.toml` (or PEP 723 inline metadata), so the build
  records dependency-faithful requirements.

  Arguments:
    SCRIPT_PATH  Path to the Python script that defines the expression.

  Examples:
    # Build with an auto-discovered project root and all extras
    xorq uv build pipeline.py -e expr --builds-dir builds
    # Build pinning a specific project root
    xorq uv build pipeline.py --project-path ./pipeline-project
    # Include only specific extras (disables --all-extras)
    xorq uv build pipeline.py --no-all-extras --extra ml --extra postgres

Options:
  -e, --expr-name TEXT            Name of the expression variable in the
                                  Python script.  [default: expr]
  --builds-dir TEXT               Directory for all generated artifacts.
                                  [default: builds]
  --cache-dir TEXT                Directory for parquet cache files.
                                  [default: (`$XORQ_CACHE_DIR` or
                                  `~/.cache/xorq`)]
  --project-path DIRECTORY        Explicit project root.  [default: (search
                                  upward from the script for pyproject.toml)]
  --pep723                        Use PEP 723 inline metadata from the script
                                  instead of a project's pyproject.toml.
  --extra TEXT                    Optional dependency group to include in
                                  requirements (repeatable).
  --all-extras / --no-all-extras  Include all optional dependency groups.
                                  [default: all-extras]
  --debug                         Output SQL files and other debug artifacts.
  --relocate-reads / --no-relocate-reads
                                  Bundle local-file Read nodes into the build
                                  so it is self-contained and runnable from
                                  anywhere. Remote reads (s3://, gs://, ...)
                                  are already location-independent and left in
                                  place. Pass --no-relocate-reads for a lean,
                                  machine-local build; this only affects reads
                                  not already bundled -- relocation discards a
                                  read's original path, so it cannot be undone
                                  by a later --no-relocate-reads on an
                                  already-relocated input.  [default:
                                  relocate-reads]
  --emit-build-path-to PATH       Write the resulting build directory path to
                                  this file. Use when stdout may be polluted
                                  (for example by OTel console fallback) and a
                                  subprocess consumer needs the path
                                  unambiguously.
  --help                          Show this message and exit.
```

## xorq run

```
Usage: xorq run [OPTIONS] BUILD_PATH

  Execute a build artifact and write results in your chosen format.

  Loads the build, resolves its data sources, executes the expression on the
  recorded backend, and writes the result as csv, json (NDJSON), parquet, or
  arrow.

  Arguments:
    BUILD_PATH  Path to the build directory produced by `xorq build`.

  Examples:
    # Write results as parquet (the default format)
    xorq run builds/f02d28198715 -o results.parquet
    # Stream CSV to stdout and pipe onward
    xorq run builds/f02d28198715 -o - -f csv | head -10
    # Sample 100 rows with a parameter override
    xorq run builds/f02d28198715 --limit 100 -p threshold=0.5 -o sample.parquet

Options:
  --cache-dir TEXT                Directory for parquet cache files.
                                  [default: (`$XORQ_CACHE_DIR` or
                                  `~/.cache/xorq`)]
  -o, --output-path TEXT          Path to write output. Use '-' for stdout.
                                  [default: (`/dev/null` (discard))]
  -f, --format [csv|json|parquet|arrow]
                                  Output format.  [default: parquet]
  --limit INTEGER                 Maximum number of rows to output.  [default:
                                  (unlimited)]
  -p, --params TEXT               Override an expression parameter as
                                  key=value (repeatable, for example --params
                                  threshold=0.5).
  --help                          Show this message and exit.
```

## xorq uv run

```
Usage: xorq uv run [OPTIONS] BUILD_PATH

  Execute a build inside a uv-managed isolated environment.

  Mirrors `xorq run`, but executes with the build's packaged sdist so the
  runtime matches the dependencies recorded at build time.

  Arguments:
    BUILD_PATH  Path to the build directory produced by `xorq uv build`.

  Examples:
    # Save results to parquet
    xorq uv run builds/7061dd65ff3c -o results.parquet
    # Stream JSON to stdout
    xorq uv run builds/7061dd65ff3c -f json -o -

Options:
  --cache-dir TEXT                Directory for parquet cache files.
                                  [default: (`$XORQ_CACHE_DIR` or
                                  `~/.cache/xorq`)]
  -o, --output-path TEXT          Path to write output. Use '-' for stdout.
                                  [default: (`/dev/null` (discard))]
  -f, --format [csv|json|parquet|arrow]
                                  Output format.  [default: parquet]
  --limit INTEGER                 Maximum number of rows to output.  [default:
                                  (unlimited)]
  -p, --params TEXT               Override an expression parameter as
                                  key=value (repeatable, for example --params
                                  threshold=0.5).
  --help                          Show this message and exit.
```

## xorq run-cached

```
Usage: xorq run-cached [OPTIONS] BUILD_PATH

  Run a build with a parquet cache wrapping the expression.

  Identical to `xorq run` in semantics, but wraps the expression in a parquet
  cache so subsequent invocations short-circuit when inputs haven't changed.

  Cache strategies:
  - `modification-time` (default): ParquetCache. Inputs are tracked by
    file modification time; the cache invalidates when an input file's
    mtime changes.
  - `snapshot`: ParquetSnapshotCache. The cache is keyed by a content
    snapshot of the inputs and never invalidates implicitly.
  - `snapshot` with `--ttl`: ParquetTTLSnapshotCache. The cache entry
    also expires after the supplied TTL (in seconds).

  Arguments:
    BUILD_PATH  Path to the build directory produced by `xorq build`.

  Examples:
    # Default modification-time cache
    xorq run-cached builds/f02d28198715 --cache-dir ./cache -o results.parquet
    # Snapshot cache with a 1-hour TTL
    xorq run-cached builds/f02d28198715 --cache-type snapshot --ttl 3600 -o results.parquet

Options:
  --cache-dir TEXT                Directory for parquet cache files.
                                  [default: (`$XORQ_CACHE_DIR` or
                                  `~/.cache/xorq`)]
  -o, --output-path TEXT          Path to write output. Use '-' for stdout.
                                  [default: (`/dev/null` (discard))]
  -f, --format [csv|json|parquet|arrow]
                                  Output format.  [default: parquet]
  --limit INTEGER                 Maximum number of rows to output.  [default:
                                  (unlimited)]
  --cache-type [modification-time|snapshot]
                                  Cache strategy: 'modification-time'
                                  (ParquetCache) or 'snapshot'
                                  (ParquetSnapshotCache).  [default:
                                  modification-time]
  --ttl INTEGER                   TTL in seconds for snapshot cache (uses
                                  ParquetTTLSnapshotCache when set).
  -p, --params TEXT               Override an expression parameter as
                                  key=value (repeatable, for example --params
                                  threshold=0.5).
  --help                          Show this message and exit.
```

## xorq uv run-cached

```
Usage: xorq uv run-cached [OPTIONS] BUILD_PATH

  Run a build with a parquet cache inside a uv-managed environment.

  Mirrors `xorq run-cached` (including its cache strategies), but executes
  with the build's packaged sdist so the runtime matches the dependencies
  recorded at build time.

  Arguments:
    BUILD_PATH  Path to the build directory produced by `xorq uv build`.

  Examples:
    # Default modification-time cache
    xorq uv run-cached builds/7061dd65ff3c --cache-dir ./cache -o results.parquet

Options:
  --cache-dir TEXT                Directory for parquet cache files.
                                  [default: (`$XORQ_CACHE_DIR` or
                                  `~/.cache/xorq`)]
  -o, --output-path TEXT          Path to write output. Use '-' for stdout.
                                  [default: (`/dev/null` (discard))]
  -f, --format [csv|json|parquet|arrow]
                                  Output format.  [default: parquet]
  --limit INTEGER                 Maximum number of rows to output.  [default:
                                  (unlimited)]
  --cache-type [modification-time|snapshot]
                                  Cache strategy: 'modification-time'
                                  (ParquetCache) or 'snapshot'
                                  (ParquetSnapshotCache).  [default:
                                  modification-time]
  --ttl INTEGER                   TTL in seconds for snapshot cache (uses
                                  ParquetTTLSnapshotCache when set).
  -p, --params TEXT               Override an expression parameter as
                                  key=value (repeatable, for example --params
                                  threshold=0.5).
  --help                          Show this message and exit.
```

## xorq run-unbound

```
Usage: xorq run-unbound [OPTIONS] BUILD_PATH

  Run an unbound expression by streaming Arrow IPC input.

  Executes a built expression after replacing one of its nodes with record
  batches streamed in over Arrow IPC—useful for piping data between
  expressions without standing up a Flight server. If neither `--to-unbind-
  hash` nor `--to-unbind-tag` is supplied, the node is inferred from graph
  analysis; supply one for determinism.

  Arguments:
    BUILD_PATH  Path to the build directory produced by `xorq build`.

  Examples:
    # Stream input from a file
    xorq run-unbound builds/transform --to-unbind-tag source_input -i input.arrow -o results.parquet
    # Pipe arrow output from one expression into another
    xorq run builds/source -o - -f arrow | xorq run-unbound builds/transform --to-unbind-tag source_input -o results.parquet

Options:
  --to-unbind-hash TEXT           Hash of the node to unbind.  [default:
                                  (inferred)]
  --to-unbind-tag TEXT            Tag of the node to unbind (alternative to
                                  --to-unbind-hash).  [default: (inferred)]
  --typ TEXT                      Type of the node to unbind.
  -o, --output-path TEXT          Path to write output. Use '-' for stdout.
                                  [default: (stdout (arrow) / discard
                                  (other))]
  -f, --format [csv|json|parquet|arrow]
                                  Output format.  [default: parquet]
  --limit INTEGER                 Maximum number of rows to output.  [default:
                                  (unlimited)]
  --batch-size INTEGER            Batch size for Arrow streaming output.
                                  [default: (table default)]
  --cache-dir TEXT                Directory for parquet cache files.
                                  [default: (`$XORQ_CACHE_DIR` or
                                  `~/.cache/xorq`)]
  -i, --instream FILENAME         Stream to read Arrow IPC record batches
                                  from.  [default: (stdin)]
  --help                          Show this message and exit.
```

## xorq uv run-unbound

```
Usage: xorq uv run-unbound [OPTIONS] BUILD_PATH

  Run an unbound expression over Arrow IPC inside a uv-managed environment.

  Mirrors `xorq run-unbound`, but executes with the build's packaged sdist so
  the runtime matches the dependencies recorded at build time.

  Arguments:
    BUILD_PATH  Path to the build directory produced by `xorq uv build`.

  Examples:
    # Stream input from a file
    xorq uv run-unbound builds/transform --to-unbind-tag source_input -i input.arrow -o results.parquet

Options:
  --to-unbind-hash TEXT           Hash of the node to unbind.  [default:
                                  (inferred)]
  --to-unbind-tag TEXT            Tag of the node to unbind (alternative to
                                  --to-unbind-hash).  [default: (inferred)]
  --typ TEXT                      Type of the node to unbind.
  -o, --output-path TEXT          Path to write output. Use '-' for stdout.
                                  [default: (stdout (arrow) / discard
                                  (other))]
  -f, --format [csv|json|parquet|arrow]
                                  Output format.  [default: parquet]
  --limit INTEGER                 Maximum number of rows to output.  [default:
                                  (unlimited)]
  --batch-size INTEGER            Batch size for Arrow streaming output.
                                  [default: (table default)]
  --cache-dir TEXT                Directory for parquet cache files.
                                  [default: (`$XORQ_CACHE_DIR` or
                                  `~/.cache/xorq`)]
  -i, --instream PATH             Path to a file with Arrow IPC data.
                                  [default: (stdin)]
  --help                          Show this message and exit.
```

## xorq serve-flight-udxf

```
Usage: xorq serve-flight-udxf [OPTIONS] BUILD_PATH

  Serve an expression's UDXF nodes as an Arrow Flight endpoint.

  Loads the built expression, detects its UDXF (user-defined exchange
  function) nodes, and hosts them with `FlightServer.from_udxf`. Clients
  connect with `xo.flight.connect`, fetch the exchange by its command name
  with `con.get_exchange`, and stream data through it.

  Arguments:
    BUILD_PATH  Path to the build directory produced by `xorq build`.

  Examples:
    # Serve a built UDXF expression
    xorq serve-flight-udxf builds/f02d28198715 --host 0.0.0.0 --port 8080

Options:
  --host TEXT                Host to bind the Flight server.  [default:
                             localhost]
  --port INTEGER             Port to bind the Flight server.  [default:
                             (random)]
  --prometheus-port INTEGER  Port to expose Prometheus metrics.  [default:
                             (off)]
  --duckdb-path TEXT         Path to the DuckDB database file used by the
                             server.  [default:
                             (`<build_path>/xorq_serve.db`)]
  --cache-dir TEXT           Directory for parquet cache files.  [default:
                             (`$XORQ_CACHE_DIR` or `~/.cache/xorq`)]
  --help                     Show this message and exit.
```

## xorq serve-unbound

```
Usage: xorq serve-unbound [OPTIONS] BUILD_PATH

  Serve an unbound expression as an Arrow Flight endpoint.

  Replaces a selected node of the built expression with an unbound table and
  serves the result; clients stream record batches to the endpoint to drive
  the computation. If neither `--to-unbind-hash` nor `--to-unbind-tag` is
  supplied, the node is inferred from graph analysis; supply one for
  determinism.

  Arguments:
    BUILD_PATH  Path to the build directory produced by `xorq build`.

  Examples:
    # Serve with an explicit node hash
    xorq serve-unbound builds/7061dd65ff3c --host 0.0.0.0 --port 8001 --to-unbind-hash b2370a29c19df8e1e639c63252dacd0e
    # Select the node to unbind by tag
    xorq serve-unbound builds/7061dd65ff3c --to-unbind-tag source_input

Options:
  --to-unbind-hash TEXT      Hash of the node to unbind.  [default:
                             (inferred)]
  --to-unbind-tag TEXT       Tag of the node to unbind (alternative to --to-
                             unbind-hash).  [default: (inferred)]
  --typ TEXT                 Type of the node to unbind.
  --host TEXT                Host to bind the Flight server.  [default:
                             localhost]
  --port INTEGER             Port to bind the Flight server.  [default:
                             (random)]
  --prometheus-port INTEGER  Port to expose Prometheus metrics.  [default:
                             (off)]
  --cache-dir TEXT           Directory for parquet cache files.  [default:
                             (`$XORQ_CACHE_DIR` or `~/.cache/xorq`)]
  --help                     Show this message and exit.
```

## xorq pin

```
Usage: xorq pin [OPTIONS] BUILD_PATH

  Freeze a build's caches into direct reads (pin).

  Each materialized cache becomes a direct read of its cache file, so the
  pinned build executes by reading artifacts instead of re-deriving them.
  Combine with --relocate-reads to produce a self-contained, portable build.

  Arguments:
    BUILD_PATH  Path to the build directory produced by `xorq build`.

  Examples:
    # Pin a build whose caches are already materialized
    xorq pin builds/f02d28198715 --cache-dir ./cache
    # Materialize any missing caches, then pin into a portable bundle
    xorq pin builds/f02d28198715 --cache-dir ./cache -e --relocate-reads

Options:
  --builds-dir TEXT               Directory for the resulting build artifact.
                                  [default: builds]
  --cache-dir TEXT                Directory for parquet cache files.
                                  [default: (`$XORQ_CACHE_DIR` or
                                  `~/.cache/xorq`)]
  -e, --ensure-materialized       Materialize any unpopulated caches (by
                                  executing) before pinning.
  --relocate-reads / --no-relocate-reads
                                  Bundle local-file Read nodes (including
                                  frozen caches) into the build so it is self-
                                  contained and runnable from anywhere. Remote
                                  reads (s3://, gs://, ...) are already
                                  location-independent and left in place. Pass
                                  --no-relocate-reads for a lean, machine-
                                  local build; this only affects reads not
                                  already bundled -- relocation discards a
                                  read's original path, so it cannot be undone
                                  by a later --no-relocate-reads on an
                                  already-relocated input.  [default:
                                  relocate-reads]
  --help                          Show this message and exit.
```

## xorq unpin

```
Usage: xorq unpin [OPTIONS] BUILD_PATH

  Thaw a pinned build's frozen caches back into recomputable caches (unpin).

  Inverse of `xorq pin`: each frozen cache read is rebuilt into its original
  cache node, restoring the recompute-capable form.

  Arguments:
    BUILD_PATH  Path to a pinned build directory.

  Examples:
    xorq unpin builds/f02d28198715 --cache-dir ./cache

Options:
  --builds-dir TEXT               Directory for the resulting build artifact.
                                  [default: builds]
  --cache-dir TEXT                Directory for parquet cache files.
                                  [default: (`$XORQ_CACHE_DIR` or
                                  `~/.cache/xorq`)]
  --relocate-reads / --no-relocate-reads
                                  Bundle local-file Read nodes into the build
                                  so it is self-contained and runnable from
                                  anywhere. Remote reads (s3://, gs://, ...)
                                  are already location-independent and left in
                                  place. Pass --no-relocate-reads for a lean,
                                  machine-local build; this only affects reads
                                  not already bundled -- relocation discards a
                                  read's original path, so it cannot be undone
                                  by a later --no-relocate-reads on an
                                  already-relocated input.  [default:
                                  relocate-reads]
  --help                          Show this message and exit.
```

## xorq catalog init

```
Usage: xorq catalog init [OPTIONS]

  Create a new catalog repository.

  The destination comes from the catalog group's selectors (`-n/--name` or
  `-p/--path`), supplied before the `init` subcommand. The command refuses to
  overwrite an existing catalog at the target path.

  Examples:
    # Create a catalog at the default location for the name
    xorq catalog --name analytics init
    # Create at an explicit path with a git remote
    xorq catalog --path ./catalogs/analytics init --remote-url git@github.com:acme/analytics-catalog.git
    # Create with an S3-backed annex remote sourced from an env file
    xorq catalog --name analytics init --env-file .env.catalog.s3

Options:
  --env-file FILE                 Env file for the annex remote (for example
                                  .env.catalog.s3; mutually exclusive with
                                  --env-prefix).
  --env-prefix TEXT               Env-var prefix for the annex remote (for
                                  example XORQ_CATALOG_S3_; mutually exclusive
                                  with --env-file).
  --gcs                           Apply GCS defaults to S3 config (annex
                                  remote or content store).
  --content-store [s3|directory]  Create a pointer-backend catalog with the
                                  given content store type.
  --remote-url TEXT               Git remote URL (sets origin).
  --help                          Show this message and exit.
```

## xorq catalog clone

```
Usage: xorq catalog clone [OPTIONS] URL

  Clone an existing catalog from a remote URL.

  `--name` and `--path` are mutually exclusive; with neither, a default
  location derives from the source URL.

  Arguments:
    URL  The git URL of the source catalog repository.

  Examples:
    # Clone to the default location
    xorq catalog clone git@github.com:acme/analytics-catalog.git
    # Clone under a specific name
    xorq catalog clone git@github.com:acme/analytics-catalog.git --name analytics
    # Clone to an explicit path
    xorq catalog clone git@github.com:acme/analytics-catalog.git --path ./catalogs/analytics

Options:
  -n, --name TEXT  Destination catalog name.
  -p, --path PATH  Destination repo path.
  --help           Show this message and exit.
```

## xorq catalog info

```
Usage: xorq catalog info [OPTIONS]

  Show catalog summary metadata.

  Prints the filesystem path, current commit, configured remotes, and entry
  and alias counts. For per-entry metadata, use `xorq catalog show` or `xorq
  catalog schema`.

  Examples:
    # Inspect the default catalog
    xorq catalog info
    # Inspect a specific catalog via the group selectors
    xorq catalog --name my-catalog info

Options:
  --help  Show this message and exit.
```

## xorq catalog default

```
Usage: xorq catalog default [OPTIONS]

  Show or change the persisted default catalog name.

  The default catalog applies when `xorq catalog` invocations supply neither
  `-n/--name` nor `-p/--path`. Without options, prints the resolved name and
  the source that supplied it. `--set` and `--unset` are mutually exclusive.

  Resolution order:
  1. The `XORQ_DEFAULT_CATALOG` environment variable.
  2. The persisted config file (written by `--set`).
  3. The built-in default name (`default`).

  Examples:
    # Show the current default and where it came from
    xorq catalog default
    # Persist a new default
    xorq catalog default --set analytics
    # Remove the persisted default
    xorq catalog default --unset

Options:
  --set TEXT  Set the default catalog name.
  --unset     Remove the persisted default.
  --help      Show this message and exit.
```

## xorq catalog tui

```
Usage: xorq catalog tui [OPTIONS]

  Browse the catalog interactively in a terminal UI.

  Shows entries, aliases, schemas, and metadata, refreshing on an interval.
  The TUI exits on `q` or `Ctrl+C`.

  Examples:
    # Browse the default catalog
    xorq catalog tui
    # Browse a named catalog with a faster refresh
    xorq catalog --name analytics tui --refresh 2

Options:
  --refresh FLOAT  Refresh interval in seconds.  [default: 10]
  --help           Show this message and exit.
```

## xorq catalog add

```
Usage: xorq catalog add [OPTIONS] PATHS...

  Add entries from build directories or archive files.

  Arguments:
    PATHS  One or more paths; each is a build directory (typically
           produced by `xorq build`) or an archive file (.zip).

  Examples:
    # Add a single build directory
    xorq catalog add builds/f02d28198715
    # Add multiple builds in one invocation
    xorq catalog add builds/f02d28198715 builds/c5a981ab43cd
    # Assign aliases at add time (applied to each added entry)
    xorq catalog add builds/f02d28198715 -a penguins-prod -a current
    # Defer the push to a later `xorq catalog push`
    xorq catalog add builds/f02d28198715 --no-sync

Options:
  --sync / --no-sync  Push the catalog to its remotes after the operation.
                      [default: sync]
  -a, --alias TEXT    Alias to assign to each added entry (repeatable).
  --help              Show this message and exit.
```

## xorq catalog remove

```
Usage: xorq catalog remove [OPTIONS] NAMES...

  Remove entries by name.

  Arguments:
    NAMES  One or more entry names; use `xorq catalog list` to see
           available names.

  Examples:
    # Remove a single entry
    xorq catalog remove f02d28198715
    # Remove multiple entries in a single commit
    xorq catalog remove f02d28198715 c5a981ab43cd
    # Remove without syncing to the remote
    xorq catalog remove f02d28198715 --no-sync

Options:
  --sync / --no-sync  Push the catalog to its remotes after the operation.
                      [default: sync]
  --help              Show this message and exit.
```

## xorq catalog list

```
Usage: xorq catalog list [OPTIONS]

  List all entries.

  Examples:
    # Names only (one per line)
    xorq catalog list
    # Names with a kind column (tab-separated)
    xorq catalog list --kind

Options:
  --kind / --no-kind  Print a second column showing each entry's kind.
                      [default: no-kind]
  --help              Show this message and exit.
```

## xorq catalog show

```
Usage: xorq catalog show [OPTIONS] NAME

  Show full metadata for a catalog entry.

  Prints name, aliases, kind, backends, schemas, parameters, builders, cache
  key, and more. `--json` and `--raw` are mutually exclusive.

  Arguments:
    NAME  Entry name or alias.

  Examples:
    # Human-readable metadata
    xorq catalog show penguins-prod
    # Structured sidecar metadata as JSON
    xorq catalog show penguins-prod --json

Options:
  --json  Output as JSON.
  --raw   Print the metadata sidecar file as-is (YAML).
  --help  Show this message and exit.
```

## xorq catalog schema

```
Usage: xorq catalog schema [OPTIONS] NAME

  Show the schemas for a catalog entry.

  Bound entries print a single output schema; unbound (partial) entries print
  both an input and an output schema.

  Arguments:
    NAME  Entry name or alias.

  Examples:
    # Formatted schema view
    xorq catalog schema penguins-prod
    # Machine-readable metadata
    xorq catalog schema penguins-prod --json

Options:
  --json  Output as JSON.
  --help  Show this message and exit.
```

## xorq catalog lineage

```
Usage: xorq catalog lineage [OPTIONS] NAME

  Show the lineage recorded in an entry's metadata sidecar.

  The sidecar stores one node table with scope-tagged edges; the compact
  boundary tree is derived from it, so no level re-walks the expression.

  `--level` picks how much detail, `--node` picks how much of the graph,
  `--format` picks the rendering, and `--expand` opens a node up: its columns
  are listed under it in the tree, or inside it in the diagram. With `--node`,
  the compact level prints the subtree feeding that node — including a Flight
  boundary's nested input lineage — and a handle matching several nodes (a
  kind, a tag) prints each match in turn.

  The TUI's Lineage panel expands the same way, with `]` and `[` on the node
  under its cursor.

  Levels:
  - `compact` (default)—the boundary-only tree, as the TUI renders it.
  - `boundaries`—one tab-separated line per boundary: id, kind, label.
  - `raw`—the stored lineage as JSON; pipe it to `jq`.

  Formats:
  - `text` (default)—the level's own format: a tree, a listing, or JSON.
  - `mermaid`—a `flowchart TD` to paste into docs or an issue. Edges point
    downstream, so sources sit at the top, and a Flight boundary's nested
    input lineage becomes a `subgraph`. With `--level compact` it draws the
    boundary graph; with `--level raw`, the expanded one, where the runs
    collapsed into `via` become real nodes. `--level boundaries` has no graph
    to draw and is rejected. A `--node` handle matching several nodes emits
    one `flowchart` block per match—paste them one at a time.

  Arguments:
    NAME  Entry name or alias.

  Examples:
    # The tree the TUI shows
    xorq catalog lineage penguins-prod
    # Grep-able boundary list
    xorq catalog lineage penguins-prod --level boundaries
    # Every stored field, for jq
    xorq catalog lineage penguins-prod --level raw | jq '.nodes[0]'
    # What feeds one node, addressed by its content hash
    xorq catalog lineage penguins-prod --node 8f3a91c2e4
    # Every UDXF's stored facts, addressed by kind
    xorq catalog lineage penguins-prod --node flight_udxf --level raw
    # A catalog-tagged source, addressed by tag value
    xorq catalog lineage penguins-prod --node catalog-source
    # A mermaid diagram, whole graph or one node's upstream
    xorq catalog lineage penguins-prod --format mermaid
    xorq catalog lineage penguins-prod --node flight_udxf -f mermaid
    # The same diagram expanded to every op, not just the boundaries
    xorq catalog lineage penguins-prod --level raw --format mermaid
    # What flows through one node: its columns, in the tree or in the diagram
    xorq catalog lineage penguins-prod --expand @cached_node_14
    xorq catalog lineage penguins-prod -f mermaid --expand @cached_node_14

Options:
  -l, --level [compact|boundaries|raw]
                                  How much lineage detail to print.  [default:
                                  compact]
  --node TEXT                     Expand one node: a snapshot hash, an @label,
                                  a tag value, or a boundary kind. Scopes
                                  every level to the match(es).
  -f, --format [text|mermaid]     Render the graph as a text tree or a mermaid
                                  flowchart.  [default: text]
  --expand TEXT                   List this node's columns under it: name and
                                  type per field, and both sides of a Flight
                                  boundary's schema change. Same handle forms
                                  as --node.
  --help                          Show this message and exit.
```

## xorq catalog get

```
Usage: xorq catalog get [OPTIONS] NAME

  Export an entry's archive to a directory.

  Writes the entry's zipped build directory to `<output_dir>/<name>.zip`,
  overwriting any existing file at that path—useful for shipping an entry to a
  system without catalog access. Re-add an exported archive with `xorq catalog
  add`.

  Arguments:
    NAME  Entry name to export.

  Examples:
    xorq catalog get f02d28198715 -o ./shipped-builds

Options:
  -o, --output PATH  Destination directory for the archive.  [default:
                     (current directory)]
  --help             Show this message and exit.
```

## xorq catalog add-alias

```
Usage: xorq catalog add-alias [OPTIONS] NAME ALIAS

  Attach an alias to an existing entry.

  Aliases are stable, human-readable names accepted anywhere an entry name is.
  To assign an alias at the same time you add an entry, use `xorq catalog add
  -a`.

  Arguments:
    NAME   The existing entry's name.
    ALIAS  The alias to register.

  Examples:
    xorq catalog add-alias f02d28198715 penguins-prod

Options:
  --sync / --no-sync  Push the catalog to its remotes after the operation.
                      [default: sync]
  --help              Show this message and exit.
```

## xorq catalog remove-alias

```
Usage: xorq catalog remove-alias [OPTIONS] ALIASES...

  Remove one or more aliases.

  The underlying entries are left untouched—use `xorq catalog remove` to
  remove an entry itself. Unknown aliases produce an error and cancel the
  operation before any change is committed.

  Arguments:
    ALIASES  One or more alias names; use `xorq catalog list-aliases`
             to see registered aliases.

  Examples:
    xorq catalog remove-alias penguins-prod
    xorq catalog remove-alias penguins-prod current

Options:
  --sync / --no-sync  Push the catalog to its remotes after the operation.
                      [default: sync]
  --help              Show this message and exit.
```

## xorq catalog list-aliases

```
Usage: xorq catalog list-aliases [OPTIONS]

  List all aliases.

  Prints one alias per line, or `No aliases.` when the catalog has none.

Options:
  --help  Show this message and exit.
```

## xorq catalog compose

```
Usage: xorq catalog compose [OPTIONS] [ENTRIES]...

  Compose entries into a new expression and persist it to the catalog.

  Assembles one or more catalog entries (and optionally inline Ibis code) into
  a new expression, builds it, and always catalogs the result. To execute a
  composed expression for data output without persisting, use `xorq catalog
  run` instead.

  Arguments:
    ENTRIES  One or more entries (names or aliases); the first is the
             source and subsequent entries apply as transforms.

  Examples:
    # Compose source + transform and catalog the result
    xorq catalog compose source-table transform-step --alias prod-pipeline
    # Add inline transformation code
    xorq catalog compose source-table -c "source.filter(source.amount > 100)" --alias high-value
    # Preview without building
    xorq catalog compose source-table transform-step --dry-run
    # Rename a parameter on one entry of the composition
    xorq catalog compose src trn --rename-params trn,threshold,cutoff

Options:
  --sync / --no-sync              Push the catalog to its remotes after the
                                  operation.  [default: sync]
  -c, --code TEXT                 Inline Ibis code expression applied to
                                  `source`.
  -a, --alias TEXT                Also register this alias for the cataloged
                                  entry.
  --cache-dir TEXT                Directory for parquet cache files.
                                  [default: (`$XORQ_CACHE_DIR` or
                                  `~/.cache/xorq`)]
  --dry-run                       Show composition plan without building.
  --rename-params TEXT            Rename a parameter on a specific entry:
                                  entry,old_name,new_name (repeatable).
  --use-this-venv / --no-use-this-venv
                                  Load and build the expression in the current
                                  Python environment instead of spawning `uv
                                  tool run` on the entries' joint bundle.
                                  Faster (no subprocess), but only correct
                                  when the calling venv already has every
                                  package each entry's wheel depends on.
                                  Default is the isolated `uv tool run` path.
  --help                          Show this message and exit.
```

## xorq catalog run

```
Usage: xorq catalog run [OPTIONS] [ENTRIES]...

  Compose and execute catalog entries, writing data to disk or stdout.

  A single entry runs directly; multiple entries compose as source +
  transforms. Unbound entries can have their input streamed in over Arrow IPC.
  To persist a composed expression back to the catalog instead of executing
  it, use `xorq catalog compose`.

  Arguments:
    ENTRIES  One or more entry names or aliases; multiple entries
             compose as source + transforms.

  Examples:
    # Run a single entry directly
    xorq catalog run src -o - -f csv
    # Compose source + transform and execute
    xorq catalog run src trn -o - -f csv
    # Add inline transformation code
    xorq catalog run src trn -c "source.filter(source.amount > 100)" -o - -f csv
    # Pipe Arrow data into an unbound entry
    xorq catalog run src -o - -f arrow | xorq catalog run trn -o - -f csv
    # Override an expression parameter
    xorq catalog run pipeline -p threshold=0.5 -o results.parquet

Options:
  -c, --code TEXT                 Inline Ibis code expression applied to
                                  `source`.
  -o, --output-path TEXT          Path to write output. Use '-' for stdout.
                                  [default: (`/dev/null` (discard))]
  -f, --format [csv|json|parquet|arrow]
                                  Output format.  [default: parquet]
  --limit INTEGER                 Maximum number of rows to output.  [default:
                                  (unlimited)]
  -i, --instream FILENAME         Stream to read Arrow IPC record batches
                                  from.  [default: (stdin)]
  --fuse / --no-fuse              Enable catalog source fusion.  [default:
                                  fuse]
  --rename-params TEXT            Rename a parameter on a specific entry:
                                  entry,old_name,new_name (repeatable).
  -p, --params TEXT               Override an expression parameter as
                                  key=value (repeatable, for example --params
                                  threshold=0.5).
  --cache-dir TEXT                Directory for parquet cache files.
                                  [default: (`$XORQ_CACHE_DIR` or
                                  `~/.cache/xorq`)]
  --use-this-venv / --no-use-this-venv
                                  Execute in the current Python environment
                                  instead of spawning `uv tool run` on the
                                  entry's pinned env. Faster (no subprocess +
                                  uv venv lookup) but only correct when the
                                  calling venv already has every package the
                                  expression needs (Xorq itself plus any UDFs
                                  from the entries' wheels). Default is the
                                  isolated `uv tool run` path.
  --help                          Show this message and exit.
```

## xorq catalog run-cached

```
Usage: xorq catalog run-cached [OPTIONS] [ENTRIES]...

  Compose and execute catalog entries with a parquet cache wrapper.

  Same semantics as `xorq catalog run`, but wraps the resulting expression
  with a cache so subsequent invocations short-circuit when inputs haven't
  changed (ParquetCache by default; snapshot and TTL variants via `--cache-
  type` and `--ttl`—see `xorq run-cached` for the strategies).

  Arguments:
    ENTRIES  One or more entry names or aliases.

  Examples:
    # Default modification-time cache
    xorq catalog run-cached src trn --cache-dir ./cache -o results.parquet
    # Snapshot cache with a 1-hour TTL
    xorq catalog run-cached pipeline --cache-type snapshot --ttl 3600 -o results.parquet

Options:
  -c, --code TEXT                 Inline Ibis code expression applied to
                                  `source`.
  -o, --output-path TEXT          Path to write output. Use '-' for stdout.
                                  [default: (`/dev/null` (discard))]
  -f, --format [csv|json|parquet|arrow]
                                  Output format.  [default: parquet]
  --limit INTEGER                 Maximum number of rows to output.  [default:
                                  (unlimited)]
  -i, --instream FILENAME         Stream to read Arrow IPC record batches
                                  from.  [default: (stdin)]
  --fuse / --no-fuse              Enable catalog source fusion.  [default:
                                  fuse]
  --rename-params TEXT            Rename a parameter on a specific entry:
                                  entry,old_name,new_name (repeatable).
  -p, --params TEXT               Override an expression parameter as
                                  key=value (repeatable, for example --params
                                  threshold=0.5).
  --cache-dir TEXT                Directory for parquet cache files.
                                  [default: (`$XORQ_CACHE_DIR` or
                                  `~/.cache/xorq`)]
  --cache-type [modification-time|snapshot]
                                  Cache strategy: 'modification-time'
                                  (ParquetCache) or 'snapshot'
                                  (ParquetSnapshotCache).  [default:
                                  modification-time]
  --ttl INTEGER                   TTL in seconds for snapshot cache (uses
                                  ParquetTTLSnapshotCache when set).
  --use-this-venv / --no-use-this-venv
                                  Execute in the current Python environment
                                  instead of spawning `uv tool run` on the
                                  entry's pinned env. Faster (no subprocess +
                                  uv venv lookup) but only correct when the
                                  calling venv already has every package the
                                  expression needs (Xorq itself plus any UDFs
                                  from the entries' wheels). Default is the
                                  isolated `uv tool run` path.
  --help                          Show this message and exit.
```

## xorq catalog serve-unbound

```
Usage: xorq catalog serve-unbound [OPTIONS] ENTRY

  Resolve a catalog entry, unbind a node, and serve it via Flight.

  The catalog-aware counterpart to the top-level `xorq serve-unbound`: instead
  of pointing at a build directory, you reference an entry by name or alias
  and Xorq resolves it from the active catalog. Clients stream record batches
  to the endpoint to drive the computation.

  Arguments:
    ENTRY  Catalog entry name or alias to serve.

  Examples:
    # Serve an aliased entry, unbinding by tag
    xorq catalog serve-unbound flights-model --to-unbind-tag source_input --host 0.0.0.0 --port 8001
    # Bind a runtime parameter
    xorq catalog serve-unbound scorer --params threshold=0.5

Options:
  --to-unbind-hash TEXT      Hash of the node to unbind.  [default:
                             (inferred)]
  --to-unbind-tag TEXT       Tag of the node to unbind (alternative to --to-
                             unbind-hash).  [default: (inferred)]
  --typ TEXT                 Type of the node to unbind.
  --host TEXT                Host to bind the Flight server.  [default:
                             localhost]
  --port INTEGER             Port to bind the Flight server.  [default:
                             (random)]
  --prometheus-port INTEGER  Port to expose Prometheus metrics.  [default:
                             (off)]
  -c, --code TEXT            Inline Ibis code expression applied to `source`.
  --fuse / --no-fuse         Enable catalog source fusion.  [default: fuse]
  --rename-params TEXT       Rename a parameter on a specific entry:
                             entry,old_name,new_name (repeatable).
  -p, --params TEXT          Override an expression parameter as key=value
                             (repeatable, for example --params threshold=0.5).
  --cache-dir TEXT           Directory for parquet cache files.  [default:
                             (`$XORQ_CACHE_DIR` or `~/.cache/xorq`)]
  --help                     Show this message and exit.
```

## xorq catalog pin

```
Usage: xorq catalog pin [OPTIONS] ENTRY

  Freeze a catalog entry's caches and persist the result as a new entry.

  Pinning changes the build hash, so it always yields a new content-named
  entry rather than mutating the source. Use --alias to name it, or --move-
  aliases to move every alias (e.g. `prod`) from the source entry onto the
  pinned entry.

  Arguments:
    ENTRY  An entry name or alias.

  Examples:
    xorq catalog pin penguins-prod --alias penguins-pinned
    xorq catalog pin penguins-prod --move-aliases

Options:
  --sync / --no-sync              Push the catalog to its remotes after the
                                  operation.  [default: sync]
  --cache-dir TEXT                Directory for parquet cache files.
                                  [default: (`$XORQ_CACHE_DIR` or
                                  `~/.cache/xorq`)]
  -a, --alias TEXT                Register this alias for the new entry.
  --move-aliases                  Move all of the source entry's aliases onto
                                  the new entry.
  -e, --ensure-materialized       Materialize any unpopulated caches (by
                                  executing) before pinning.
  --relocate-reads / --no-relocate-reads
                                  Bundle local-file Read nodes (including
                                  frozen caches) into the entry so it is self-
                                  contained and runnable from anywhere. Remote
                                  reads (s3://, gs://, ...) are already
                                  location-independent and left in place. Pass
                                  --no-relocate-reads for a lean, machine-
                                  local entry; this only affects reads not
                                  already bundled -- relocation discards a
                                  read's original path, so it cannot be undone
                                  by a later --no-relocate-reads on an
                                  already-relocated input.  [default:
                                  relocate-reads]
  --help                          Show this message and exit.
```

## xorq catalog unpin

```
Usage: xorq catalog unpin [OPTIONS] ENTRY

  Thaw a pinned catalog entry and persist the result as a new entry.

  Inverse of `xorq catalog pin`: rebuilds frozen cache reads back into
  recomputable caches. Like pin, this yields a new content-named entry.

  Arguments:
    ENTRY  An entry name or alias.

  Examples:
    xorq catalog unpin penguins-pinned --move-aliases

Options:
  --sync / --no-sync              Push the catalog to its remotes after the
                                  operation.  [default: sync]
  --cache-dir TEXT                Directory for parquet cache files.
                                  [default: (`$XORQ_CACHE_DIR` or
                                  `~/.cache/xorq`)]
  -a, --alias TEXT                Register this alias for the new entry.
  --move-aliases                  Move all of the source entry's aliases onto
                                  the new entry.
  --relocate-reads / --no-relocate-reads
                                  Bundle local-file Read nodes into the entry
                                  so it is self-contained and runnable from
                                  anywhere. Remote reads (s3://, gs://, ...)
                                  are already location-independent and left in
                                  place. Pass --no-relocate-reads for a lean,
                                  machine-local entry; this only affects reads
                                  not already bundled -- relocation discards a
                                  read's original path, so it cannot be undone
                                  by a later --no-relocate-reads on an
                                  already-relocated input.  [default:
                                  relocate-reads]
  --help                          Show this message and exit.
```

## xorq catalog push

```
Usage: xorq catalog push [OPTIONS]

  Push the catalog's commits and annex content to its remotes.

  Pair with `--no-sync` on mutating commands (`add`, `remove`, `add-alias`,
  `remove-alias`) when batching changes for a single explicit push.

  Examples:
    xorq catalog add builds/f02d28198715 --no-sync
    xorq catalog add-alias f02d28198715 prod --no-sync
    xorq catalog push

Options:
  --help  Show this message and exit.
```

## xorq catalog pull

```
Usage: xorq catalog pull [OPTIONS]

  Pull the catalog's commits and annex content from its remotes.

Options:
  --help  Show this message and exit.
```

## xorq catalog sync

```
Usage: xorq catalog sync [OPTIONS]

  Pull then push.

  Equivalent to running `xorq catalog pull` followed by `xorq catalog push`.

Options:
  --help  Show this message and exit.
```

## xorq catalog set-remote

```
Usage: xorq catalog set-remote [OPTIONS] URL

  Configure the catalog's git remote.

  The catalog supports at most one git remote (ADR-0011). If no git remote is
  configured, this command sets one. If a git remote is already configured,
  this command refuses unless `--force` is passed— guarding against typos that
  would silently delete the configured remote.

  Arguments:
    URL  Git remote URL to configure.

  Examples:
    # First-time setup
    xorq catalog --path ~/flights-catalog set-remote git@github.com:me/flights-catalog.git
    # Replace an existing remote
    xorq catalog set-remote git@github.com:me/flights-catalog.git --force

Options:
  --name TEXT  Remote name.  [default: (origin)]
  --force      Replace the existing git remote (otherwise this command refuses
               to overwrite).
  --help       Show this message and exit.
```

## xorq catalog embed-readonly

```
Usage: xorq catalog embed-readonly [OPTIONS]

  Embed read-only S3 credentials into the catalog's git-annex branch.

  Lets consumers cloning the catalog fetch annexed content without supplying
  credentials of their own. One of `--env-file` or `--env-prefix` is required.

  Before embedding, the command probes the bucket by initiating (and
  canceling) a multipart upload with the supplied credentials. If that write
  succeeds, the credentials aren't read-only and the command raises an error
  instead of embedding them. Embedding sets `embedcreds=yes` on the S3 remote
  config and writes the result to `remote.log` on the git-annex branch.

  Examples:
    xorq catalog embed-readonly --env-file .env.catalog.readonly
    xorq catalog embed-readonly --env-prefix XORQ_CATALOG_RO_

Options:
  --env-file FILE    Env file for the annex remote (for example
                     .env.catalog.s3; mutually exclusive with --env-prefix).
  --env-prefix TEXT  Env-var prefix for the annex remote (for example
                     XORQ_CATALOG_S3_; mutually exclusive with --env-file).
  --gcs              Apply GCS defaults to S3 config (annex remote or content
                     store).
  --help             Show this message and exit.
```

## xorq catalog check

```
Usage: xorq catalog check [OPTIONS]

  Validate the catalog for consistency.

  Verifies that entries declared in the catalog manifest are present on disk,
  that aliases point to existing entries, and that recorded metadata matches
  actual content. Prints `OK` on success; otherwise exits non-zero with a
  description of the inconsistency.

Options:
  --help  Show this message and exit.
```

## xorq catalog gc

```
Usage: xorq catalog gc [OPTIONS]

  Remove orphaned content store objects (pointer backend only).

Options:
  --dry-run / --no-dry-run  List orphans without deleting.
  --help                    Show this message and exit.
```

## xorq catalog log

```
Usage: xorq catalog log [OPTIONS]

  Show the catalog's history as structured operations.

  These are the same operations `xorq catalog replay` consumes when copying a
  catalog.

  Examples:
    xorq catalog log
    xorq catalog log --json | jq '.[] | select(.type == "Add")'

Options:
  --json  Output as JSON.
  --help  Show this message and exit.
```

## xorq catalog replay

```
Usage: xorq catalog replay [OPTIONS] TARGET_PATH

  Replay the catalog's operation log into a target catalog.

  Useful for mirroring a catalog with a different remote backend, or for
  migrating between storage providers.

  With `--rebuild`, each entry is re-added under current code: entries with no
  catalog references are re-added from their stored expression; entries
  containing catalog references (Composed, or ExprBuilder wrapping a
  composition) have the catalog subtree recomposed against their already-
  rebuilt dependencies in the target catalog. Outer builder wrappings pass
  through untouched.

  Arguments:
    TARGET_PATH  Filesystem path where the target catalog is initialized.

  Examples:
    # Preview the operations that would be replayed
    xorq catalog replay ./mirrored-catalog --dry-run
    # Replay into a new catalog and push to a fresh remote
    xorq catalog replay ./mirrored-catalog --env-file .env.target.s3 --remote-url git@github.com:acme/mirror-catalog.git

Options:
  --env-file FILE                 Env file for the annex remote (for example
                                  .env.catalog.s3; mutually exclusive with
                                  --env-prefix).
  --env-prefix TEXT               Env-var prefix for the annex remote (for
                                  example XORQ_CATALOG_S3_; mutually exclusive
                                  with --env-file).
  --gcs                           Apply GCS defaults to S3 config (annex
                                  remote or content store).
  --content-store [s3|directory]  Create a pointer-backend catalog with the
                                  given content store type.
  --remote-url TEXT               Git remote URL for the target catalog (sets
                                  origin and pushes).
  --preserve-commits / --no-preserve-commits
                                  Preserve original commit authors and
                                  timestamps.  [default: preserve-commits]
  --force                         Force-push to the remote.
  --dry-run                       Show what would be replayed without
                                  executing.
  --rebuild                       Rebuild each entry under current code
                                  (refreshes build_metadata and entry hashes).
  --help                          Show this message and exit.
```

----------------------------------------------------------------------
This is the user guide documentation for xorq.
----------------------------------------------------------------------

## Get started — Quickstart

After completing this guide, you have a working ML pipeline. This pipeline loads data, trains a model, and generates predictions. It uses Xorq's deferred execution model.

## What you'll build

In this quickstart, you:

1. Set up your environment and install Xorq
2. Initialize a project
3. Build a pipeline expression
4. Run your pipeline and save results
5. Deploy your pipeline as an API endpoint
6. Serve a UDXF expression as an endpoint

The entire process takes about five minutes. By the end, you understand how Xorq transforms Python code into executable, servable pipelines.

## Step 1: Set up your environment and install Xorq

This step covers environment setup, Xorq installation, and verification.

### Check your Python version

::: {.callout-tip}
### Python version
Xorq requires Python 3.10 or higher, but also no higher than 3.13. Check your version with `python --version`. If you need to install or update Python, then visit the [official Python downloads page](https://www.python.org/downloads/).
:::

### Create a virtual environment

Create and activate a virtual environment:

::: {.panel-tabset}

### macOS/Linux
```bash
python -m venv .venv
source .venv/bin/activate
```

### Windows
```bash
python -m venv .venv
.venv\Scripts\activate
```

:::

### Update pip

Before installing Xorq, update pip to the latest version. This avoids compatibility issues:
```bash
python -m pip install --upgrade pip
```

### Install Xorq

Now install Xorq using your preferred package manager:

::: {.panel-tabset}

### pip (macOS/Linux/Windows)
```bash
pip install "xorq[examples]"
```

### nix
```bash
nix run github:xorq-labs/xorq
```

:::

### Verify installation

Verify your installation by checking the version. Open a Python shell:
```bash
python
```

::: {.callout-note}
If `python` doesn't work, then try `python3` instead.
:::

Import Xorq and check the version:
```python
import xorq
xorq.__version__
```

You see a version number like `'0.3.4'`.

Exit the Python shell:
```python
exit()
```

If you see a version number, then Xorq is installed correctly.

With Xorq installed, the next step is to create your first project.

## Step 2: Initialize a project

Create a new Xorq project using the built-in penguins template. This template demonstrates a complete ML workflow with the Palmer Penguins dataset:
```bash
xorq init -t penguins -p penguins_example # <1>
cd penguins_example # <2>
```
1. Creates a new project called penguins_example using the penguins template.
2. Moves into the project directory.

The template generates an `expr.py` file. This file contains a complete ML pipeline with data loading, train/test splitting, model training, and prediction.


With your project initialized, the next step is to build it into an executable format.

## Step 3: Build your expression

Convert your pipeline into a serialized, executable format using the `build` command:
```bash
xorq build expr.py
```

This serializes your pipeline and generates a content-addressed build directory.

**Output:**
```
Building expr from expr.py
Written 'expr' to builds/12287e173c17
builds/12287e173c17
```

The build creates a directory (like `builds/12287e173c17`) containing your serialized pipeline. This hash uniquely identifies your build.

![Terminal output showing the build command completing and writing the serialized pipeline to a builds directory with a content-addressed hash.](../images/quickstart/build-output-hash.png)

Save this hash as an environment variable:

::: {.panel-tabset}

### macOS/Linux
```bash
export BUILD_HASH=12287e173c17
```

Replace `12287e173c17` with your actual hash.

### Windows (PowerShell)
```powershell
$env:BUILD_HASH="12287e173c17"
```

Replace `12287e173c17` with your actual hash.

### Windows (CMD)
```cmd
set BUILD_HASH=12287e173c17
```

Replace `12287e173c17` with your actual hash.

:::

::: {.callout-important}
### Replace with your hash
The hash `12287e173c17` is an example. Copy the actual hash from your build output.
:::

## Step 4: Run your pipeline

Execute your built pipeline and view the results. Replace `<BUILD_HASH>` with the hash from your build output in step 3:

::: {.callout-warning}
### Windows users
Skip the first tab and go straight to "Save to file." Running without an output file might cause issues on Windows.
:::

::: {.panel-tabset}

### View results (macOS/Linux)
```bash
xorq run builds/$BUILD_HASH
```

This runs the pipeline and displays results in your terminal.

### Save to file (macOS/Linux)
```bash
xorq run builds/$BUILD_HASH -o predictions.parquet
```

This runs the pipeline and saves results to `predictions.parquet`.

### Save to file (Windows PowerShell)
```powershell
xorq run builds/$env:BUILD_HASH -o predictions.parquet
```

This runs the pipeline and saves results to `predictions.parquet`.

### Save to file (Windows CMD)
```cmd
xorq run builds/%BUILD_HASH% -o predictions.parquet
```

This runs the pipeline and saves results to `predictions.parquet`.

:::

With your pipeline results saved, the next step is to deploy the pipeline as a live API endpoint.

## Step 5: Deploy your pipeline as an API endpoint

Deploy the pipeline you built in step 3 as a live API endpoint.

::: {.callout-important}
### Using your build hash
This step uses the same `BUILD_HASH` from step 3. You're deploying the exact pipeline you just built and ran.
:::

### Start the server

Run the server with your build hash:

::: {.panel-tabset}

### macOS/Linux
```bash
xorq serve-unbound builds/$BUILD_HASH \
  --host localhost \
  --port 8001 \
  --to-unbind-hash d9cebcfbeeadc40f4a39814357716481
```

### Windows (PowerShell)
```powershell
xorq serve-unbound builds/$env:BUILD_HASH `
  --host localhost `
  --port 8001 `
  --to-unbind-hash d9cebcfbeeadc40f4a39814357716481
```

### Windows (CMD)
```cmd
xorq serve-unbound builds/%BUILD_HASH% --host localhost --port 8001 --to-unbind-hash d9cebcfbeeadc40f4a39814357716481
```

:::

The server starts and listens for incoming requests on the specified host and port.

With your pipeline running locally, the next step is to serve a UDXF expression as an endpoint.


## Step 6: Serve a UDXF expression

::: {.callout-note}
### New terminal setup
Open a new terminal window. Activate the same virtual environment (`.venv`) and navigate to your project directory.
:::

Create a file called `udxf_example.py`:

```python
# udxf_example.py
import pandas as pd
import xorq.api as xo

# <1>
def add_computed_column(df: pd.DataFrame) -> pd.DataFrame:
    """Add a computed column that doubles the input value."""
    result = df.copy()
    result["doubled"] = result["value"] * 2
    return result

# <2>
input_schema = xo.schema({"value": "int64"})
output_schema = xo.schema({"value": "int64", "doubled": "int64"})

# <3>
con = xo.connect()
input_table = xo.memtable({"value": [1, 2, 3, 4, 5]}, schema=input_schema)

# <4>
expr = xo.expr.relations.flight_udxf(
    input_table,
    process_df=add_computed_column,
    maybe_schema_in=input_schema,
    maybe_schema_out=output_schema,
    con=con,
    make_udxf_kwargs={
        "name": "double_value",
        "command": "double_value"
    }
)
```
1. Defines a transformation function that doubles input values.
2. Specifies input and output schemas for type safety.
3. Creates a simple input table with test data.
4. Creates the UDXF expression with the transformation function.

### Build the UDXF expression

Build the expression:

```bash
xorq build udxf_example.py --expr-name expr
```

This creates a second build directory in `builds/<BUILD_HASH>/` with your serialized UDXF expression.

![Terminal output showing a second build hash generated for the UDXF expression.](../images/quickstart/second-hash.png)

### Start the Flight server

Start the Flight server with your built UDXF expression. Replace `<BUILD_HASH>` with the second hash from your build output:

```bash
xorq serve-flight-udxf --port 8001 builds/<BUILD_HASH>
```

Your UDXF is now running as an endpoint on `localhost:8001`. Keep this terminal window open while you query it.

### Query your served UDXF

With the server running, create a new Python file called `query_udxf.py`:

```python
# query_udxf.py
import xorq.api as xo
import random as rnd

# <1>
con = xo.flight.connect(port=8001)

# <2>
exchange = con.get_exchange("default")

# <3>
expr = xo.memtable({ "value" : rnd.choices(tuple(range(100)), k=10) }, schema= xo.schema({"value": "int64"})).pipe(exchange)

# <4>
print("Executing via Flight do_exchange...")
result_df = expr.execute()

print(result_df)
```
1. Connect to the Flight server running on port 8001. This establishes a connection to communicate with your deployed server.
2. Get the exchange named "default" from the connection. This exchange represents the UDXF transformation you deployed.
3. Create an in-memory table with random integer values and pipe it through the exchange. This builds an expression that applies your UDXF transformation to the input data.
4. Execute the expression via Flight `do_exchange` and print the results. The server applies the transformation (doubling the values) and returns the result as a pandas DataFrame.

### Run the query script

Run your query script:

```bash
python query_udxf.py
```

**Output:**
```
Executing via Flight do_exchange...
   value  doubled
0     42       84
1     17       34
2     63      126
3      8       16
4     91      182
5     55      110
6     29       58
7     74      148
8     36       72
9     12       24
```

Your UDXF now runs as a live API. You sent integer values to the server, the server applied the doubling transformation, and you received the results back—all via Flight protocol.

## What you built

You just built your first Xorq ML pipeline. You initialized a project, built it into a portable format, ran it to generate predictions, and served it as an API endpoint.

The entire workflow uses Xorq's deferred execution model. No computation runs until you explicitly execute or query the pipeline.

## Next steps

Continue learning:

- [Your first expression](/getting_started/your_first_expression.qmd) shows you how to build expressions from scratch without templates.
- [Understand deferred execution](/getting_started/understand_deferred_execution.qmd) explains how Xorq builds computation graphs before executing them.
- [Explore caching](/getting_started/explore_caching.qmd) demonstrates how Xorq caches results to speed up your workflows.
- [Join the community](https://discord.gg/8Kma9DhcJG) on Discord to get help and share your projects.

## Get started — Defer query execution

This tutorial helps you understand when Xorq runs computation versus when it builds expressions. You'll learn through hands-on examples how deferred execution works and why it matters.

After completing this tutorial, you'll know exactly when your code triggers actual computation.

## What's deferred execution?

Deferred execution means that Xorq waits to run computations until you explicitly ask for results. When you chain operations like `.filter()` and `.group_by()`, Xorq builds an expression graph but doesn't run anything yet.

This approach gives Xorq time to optimize your query before running it. Think of it as planning a route before you start driving: you see the full journey and choose the most efficient path.


### Why defer?
Deferred execution lets Xorq eliminate unnecessary steps, push computations to faster backends, and cache intermediate results. You write clear code, and Xorq handles the optimization.


## How to follow along

Run the code examples in order using any of these methods:

- **Python interactive shell (recommended)**: Open a terminal, run `python`, then copy and paste each code block.
- **Jupyter notebook**: Create a new notebook and run each code block in a separate cell.
- **Python script**: Copy all code blocks into a `.py` file and run it with `python script.py`.

The code blocks build on each other. Variables like `con`, `iris`, and `filtered` are created in earlier blocks and used in later ones.


## Build expressions without executing

You'll build an expression that loads and filters data. Notice how you can create the expression without triggering any computation.
```{python}
import xorq.api as xo

# <1>
con = xo.connect()

# <2>
iris = xo.examples.iris.fetch(backend=con)

# <3>
filtered = iris.filter(xo._.sepal_length > 6)

# <4>
print(f"Expression type: {type(filtered)}")
print(f"Has this executed? Not yet!")
```
1. Connect to the embedded backend.
2. Load the iris dataset. This creates a table reference, not the actual data.
3. Build a filter expression. Still no computation
4. Print the expression type to confirm it's just an expression object.

At this point, Xorq knows what you want to do (filter rows), but it hasn't read any data or applied any filters.

## Inspect the expression

You can look at what operations Xorq has queued up by examining the expression.
```{python}
# <1>
print("\nExpression structure:")
print(filtered)

# <2>
print(f"\nBackends involved: {filtered.ls.backends}")
```
1. Print the expression to see the operation tree.
2. Check which backends this expression would use.

The output shows you the chain of operations Xorq performs when you execute the expression. This is your expression graph.

Now that you've seen what Xorq plans to do, run the computation and see the results.

## Execute and observe computation

You'll trigger the computation by calling `.execute()`. This is when Xorq runs your query.
```{python}
# <1>
print("\nBefore execute: building plan...")

# <2>
result = filtered.execute()

# <3>
print(f"After execute: got results!")
print(f"Result type: {type(result)}")
print(f"Number of rows: {len(result)}")
print(result.head(5))
```
1. You're about to trigger computation.
2. This line executes the expression and computation happens here.
3. You now have actual results, not just an expression.

The moment you called `.execute()`, Xorq:
- Compiled your expression into an execution plan.
- Loaded the data from the iris dataset.
- Applied the filter.
- Returned the results as a PyArrow Table.

:::{.callout-note}
### Execution is explicit
Xorq never runs queries behind your back. You control exactly when computation happens by calling `.execute()` or similar methods like `.to_pandas()` or `.to_pyarrow()`.
:::

## Build complex expressions

You'll build a more complex expression with multiple operations. Watch how Xorq still defers everything.
```{python}
# <1>
complex_expr = (
    iris
    .filter(xo._.sepal_length > 5.5)
    .mutate(sepal_ratio=xo._.sepal_length / xo._.sepal_width)
    .group_by("species")
    .agg(
        avg_ratio=xo._.sepal_ratio.mean(),
        count=xo._.species.count()
    )
)

# <2>
print("Built complex expression (not executed yet):")
print(complex_expr)

# <3>
print("\nNow executing...")
result = complex_expr.execute()
print(result)
```
1. Build an expression with filtering, adding a column, grouping, and aggregating.
2. The expression exists, but no computation has run.
3. Execute and see all operations run at once.

Xorq deferred all five operations (filter, mutate, group, two aggregations) until you called `.execute()`. This gives it room to optimize the entire workflow.

But what happens if you don't defer? Compare the two approaches to see why deferred execution matters.

## Compare: Immediate vs deferred

You'll see what happens if you force early execution versus deferring.
```{python}
# <1>
immediate = iris.filter(xo._.sepal_length > 6).execute()  # Executes immediately

# <2>
print(f"Immediate approach - result type: {type(immediate)}")
print(f"This is already executed data, not an expression!")
print(f"Cannot chain more Xorq operations on materialized results")

# <3>
deferred = (
    iris
    .filter(xo._.sepal_length > 6)
    .group_by("species")
    .agg(xo._.sepal_width.sum())
)

# <4>
print(f"\nDeferred approach - expression type: {type(deferred)}")
print(f"This is an expression that can still be optimized!")
print(f"Can chain more operations or execute when ready")
```
1. Execute early by calling `.execute()` after the first operation.
2. You now have materialized data, not an expression.
3. Build the full expression without executing.
4. This stays as an expression until you explicitly execute it.

The deferred approach lets Xorq optimize the entire pipeline. The immediate approach locks in results after each step, preventing optimization.

:::{.callout-warning}
### Avoid premature execution
If you execute too early, you lose the benefits of deferred execution. Let Xorq see your full query before running it.
:::

## Complete example

Here's a full example showing deferred execution in action:

```python
import xorq.api as xo

# Connect and load data
con = xo.connect()
iris = xo.examples.iris.fetch(backend=con)

# Build expression (deferred, no computation yet)
expr = (
    iris
    .filter(xo._.sepal_length > 5.5)
    .group_by("species")
    .agg(avg_width=xo._.sepal_width.mean())
)

# Inspect without executing
print("Expression ready:", type(expr))

# Execute when you're ready
result = expr.execute()
print("Results:", result)
```

## Next steps

Now you understand deferred execution. Continue exploring:

- [Explore caching](explore_caching.qmd) shows how deferred execution enables intelligent caching
- [Your first build](../tutorials/core_tutorials/your_first_build.qmd) explains how Xorq captures expressions as portable manifests
- [Switch backends](switch_backends.qmd) demonstrates how deferred execution works across different engines

## Get started — Cache expression results

This tutorial shows you how Xorq's caching system works through hands-on examples. You'll see cache hits and misses in real time, and understand when Xorq reuses results versus recomputing them.

After completing this tutorial, you'll know how to use caching to speed up your workflows.

## Why caching matters

Running the same query twice shouldn't mean doing the work twice. Xorq caches expression results so repeated queries return instantly from the cache instead of recomputing.

This is especially powerful for expensive operations:

- Loading large datasets from remote databases
- Training machine learning models
- Calling external APIs
- Running complex aggregations

:::{.callout-tip}
### Smart caching
Xorq uses content-addressed hashing to determine if an expression matches cached results. Same computation = same hash = cache hit.
:::

## How to follow along

Run the code examples in order using any of these methods:

- **Python interactive shell (recommended)**: Open a terminal, run `python`, then copy and paste each code block.
- **Jupyter notebook**: Create a new notebook and run each code block in a separate cell.
- **Python script**: Copy all code blocks into a `.py` file and run it with `python script.py`.

The code blocks build on each other. Variables like `iris`, `storage`, and `cached_expr` are created in earlier blocks and used in later ones.


## Set up caching

You’ll start by connecting to a backend and setting up a cache storage location.

```{python}
import xorq.api as xo
from xorq.caching import SourceCache

# <1>
con = xo.connect()

# <2>
storage = SourceCache.from_kwargs(source=con)

print(f"Connected to: {con}")
print(f"Cache storage ready!")
```
1. Connect to the embedded backend where cached data is stored.
2. Create a SourceCache object that manages the cache.

`SourceCache` stores cached results in your backend as tables. When you run an expression with `.cache()`, Xorq saves the results and reuses them on subsequent runs.

## Cache your first expression

Now you’ll build an expression and add caching to it.

```{python}
# <1>
iris = xo.examples.iris.fetch(backend=con)

# <2>
cached_expr = (
    iris
    .filter(xo._.sepal_length > 6)
    .cache(cache=storage)  # <3>
)

print(f"Expression with caching: {type(cached_expr)}")
```
1. Load the iris dataset.
2. Build a filter expression.
3. Add caching with `.cache(cache=storage)`.

The `.cache()` method tells Xorq to store results from this expression. On the first run, Xorq computes and caches the results. On subsequent runs, it retrieves them directly from cache.

## Observe cache miss (first run)

You’ll execute the expression for the first time. This is a cache miss, Xorq has to compute the results.

```{python}
import time

# <1>
print("First execution (cache miss)...")
start = time.time()

# <2>
result1 = cached_expr.execute()

# <3>
elapsed = time.time() - start
print(f"✗ Cache miss: computed in {elapsed:.4f} seconds")
print(f"Result shape: {result1.shape}")
print(f"\nFirst few rows:")
print(result1.head(3))
```
1. Start timing the execution.
2. Execute the expression, triggers computation and caching.
3. Print how long it took.

Since this is the first run, Xorq computed the filter operation and stored the results in cache.

## Observe cache hit (second run)

Now you can run the same expression again. This time you'll see a cache hit.

```{python}
# <1>
print("\nSecond execution (cache hit)...")
start = time.time()

# <2>
result2 = cached_expr.execute()

# <3>
elapsed = time.time() - start
print(f"✓ Cache hit: returned in {elapsed:.4f} seconds")
print(f"Results match: {result1.equals(result2)}")
```
1. Time the second execution.
2. Run the same expression again.
3. See how much faster it was.

The second execution should be significantly faster because Xorq fetched results from cache instead of recomputing the filter operation.

:::{.callout-note}
### Cache key
Xorq computes a hash from your expression's structure and data sources. If the expression is identical, then the hash matches, and you get a cache hit.
:::

## Understand cache invalidation

What happens if you change the expression? You’ll modify the filter and see cache invalidation in action.

```{python}
# <1>
modified_expr = (
    iris
    .filter(xo._.sepal_length > 6.5)  # <2>
    .cache(cache=storage)
)

# <3>
print("Modified expression (different filter)...")
start = time.time()
result3 = modified_expr.execute()
elapsed = time.time() - start

# <4>
print(f"✗ Cache miss: computed in {elapsed:.4f} seconds")
print(f"Different result shape: {result3.shape}")
```
1. Create a new expression with a different filter threshold.
2. Changed from `> 6` to `> 6.5`—this is a different computation.
3. Execute the modified expression.
4. Cache miss because the expression changed.

Since you changed the filter threshold, Xorq computed a different hash. The cache from the previous expression doesn't match, so Xorq recomputed.

## Compare multiple runs

You’ll run several executions and see the timing difference between cache hits and misses.

```{python}
# <1>
def time_execution(expr, label):
    start = time.time()
    result = expr.execute()
    elapsed = time.time() - start
    return elapsed, len(result)

# <2>
print("\nTiming comparison:")
print("-" * 50)

# <3>
t1, rows1 = time_execution(cached_expr, "First run")
print(f"Run 1 (miss):  {t1:.4f}s - {rows1} rows")

t2, rows2 = time_execution(cached_expr, "Second run")
print(f"Run 2 (hit):   {t2:.4f}s - {rows2} rows")

t3, rows3 = time_execution(cached_expr, "Third run")
print(f"Run 3 (hit):   {t3:.4f}s - {rows3} rows")

# <4>
speedup = t1 / t2 if t2 > 0 else float('inf')
print(f"\nSpeedup from caching: {speedup:.1f}x faster")
```
1. Create a helper function to time executions.
2. Print a header for the comparison.
3. Run the same expression three times.
4. Calculate the speedup from caching.

The first execution is a cache miss (slower), but the second and third are cache hits (much faster). This shows how caching eliminates redundant computation.

:::{.callout-warning}
### Cache storage
SourceCache keeps cached data in your backend as tables. Make sure you have enough storage space for cached results, especially with large datasets.
:::

## Chain cached expressions

You can cache multiple steps in a pipeline. Each cached expression can reuse results from previous runs.

```{python}
# <1>
step1 = iris.filter(xo._.sepal_length > 5).cache(cache=storage)

# <2>
step2 = step1.group_by("species").agg(
    avg_width=xo._.sepal_width.mean()
).cache(cache=storage)

# <3>
print("First execution of step2...")
result_a = step2.execute()

# <4>
print("\nSecond execution of step2...")
result_b = step2.execute()

print("\nBoth steps now cached!")
print(result_a)
```
1. Cache the filtered dataset.
2. Build on the cached result and cache the aggregation too.
3. First execution caches both steps.
4. Second execution hits cache for both steps.

When you cache multiple steps, Xorq can reuse intermediate results, making complex pipelines faster on repeated runs.

## Complete example

Here's a full caching workflow in one place:

```python
import xorq.api as xo
from xorq.caching import SourceCache

# Set up connection and load data
con = xo.connect()
storage = SourceCache.from_kwargs(source=con)
iris = xo.examples.iris.fetch(backend=con)

# Build cached expression
cached_expr = (
    iris
    .filter(xo._.sepal_length > 6)
    .cache(cache=storage)
)

# First run: cache miss
result1 = cached_expr.execute()
print("First run complete (cached)")

# Second run: cache hit
result2 = cached_expr.execute()
print("Second run complete (from cache)")
```

## Next steps

Now you understand how caching works. Continue learning:

- [Switch backends](switch_backends.qmd) shows how caching works when moving data between engines
- [Your first build](../tutorials/core_tutorials/your_first_build.qmd) explains how cached expressions become portable artifacts

## Get started — Switch between backends

This tutorial shows you how to run the same expression on different execution engines. You'll learn when to choose each backend and see how Xorq moves data between them using Apache Arrow.

After completing this tutorial, you'll know how to pick the right backend for your workload.

## Why switch backends?

Different backends excel at different tasks. DuckDB handles analytical queries efficiently, Pandas works great for small datasets and prototyping, and DataFusion gives you custom UDF capabilities.

Xorq lets you write your expression once and run it anywhere. Same code, different engines.

:::{.callout-tip}
### Zero-copy transfers
Xorq uses Apache Arrow to move data between backends without serialization overhead. This makes backend switching fast and memory-efficient.
:::

You'll see this portability in action by running the same expression across three backends: embedded, DuckDB, and Pandas. Start with the default.

## Run on the embedded backend

You'll start with Xorq's default embedded backend. This uses a modified DataFusion engine optimized for Arrow operations.
```{python}
import xorq.api as xo

# <1>
con = xo.connect()

# <2>
iris = xo.examples.iris.fetch(backend=con)

# <3>
expr = (
    iris
    .filter(xo._.sepal_length > 6)
    .group_by("species")
    .agg(avg_width=xo._.sepal_width.mean())
)

# <4>
result = expr.execute()
print(f"Backend: {con}")
print(result)
```
1. Connect to the embedded backend (DataFusion-based).
2. Load the iris dataset into this backend.
3. Build a filter and aggregation expression.
4. Execute on the embedded backend.

The embedded backend is the default. It's fast, supports all Xorq features, and doesn't require external setup.

## Switch to DuckDB

Now you'll run the same expression on DuckDB. DuckDB excels at analytical queries and works well with larger datasets.
```{python}
# <1>
duckdb_con = xo.duckdb.connect()

# <2>
iris_duck = xo.examples.iris.fetch(backend=duckdb_con)

# <3>
duck_expr = (
    iris_duck
    .filter(xo._.sepal_length > 6)
    .group_by("species")
    .agg(avg_width=xo._.sepal_width.mean())
)

# <4>
duck_result = duck_expr.execute()
print(f"\nBackend: {duckdb_con}")
print(duck_result)
```
1. Connect to DuckDB (in-memory by default).
2. Load iris data into DuckDB.
3. Build the same expression as before.
4. Execute on DuckDB.

Notice how the expression code is identical. Only the backend connection changed.

:::{.callout-note}
### In-memory vs persistent
This DuckDB connection is in-memory. To use a persistent database file, pass `database="my_db.duckdb"` to `connect()`.
:::

## Switch to Pandas

Pandas is great for small datasets and interactive analysis. You'll run the expression there.
```{python}
# <1>
pandas_con = xo.pandas.connect()

# <2>
iris_pandas = xo.examples.iris.fetch(backend=pandas_con)

# <3>
pandas_expr = (
    iris_pandas
    .filter(xo._.sepal_length > 6)
    .group_by("species")
    .agg(avg_width=xo._.sepal_width.mean())
)

# <4>
pandas_result = pandas_expr.execute()
print(f"\nBackend: {pandas_con}")
print(pandas_result)
```
1. Connect to Pandas backend.
2. Load data into Pandas.
3. Same expression, different backend.
4. Execute on Pandas.

The Pandas backend is perfect for prototyping and working with small datasets that fit in memory.

So far, you've loaded data separately into each backend. But what if you start analysis in one backend and need to switch to another mid-workflow? That's where data transfer comes in.

## Move data between backends

Sometimes you need to move data from one backend to another. Xorq makes this easy with `.into_backend()`.
```{python}
# <1>
con = xo.connect()
duckdb_con = xo.duckdb.connect()

# <2>
data_in_embedded = xo.examples.iris.fetch(backend=con)

# <3>
data_in_duckdb = data_in_embedded.into_backend(duckdb_con)

# <4>
result = data_in_duckdb.filter(xo._.sepal_length > 6).execute()

print(f"Original backend: {con}")
print(f"Moved to backend: {duckdb_con}")
print(f"Result shape: {result.shape}")
```
1. Connect to both backends.
2. Load data into the embedded backend.
3. Move the data to DuckDB using `.into_backend()`.
4. Now you can run queries in DuckDB.

`.into_backend()` transfers data between backends using Arrow's zero-copy protocol. This is fast even for large datasets.

:::{.callout-tip}
### When to move data
Move data to a different backend when you need specific features (like DuckDB's AsOf joins) or better performance for your query type.
:::

## Compare backend performance

You'll time the same query on different backends to see performance characteristics.
```{python}
import time

def time_query(backend, name):
    """Time a query execution."""
    iris = xo.examples.iris.fetch(backend=backend)
    expr = (
        iris
        .filter(xo._.sepal_length > 5)
        .group_by("species")
        .agg(
            count=xo._.species.count(),
            avg_width=xo._.sepal_width.mean()
        )
    )
    
    start = time.time()
    result = expr.execute()
    elapsed = time.time() - start
    
    return elapsed, len(result)

# <1>
con = xo.connect()
duck = xo.duckdb.connect()
pandas = xo.pandas.connect()

# <2>
print("Timing comparison:")
print("-" * 50)

# <3>
t1, rows1 = time_query(con, "Embedded")
print(f"Embedded:  {t1:.4f}s - {rows1} rows")

t2, rows2 = time_query(duck, "DuckDB")
print(f"DuckDB:    {t2:.4f}s - {rows2} rows")

t3, rows3 = time_query(pandas, "Pandas")
print(f"Pandas:    {t3:.4f}s - {rows3} rows")
```
1. Connect to all three backends.
2. Print a comparison header.
3. Time the same query on each backend.

For small datasets like iris, performance differences are minimal. With larger datasets, you'll see DuckDB and the embedded backend outperform Pandas.

## Choose the right backend

Here's when to use each backend:

**Embedded (DataFusion):**
- Default choice for most workloads.
- Excellent UDF support.
- Fast analytical queries.
- No external dependencies.

**DuckDB:**
- Analytical queries on moderate-to-large datasets.
- AsOf joins and time-series operations.
- Efficient with Parquet files.
- Persistent storage needs.

**Pandas:**
- Small datasets (<1 GB).
- Interactive prototyping.
- Integration with existing Pandas code.
- Quick exploration.

:::{.callout-warning}
### Backend capabilities
Not all backends support every operation. For example, some complex window functions might work in DuckDB but not in Pandas. Check the documentation if you hit an unsupported operation error.
:::

Now that you understand when to use each backend, here's a complete workflow that ties everything together.

## Complete example

Here's a full example showing backend switching:
```python
import xorq.api as xo

# Connect to backends
embedded = xo.connect()
duckdb = xo.duckdb.connect()

# Load data in embedded backend
data = xo.examples.iris.fetch(backend=embedded)

# Build expression
expr = (
    data
    .filter(xo._.sepal_length > 6)
    .group_by("species")
    .agg(avg_width=xo._.sepal_width.mean())
)

# Execute on embedded backend
result1 = expr.execute()
print("Embedded result:", result1)

# Move to DuckDB and execute there
data_in_duck = data.into_backend(duckdb)
expr_duck = (
    data_in_duck
    .filter(xo._.sepal_length > 6)
    .group_by("species")
    .agg(avg_width=xo._.sepal_width.mean())
)
result2 = expr_duck.execute()
print("DuckDB result:", result2)
```

## Next steps

Now you know how to switch backends. Continue learning:

- [Your first build](../tutorials/core_tutorials/your_first_build.qmd) shows how to package expressions for deployment across backends

## Get started — Your first expression

This tutorial shows you how to write and run your first Xorq expression. You'll load data, apply a filter, and see results.

After completing this tutorial, you'll understand how Xorq builds expression graphs before executing them.

## What you'll build

You'll create an expression that:

1. Loads the iris dataset.
2. Filters rows where sepal length is greater than five.
3. Groups by species.
4. Sums the sepal widths for each group.

The entire process takes about two minutes.

::: {.callout-note}
### Prerequisites
Before starting, make sure you have Xorq installed. See [Install Xorq](../how_to/install_xorq.qmd) if you need help.
:::

:::{.callout-tip}
### Dataset included
The iris dataset comes with Xorq's examples package. You don't need to download anything separately.
:::

:::{.callout-note}
### How to follow along
Run the code examples using any of these methods:

- **Python interactive shell**: Open a terminal, run `python`, then copy and paste each code block
- **Jupyter notebook**: Create a new notebook and run each code block in a separate cell
- **Python script**: Copy all code blocks into a `.py` file and run it with `python script.py`

Run the code blocks in order, because they build on each other. Variables like `iris` and `expr` are created in earlier blocks and used in later ones.
:::

## Load data

You’ll start by loading the iris dataset. This dataset contains measurements of iris flowers across three species.

```{python}
import xorq.api as xo

# <1>
con = xo.connect()

# <2>
iris = xo.examples.iris.fetch(backend=con)

# <3>
print(iris.head(5).execute())
```
1. Connect to the embedded backend.
2. Load the iris dataset from Xorq's examples.
3. Preview the first five rows to see the data structure.

The dataset has columns for `sepal_length`, `sepal_width`, `petal_length`, `petal_width`, and `species`. You'll work with these columns to build your expression.

## Build an expression

Now you’ll build an expression that filters and aggregates the data. Here's where Xorq's deferred execution model shows its power.

```{python}
# <1>
expr = (
    iris
    .filter(xo._.sepal_length > 5)  # <2>
    .group_by("species")  # <3>
    .agg(xo._.sepal_width.sum())  # <4>
)

# <5>
print(type(expr))
print(expr)
```
1. Start building an expression from the iris table.
2. Filter rows where sepal length is greater than five.
3. Group the filtered data by species.
4. Sum the sepal widths for each species group.
5. Print the expression type and structure.

Notice that nothing has run yet. Xorq builds an expression graph that describes what you want to do, but it doesn't run the computation until you explicitly ask for it.

:::{.callout-note}
The `xo._` accessor lets you reference columns without knowing the full table schema ahead of time. It's shorthand for creating column references.
:::

## Execute the expression

Once you've built your expression, you can execute it to see results.

```{python}
# <1>
result = expr.execute()

# <2>
print(result)
print(f"\nResult type: {type(result)}")
```
1. Execute the expression to trigger computation.
2. Print the results and their type.

The result is a PyArrow Table with two columns: `species` and the summed sepal widths. This confirms your expression ran successfully.

## Understand what happened

Here’s what Xorq did:

1. **Built an expression graph**: When you chained `.filter()`, `.group_by()`, and `.agg()`, Xorq created a graph representing these operations.
2. **Waited for execution**: No computation happened until you called `.execute()`.
3. **Optimized the plan**: Xorq compiled your expression into an efficient execution plan.
4. **Ran the query**: The embedded DataFusion backend executed the plan.
5. **Returned results**: You got back a PyArrow Table with your aggregated data.

This deferred execution model gives Xorq room to optimize your queries before running them.

:::{.callout-tip}
### Try experimenting
Modify the filter condition or try different aggregation functions like `.mean()` or `.count()`. The expression-building pattern stays the same.
:::

## Complete example

Here's the full code in one place:

```python
import xorq.api as xo

# Connect and load data
con = xo.connect()
iris = xo.examples.iris.fetch(backend=con)

# Build and execute expression
expr = (
    iris
    .filter(xo._.sepal_length > 5)
    .group_by("species")
    .agg(xo._.sepal_width.sum())
)

result = expr.execute()
print(result)
```

## Next steps

Now that you've written your first expression, explore these concepts:

- [Understand deferred execution](understand_deferred_execution.qmd) explains when computation happens versus when expressions are built
- [Explore caching](explore_caching.qmd) shows you how to speed up repeated queries
- [Switch backends](switch_backends.qmd) teaches you how to run the same expression on different engines

## Get started — Claude Code plugin

You talk to Claude in plain language. It picks the matching skill and runs the
`xorq` commands for you. Source and full skill docs:
<!-- vale Xorq.Terms = NO -->
[xorq-labs/claude-plugins](https://github.com/xorq-labs/claude-plugins).
<!-- vale Xorq.Terms = YES -->

## Install

```
/plugin marketplace add xorq-labs/claude-plugins
/plugin install xorq@xorq-plugins
```

A SessionStart hook loads the [essentials.md](https://github.com/xorq-labs/claude-plugins/blob/main/xorq/skills/_shared/essentials.md) once per session, so each skill has access to the same information in context without duplication.

## Skills

| Skill | What it does |
|-------|--------------|
| [**ingest**](https://github.com/xorq-labs/claude-plugins/blob/main/xorq/skills/ingest/SKILL.md) | Bring data not yet catalogued (csv, parquet, or a DuckDB / SQLite / Postgres table) into a `source`. |
| [**composer**](https://github.com/xorq-labs/claude-plugins/blob/main/xorq/skills/composer/SKILL.md) | Shape catalogued data into a new `composed` entry with inline code, reusable transforms, or both. |
| [**catalog-explore**](https://github.com/xorq-labs/claude-plugins/blob/main/xorq/skills/catalog-explore/SKILL.md) | Read-only list, schema, row preview, and history. Also how a build is verified. |
| [**ml**](https://github.com/xorq-labs/claude-plugins/blob/main/xorq/skills/ml/SKILL.md) | Fit an sklearn pipeline over an expression, save it as an `expr_builder`, run it on new data. |
| [**builder**](https://github.com/xorq-labs/claude-plugins/blob/main/xorq/skills/builder/SKILL.md) | Round-trip tagged objects (semantic models, fitted pipelines, your own) through the catalog. |
| [**diagnose**](https://github.com/xorq-labs/claude-plugins/blob/main/xorq/skills/diagnose/SKILL.md) | Failed, slow, or stale runs, and caching. |

## Example

A session, one line per turn—what you say, and the skill Claude reaches for.

| You ask | Skill |
|---------|-------|
| "Use a catalog called `demo`." | _(setup)_ |
| "Ingest `customers.csv` as `customers`." | ingest |
| "Show the catalog and preview `customers`." | catalog-explore |
| "From `customers`, keep `age > 30`, select name and age." | composer |
| "Fit a classifier on `transactions` for `is_fraud`, save it." | ml |
| "That run was slow, what happened?" | diagnose |

Full command-line and Python API index: <https://docs.xorq.dev/llms.txt>

## How-to guides — Install Xorq

This guide shows you how to install Xorq and confirm the installation works. It also covers the optional extras that add support for specific backends.

If you want to connect to a database after installing, see [Connect to a backend](connect_to_backends.qmd).

## Prerequisites

- Python 3.10 or higher. Check with `python --version`; if you're below 3.10, grab a newer release from the [Python downloads page](https://www.python.org/downloads/).
- `pip` or [`uv`](https://docs.astral.sh/uv/).

## Steps

### 1. Install the package

::: {.panel-tabset}

#### pip

The base package includes the core library, an embedded DataFusion backend, and Pandas support.

```bash
pip install xorq
```

#### uv

For a standalone install:

```bash
uv pip install xorq
```

For a project with locked dependencies:

```bash
uv init my-xorq-project
cd my-xorq-project
uv add xorq
```

#### From source

Install the latest development version from GitHub:

```bash
pip install git+https://github.com/xorq-labs/xorq.git
```

For local development, clone and install in editable mode:

```bash
git clone https://github.com/xorq-labs/xorq.git
cd xorq
pip install -e ".[examples]"
```

#### nix

```bash
nix run github:xorq-labs/xorq
```

:::

### 2. Verify the installation

Run a query against the embedded backend. It ships with the base package, so no extra setup is needed.

```{python}
import xorq.api as xo

con = xo.connect()
iris = xo.examples.iris.fetch(backend=con)

result = (
    iris.filter(xo._.sepal_length > 5)
    .group_by("species")
    .agg(total_width=xo._.sepal_width.sum())
    .execute()
)
print(result)
```

If you see sepal widths aggregated by species, the installation works.

### 3. Install optional extras

Each extra adds support for one backend or feature set. Install only what you need.

| Extra | Command | Adds |
|---|---|---|
| `examples` | `pip install "xorq[examples]"` | Example datasets plus scikit-learn, XGBoost, and the OpenAI SDK |
| `duckdb` | `pip install "xorq[duckdb]"` | DuckDB backend |
| `postgres` | `pip install "xorq[postgres]"` | PostgreSQL backend |
| `snowflake` | `pip install "xorq[snowflake]"` | Snowflake backend |
| `datafusion` | `pip install "xorq[datafusion]"` | Standalone DataFusion backend (an embedded one is already included) |
| `sqlite` | `pip install "xorq[sqlite]"` | SQLite backend |
| `pyiceberg` | `pip install "xorq[pyiceberg]"` | Apache Iceberg tables |
| Trino | `pip install trino` | Trino client (separate package, not an extra) |

To install everything at once:

```bash
pip install "xorq[examples,duckdb,snowflake,postgres,pyiceberg,datafusion,sqlite]"
```

## Troubleshooting

**`ModuleNotFoundError: No module named 'xorq'`**: the install landed in a different environment than the one running your code. Compare `which python` against the environment you installed into, or run `pip show xorq` to see where it went.

**`xo.duckdb.connect()` (or another backend) raises an import error**: the backend extra isn't installed. Install it from the extras table.

**Build errors during `pip install`**: your Python is probably older than 3.10. Check `python --version` and upgrade if needed.

## See also

- [Connect to a backend](connect_to_backends.qmd)
- [Quickstart](../getting_started/quickstart.qmd)

## How-to guides — Connect to a backend

This guide shows you how to create a connection for each backend Xorq supports. Every connection object exposes the same expression API, so the code you write after connecting doesn't change between engines.

For picking which backend a pipeline step should run on, see [Route a step to a specific backend](switch_backends.qmd).

## Prerequisites

- Xorq installed ([Install Xorq](install_xorq.qmd))
- The extra for your backend, for example `pip install "xorq[postgres]"`
- Credentials, for remote backends like PostgreSQL or Snowflake

## Steps

### 1. Create the connection

::: {.panel-tabset}

#### Embedded

The default backend. A modified DataFusion engine, included in the base package; it needs no setup.

```{python}
import xorq.api as xo

con = xo.connect()
print(f"Connected to: {con}")
```

#### DuckDB

In-memory by default:

```{python}
import xorq.api as xo

duck_con = xo.duckdb.connect()
```

For a database that survives between sessions, pass a path:

```python
duck_con = xo.duckdb.connect(database="my_database.duckdb")
```

#### PostgreSQL

Set these environment variables, then connect with [`connect_env`](../reference/connect.qmd):

- `POSTGRES_HOST`
- `POSTGRES_PORT`
- `POSTGRES_DATABASE`
- `POSTGRES_USER`
- `POSTGRES_PASSWORD`

```{python}
import xorq.api as xo

pg_con = xo.postgres.connect_env()
```

You can also pass credentials directly:

```python
pg_con = xo.postgres.connect(
    host="localhost",
    port=5432,
    database="your_database",
    user="your_user",
    password="your_password",
)
```

:::{.callout-warning}
Don't hardcode credentials in production code. Use environment variables or the [Profiles API](../api_reference/backends/profiles_api.qmd), which stores environment-variable references instead of values.
:::

#### Snowflake

```python
import xorq.api as xo

snow_con = xo.snowflake.connect(
    user="your_user",
    password="your_password",
    account="your_account",
    role="your_role",
    warehouse="your_warehouse",
    database="your_database",
    schema="your_schema",
)
```

#### Trino

```python
import xorq.api as xo

trino_con = xo.trino.connect(
    host="localhost",
    port=8080,
    user="your_user",
    database="your_catalog",
    schema="your_schema",
)
```

#### SQLite

```{python}
import tempfile
from pathlib import Path

import xorq.api as xo

db_path = Path(tempfile.mkdtemp()) / "my_data.db"
sqlite_con = xo.sqlite.connect(database=str(db_path))
```

`connect` creates the database file if it doesn't exist. Point `database` at any path you like; this example uses a temporary one.

#### Pandas

Useful for small local datasets and tests:

```{python}
import pandas as pd
import xorq.api as xo

pandas_con = xo.pandas.connect()

df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
table = pandas_con.create_table("my_table", df)
```

:::

### 2. Verify the connection

Whichever backend you connected to, give its variable a common name so the rest of this guide can refer to it. The examples below use the embedded backend, since it needs no setup; swap in `duck_con`, `pg_con`, or any other connection you created.

```{python}
import xorq.api as xo

con = xo.connect()
```

List the tables the backend can see. An empty list is fine: the connection works, there just aren't any tables yet.

```{python}
print(con.list_tables())
```

For a deeper check, run a small expression end to end:

```{python}
penguins = xo.examples.penguins.fetch(backend=con)
print(penguins.count().execute())
```

### 3. Reuse the connection across your pipeline

A connection is the entry point for everything that follows: reading tables, registering data, and receiving data from other backends. Create it once and pass it around.

```{python}
expr = penguins.filter(xo._.body_mass_g > 4000)
print(expr.ls.backends)
```

The [`.ls` accessor](../reference/Table.qmd) tells you which backends an expression touches, which becomes useful once a pipeline spans more than one engine.

## See also

- [Supported backends](../api_reference/backends/supported_backends.qmd)
- [Environment variables](../api_reference/backends/env_variables.qmd)
- [Profiles API](../api_reference/backends/profiles_api.qmd)
- [Route a step to a specific backend](switch_backends.qmd)

## How-to guides — Cache results by backend

This guide shows you how to pick a cache class and attach it to an expression. Xorq ships four cache classes, and the right one depends on two questions: should the cache invalidate when source data changes, and where should the cached result live?

Caching is lazy. Attaching `.cache(...)` to an expression does nothing until you call `.execute()`, at which point Xorq either writes the result to storage (miss) or reads it back (hit).

## Prerequisites

- Xorq installed ([Install Xorq](install_xorq.qmd))
- A backend connection ([Connect to a backend](connect_to_backends.qmd))

## Steps

### 1. Pick the cache class

Answer two questions and read the class off the table:

| Goal | Store in a backend | Store as Parquet on disk |
|---|---|---|
| Recompute when source data changes | [`SourceCache`](../reference/SourceCache.qmd) | [`ParquetCache`](../reference/ParquetCache.qmd) |
| Keep the first result, even if data changes | [`SourceSnapshotCache`](../reference/SourceSnapshotCache.qmd) | [`ParquetSnapshotCache`](../reference/ParquetSnapshotCache.qmd) |

Rules of thumb:

- Store in a backend when downstream steps run on that backend anyway, or when the cache should live in a shared database.
- Store as Parquet when the source backend is in-memory and you want the cache to outlive it, or when no database is involved.
- Pick a Snapshot variant when you'd rather serve stale data than pay for recomputation.

### 2. Cache into a backend with `SourceCache`

`SourceCache` stores the result as a table in whatever backend you pass as `source`. The cache key includes source-data metadata, so changes upstream produce a new key and force recomputation.

```{python}
import xorq.api as xo
from xorq.caching import SourceCache

ddb = xo.duckdb.connect()
penguins = xo.examples.penguins.fetch(backend=ddb)

expr = (
    penguins.group_by("species")
    .agg(avg_mass=xo._.body_mass_g.mean())
    .cache(SourceCache.from_kwargs(source=ddb))
)
print(expr.execute())
```

Calling `.cache()` with no arguments does the same thing, using the expression's own backend as the source.

### 3. Cache to disk with `ParquetCache`

`ParquetCache` writes the result as a Parquet file, by default under `~/.cache/xorq/`.

```{python}
from xorq.caching import ParquetCache

con = xo.connect()

expr = (
    penguins.filter(xo._.sex.notnull())
    .cache(ParquetCache.from_kwargs(source=con))
)
print(expr.count().execute())
```

To put the files somewhere else, pass `base_path=Path("/some/dir")` to `from_kwargs`, or set the `XORQ_CACHE_DIR` environment variable.

Note that `penguins` lives in DuckDB while `source` here is the embedded backend. That's deliberate: for `ParquetCache` the `source` backend isn't where the data lives, it's only what writes the Parquet file on a miss and reads it back on a hit. The cache itself lives on disk, so the source can be any backend. (For `SourceCache`, by contrast, the cache *is* a table in the source backend, so the choice matters more.)

### 4. Freeze results with the Snapshot variants

[`SourceSnapshotCache`](../reference/SourceSnapshotCache.qmd) and [`ParquetSnapshotCache`](../reference/ParquetSnapshotCache.qmd) compute the cache key from the expression structure only: table name, path, schema. The key stays stable when the underlying data changes, so the first cached result is served until you delete it.

```{python}
from xorq.caching import SourceSnapshotCache

frozen = (
    penguins.group_by("island")
    .agg(n=xo._.count())
    .cache(SourceSnapshotCache.from_kwargs(source=ddb))
)
print(frozen.execute())
```

Use this when reproducibility beats freshness, like model training inputs or an expensive backfill you don't want to rerun.

### 5. Combine with `into_backend()`: DuckDB source, DataFusion target

A common pattern: read and filter in DuckDB, move the result to the embedded DataFusion backend for further work, and cache the transferred result as Parquet so the transfer doesn't repeat.

```{python}
from xorq.caching import ParquetCache

con = xo.connect()

pipeline = (
    penguins.filter(xo._.species == "Adelie")
    .into_backend(con, "adelie")
    .cache(ParquetCache.from_kwargs(source=con))
    .group_by("island")
    .agg(avg_bill=xo._.bill_length_mm.mean())
)
print(pipeline.execute())
```

On the second `.execute()`, the DuckDB read and the cross-backend transfer are skipped entirely; the pipeline starts from the Parquet file.

### 6. Check the backend caveats

Automatic invalidation relies on backend-specific change signals, and they're not all equally sharp:

| Backend | Change signal | Caveat |
|---|---|---|
| PostgreSQL | `reltuples` estimate | Updates after `ANALYZE`/autovacuum; an insert may not invalidate immediately. Run `ANALYZE <table>` to force it. |
| Snowflake | `LAST_ALTERED` timestamp | Reliable; updates on any DDL or DML. |
| DuckDB | File metadata or data bytes | In-memory tables hash the actual data, which is precise but costly for large tables. |
| SQLite (on disk) | `COUNT(*)` and `MAX(id)` | Needs an `id` column; misses updates that change neither count nor max id. |
| PyIceberg | Snapshot IDs | Tied to Iceberg's snapshot model. |

If your backend's signal is too coarse for your use case, a Snapshot variant plus explicit cache deletion gives you manual control.

## See also

- [Caching](../concepts/understanding_xorq/intelligent_caching_system.qmd): how cache keys are computed and when caches invalidate
- [Cache API overview](../api_reference/cache_api_overview.qmd): the full class matrix, how to build each cache, and the `GCSCache` and TTL variants not covered here
- [Explore caching](../getting_started/explore_caching.qmd): a hands-on walkthrough of hits, misses, and invalidation

## How-to guides — Route a step to a specific backend

This guide shows you how to move a pipeline step onto a specific backend with [`into_backend()`](../reference/Table.qmd#xorq.vendor.ibis.expr.types.relations.Table.into_backend). You have a working pipeline; one step would run better elsewhere, like a heavy filter near the data in PostgreSQL or a UDF that needs the embedded engine. One method call reroutes it.

Data moves between engines as Arrow record batches, streamed rather than materialized, so the transfer doesn't require temp files or a full copy in memory.

## Prerequisites

- Xorq installed with the DuckDB extra ([Install Xorq](install_xorq.qmd))
- Connections to at least two backends ([Connect to a backend](connect_to_backends.qmd))

## Steps

### 1. Decide where the step should run

The win comes from putting each step where it's strongest. Typical reasons to move:

| Move to | When |
|---|---|
| PostgreSQL (or your warehouse) | The data already lives there and the step reduces it: filter and aggregate before pulling anything out |
| DuckDB | The step needs analytical strength: AsOf joins, window-heavy queries, scanning Parquet files |
| Embedded (DataFusion) | The step uses Xorq features tied to the embedded engine: Python UDFs, ML steps, Flight serving |
| Nowhere (stay put) | The current backend handles the step fine; every hop has a cost, so don't pay it without a reason |

A useful default: reduce data on the backend that stores it, then move the small result to the engine doing the specialized work.

### 2. Move the expression with `into_backend()`

Start with a filter in DuckDB, then hand the result to the embedded backend:

```{python}
import xorq.api as xo

ddb = xo.duckdb.connect()
con = xo.connect()

penguins = xo.examples.penguins.fetch(backend=ddb)

filtered = penguins.filter(
    xo._.body_mass_g.notnull(),
    xo._.sex == "female",
)

moved = filtered.into_backend(con, "female_penguins")
```

`into_backend()` takes the target connection and an optional table name. The result is a new expression rooted in the target backend; the original is untouched.

### 3. Continue the pipeline on the new backend

Everything after the move runs on the target engine:

```{python}
result = (
    moved.group_by("species")
    .agg(
        n=xo._.count(),
        avg_mass=xo._.body_mass_g.mean(),
    )
    .execute()
)
print(result)
```

The filter executed in DuckDB; the aggregation executed on the embedded backend. The expression stays deferred until `.execute()`, so the transfer also waits until then.

### 4. Check which engines the pipeline touches

The `.ls` accessor shows the backends an expression spans:

```{python}
expr = moved.group_by("species").agg(n=xo._.count())
print(expr.ls.backends)
print(expr.ls.is_multiengine)
```

If `is_multiengine` is `True` when you expected a single engine (or the other way around), a step landed somewhere you didn't intend.

:::{.callout-tip}
Every `.execute()` re-reads the source and moves the data across again. If you run the pipeline more than once, add `.cache()` right after `.into_backend(...)` so the move only happens on the first run and later runs read the cached copy. See [Cache results by backend](cache_by_backend.qmd).
:::

## See also

- [Multi-engine execution](../concepts/understanding_xorq/multi_engine_execution.qmd): how cross-engine transfers work
- [Cache results by backend](cache_by_backend.qmd)
- [Switch between backends](../getting_started/switch_backends.qmd): a tutorial introduction to the same machinery

## How-to guides — Compose catalog entries

This guide shows you how to build on work that's already in a catalog: load an entry someone published, chain a new expression onto it, and register the result as a new entry. The catalog is a git repository, so each `add` becomes one reviewable commit.

The snippets below create a throwaway catalog in a temporary directory so the whole flow runs end to end. With a real catalog, skip the setup and point [`Catalog`](../reference/Catalog.qmd) at your repository instead.

## Prerequisites

- Xorq installed ([Install Xorq](install_xorq.qmd))
- An initialized catalog: either `xorq catalog init` on the command line or `Catalog.from_repo_path(path, init=True)` in Python
- A `pyproject.toml` in your project: `catalog.add()` packages your project as a wheel so each entry records the dependencies it was built with

## Steps

### 1. Open the catalog

```{python}
#| output: false
import tempfile
from pathlib import Path

import xorq.api as xo
from xorq.catalog.catalog import Catalog

catalog_dir = Path(tempfile.mkdtemp()) / "catalog"
catalog = Catalog.from_repo_path(catalog_dir, init=True)

# Seed the catalog with a base entry, standing in for one a teammate published
orders = xo.memtable(
    {
        "order_id": [1, 2, 3, 4],
        "region": ["EU", "US", "EU", "APAC"],
        "amount": [100.0, 250.0, 175.0, 90.0],
    },
    name="orders",
)
catalog.add(orders, aliases=("orders",), sync=False)
```

For an existing catalog, the openers are:

```python
catalog = Catalog.from_repo_path(Path("~/work/my-catalog").expanduser(), init=False)
# or, for a catalog created with `xorq catalog init`:
catalog = Catalog.from_name("my-catalog")
```

:::{.callout-note}
This demo seeds the entry with a `memtable` so the page is self-contained: the data is serialized into the entry and travels with it. Real entries are usually backed by a named table or a deferred file or SQL read, where the entry references the source rather than embedding the rows. The composition steps below work the same either way.
:::

### 2. Load the entry you want to build on

Fetch the entry by alias and materialize its expression:

```{python}
entry = catalog.get_catalog_entry("orders", maybe_alias=True)
base = entry.expr
print(base.schema())
```

`entry.expr` rebuilds the full deferred expression, including any serialized data it carries. Nothing executes yet.

### 3. Chain a new expression onto it

The loaded entry is an ordinary Xorq expression. Compose on top of it like any other table:

```{python}
summary = base.group_by("region").agg(
    total=xo._.amount.sum(),
    n_orders=xo._.count(),
)
print(summary.execute())
```

### 4. Register the result back

Add the composed expression as a new entry, with an alias so others can find it:

```{python}
#| output: false
catalog.add(summary, aliases=("orders-by-region",), sync=False)
```

`sync=False` commits locally without pushing, so you can review the diff first. Each `add` builds a wheel from your project's `pyproject.toml`; expect `Building wheel...` output, it's not an error.

### 5. Confirm the new entry exists

```{python}
print(catalog.list_aliases())
```

Both the original entry and your composition are now in the catalog. Push with `catalog.push()` (or plain `git push` from the catalog directory) when you're ready to share.

## See also

- [Catalog reference](../reference/Catalog.qmd)
- [Working with the catalog](../tutorials/core_tutorials/working_with_the_catalog.qmd): the full publish/clone/branch/merge collaboration loop
- [`xorq catalog` command-line reference](../api_reference/cli/catalog/init.qmd)

## Tutorials — Core tutorials — Get started with Xorq

Scaffold a Xorq project, load Moneyball CSVs, build a top-batters leaderboard.

## Prerequisites

[uv](https://docs.astral.sh/uv/):

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

## Scaffold

Xorq supports Python 3.10 through 3.13. This tutorial pins 3.13 so uv scaffolds and syncs against one explicit interpreter:

```bash
uv python install 3.13
uvx -p3.13 xorq@latest init --path moneyball
cd moneyball
uv sync -p3.13
source .venv/bin/activate
```

`uvx` bootstraps Xorq once. From here on, the `xorq` on your PATH comes from the project's pinned `.venv`.

::: {.callout-note}
###  Use uvx, not a permanent install
Prefer `uvx xorq@latest ...` for global Xorq commands over other install methods --- it guarantees the latest version. After creating the Xorq project, activate the uv environment and use the `xorq` in the venv from then on, so the `xorq` you invoke matches the `xorq` pinned in the pyproject.
:::

List everything the scaffold created, including dotfiles:

```bash
ls -1A
```

A complete locked project: `pyproject.toml`, `uv.lock`, an exported `requirements.txt`, a `src/` package, and a starter `expr.py`. The `with-uvenv` script and `.envrc` are helpers Xorq uses internally --- you don't need to invoke them.

## Get the data

```bash
curl -fL -o Batting.csv https://raw.githubusercontent.com/xorq-labs/baseballdatabank/master/core/Batting.csv
curl -fL -o People.csv https://raw.githubusercontent.com/xorq-labs/baseballdatabank/master/core/People.csv
```

## Write the expression

Replace `expr.py` with:

```python
# expr.py
import xorq.api as xo

# in-process backend; no server, no config
con = xo.connect()

# register CSVs as typed expressions, no read until build time
batting = xo.deferred_read_csv(con=con, path="Batting.csv", table_name="Batting")
people  = xo.deferred_read_csv(con=con, path="People.csv",  table_name="People")

# attach player bio columns to each player-season row;
# inner (the default) drops batting rows with no matching player in People
batting = batting.join(
    people["playerID", "nameFirst", "nameLast", "bats", "throws", "birthYear"],
    "playerID",
    how="inner",
)

# HBP and SF are null in older seasons, coalesce so arithmetic doesn't null-poison
batting = batting.mutate(
    HBP=batting.HBP.fill_null(0),
    SF=batting.SF.fill_null(0),
)

# modern era, AL/NL only, real hitters (>=100 AB so batting avg is meaningful)
batting = batting.filter(
    batting.lgID.isin(["AL", "NL"]),
    batting.yearID > 1965,
    batting.AB >= 100,
)

# on-base percentage: (H + BB + HBP) / (AB + BB + HBP + SF) --- computed after
# the AB >= 100 filter so the denominator is never zero
batting = batting.mutate(
    OBP=(batting.H + batting.BB + batting.HBP)
        / (batting.AB.cast("float64") + batting.BB + batting.HBP + batting.SF)
)

# batting average per player-season
ranked = batting.mutate(batting_avg=batting.H / batting.AB.cast("float64"))

# rank within each (league, year) by batting_avg, descending
win    = xo.window(group_by=["lgID", "yearID"], order_by=xo.desc("batting_avg"))
ranked = ranked.mutate(rank=xo.row_number().over(win) + 1)

# top 10 per league-year; `expr` is what `xorq build` compiles
expr = ranked.filter(ranked.rank <= 10).drop("rank")
```

`xo.deferred_read_csv` registers each CSV against any backend, sampling the file to infer its schema at build time --- the bulk read stays deferred until execution. Each binding is a typed Xorq expression you can join, filter, and aggregate without writing pandas. `expr` describes the computation. `xorq uv build` compiles it.

## Build and run

```bash
BUILD=$(xorq uv build expr.py -e expr --builds-dir builds | tail -1)
echo "$BUILD"
```

You'll see something like `builds/f22643d56d2c` --- Xorq derives the hash from the expression graph, so yours may differ.

```bash
xorq uv run "$BUILD" --output-path top_batters.parquet
```

Each build is self-contained:

```bash
ls "$BUILD"
```

`expr.yaml` is the serialized graph. `requirements.txt` pins the dependencies from your lock. Hand the directory to anyone with `xorq` and `xorq uv run` reproduces the result.

If `expr.py` changes its build changes. Edit `expr.py` and run `xorq uv build` again --- bump the year filter from 1965 to 1970:

```bash
xorq uv build expr.py -e expr --builds-dir builds
```

You'll get a new directory next to the old one:

```bash
ls -1 builds
```

Nothing is overwritten. Re-running `xorq uv build` with no changes returns the same hash, no new directory.

## Inspect

```bash
xorq uv run "$BUILD" --format csv -o /dev/stdout 2>/dev/null | head -6
```

## What you used

- `xo.deferred_read_csv` --- lazy CSV registration, schema inferred at build time
- `.join`, `.mutate`, `.filter` --- expression construction, nothing executes
- `xo.window` / `xo.row_number` / `xo.desc` --- windowing, all on the `xo` namespace
- `xorq uv build` / `xorq uv run` --- compile to `builds/<hash>/`, then execute in the project's locked env

## Next

- [Why deferred execution?](/concepts/understanding_xorq/why_deferred_execution.qmd)
- [Working with the catalog](/tutorials/core_tutorials/working_with_the_catalog.qmd)
- [`xorq uv build` reference](/api_reference/cli/uv-build.qmd)
- [`xorq uv run` reference](/api_reference/cli/uv-run.qmd)

## Tutorials — Core tutorials — Your first build

This tutorial shows you how to create builds: portable, versioned artifacts from your Xorq expressions. You'll run `xorq build`, inspect the generated files, and understand how builds work.

After completing this tutorial, you know what builds are and why they matter for versioning and deployment.

:::{.callout-warning}
### Internet connection required
This tutorial uses the iris dataset from `xo.examples.iris.fetch()`, which loads data from a remote URL. You need an active internet connection to run the build. If you're offline, you'll see a connection error like `Cannot connect to host storage.googleapis.com`.
:::

## Prerequisites

You need:

- Xorq installed: `pip install xorq`
- Basic familiarity with Xorq expressions
- Completed the [Quickstart](../../getting_started/quickstart.qmd) recommended

## What's a build?

A build captures your Xorq expression as a portable artifact. When you run `xorq build`, Xorq analyzes your code and generates files that describe the computation.

Here's why this matters: imagine you write an expression in a notebook, it works perfectly, but you can't easily version it or deploy it to production. Builds solve this by turning your expression into files you can check into git, share with teammates, and deploy anywhere Xorq runs.

:::{.callout-tip}
### Content-addressed artifacts
Builds use content hashing. Same expression = same hash, always. This gives you automatic versioning based on computational content.
:::

## Create an expression file

Start by creating a simple Python file with a Xorq expression. Create a file called `my_first_expr.py`:

```python
# my_first_expr.py
import xorq.api as xo

# <1>
con = xo.connect()
iris = xo.examples.iris.fetch(backend=con)

# <2>
filtered_iris = (
    iris
    .filter(xo._.sepal_length > 6)
    .group_by("species")
    .agg(
        count=xo._.species.count(),
        avg_width=xo._.sepal_width.mean()
    )
)
```
1. Connect to the backend and load the iris dataset.
2. Create an expression that filters, groups, and aggregates.

This expression filters iris flowers by sepal length and aggregates by species. Nothing executes yet—it's just a computation description.

## Run your first build

Now you'll build this expression using the `xorq build` command.

```bash
xorq build my_first_expr.py -e filtered_iris
```

The `-e filtered_iris` flag tells Xorq which expression variable to build from the file.

**Output:**
```
Building filtered_iris from my_first_expr.py
Written 'filtered_iris' to builds/050dac72b4d8
builds/050dac72b4d8
```

Xorq created a directory called `builds/050dac72b4d8/` containing your build artifacts. The hash `050dac72b4d8` identifies this specific build.

:::{.callout-note}
### Build hash
The hash (like `050dac72b4d8`) comes from your expression's structure. If you run the same build command again, you get the same hash. Different expression = different hash.
:::

Save this hash as an environment variable for the remaining steps:

::: {.panel-tabset}

### macOS/Linux
```bash
export BUILD_HASH=050dac72b4d8
```

Replace `050dac72b4d8` with your actual hash from the output.


### Windows (CMD)
```cmd
set BUILD_HASH=050dac72b4d8
```

Replace `050dac72b4d8` with your actual hash from the output.

:::

::: {.callout-important}
### Use your hash
The hash `050dac72b4d8` is an example. Copy the actual hash from your build output.
:::

## Inspect build artifacts

Now inspect what Xorq generated. List the build directory contents:

::: {.panel-tabset}

### macOS/Linux
```bash
ls -la builds/$BUILD_HASH/
```


### Windows (CMD)
```cmd
dir builds\%BUILD_HASH%\
```

:::

You see several files:

```
builds/050dac72b4d8/
├── expr.yaml
├── metadata.json
└── profiles.yaml
```

The key artifacts are:
- **expr.yaml**: Your expression as a declarative manifest
- **manifest.json**: Build metadata and structure
- **profiles.yaml**: Backend connection information

## Understand the manifest

Open `expr.yaml` in a text editor to see how Xorq serialized your expression.

The manifest describes your expression as a tree of operations:

```yaml
# Simplified expr.yaml structure
definitions:
  schemas:
    schema_0:
      species: String
      count: Int64
      avg_width: Float64
  
  nodes:
    filter_node:
      op: Filter
      predicates:
        - sepal_length > 6
      parent:
        op: Read
        name: iris
    
    species_field:
      op: Field
      name: species
      relation: filter_node

expression:
  op: Aggregate
  parent: filter_node
  by:
    species: species_field
  metrics:
    count: Count(species)
    avg_width: Mean(sepal_width)
```

The manifest defines schemas and nodes in the `definitions` section, then references them in the `expression` section. Each node specifies an operation (like `Filter` or `Aggregate`) and connects to other nodes. This declarative structure means you can run the manifest on any system with Xorq.


## See content-addressed hashing

The build hash comes from the manifest content. You'll verify this by rebuilding the same expression.

Run the build command again:

```bash
xorq build my_first_expr.py -e filtered_iris
```

**Output:**
```
Building filtered_iris from my_first_expr.py
Written 'filtered_iris' to builds/050dac72b4d8
builds/050dac72b4d8
```

You get the same hash: `050dac72b4d8`. Same expression = same hash.

Now modify the filter threshold in `my_first_expr.py`:

```python
# Change from > 6 to > 6.5
filtered_iris = (
    iris
    .filter(xo._.sepal_length > 6.5)  # Changed!
    .group_by("species")
    .agg(
        count=xo._.species.count(),
        avg_width=xo._.sepal_width.mean()
    )
)
```

Build again:

```bash
xorq build my_first_expr.py -e filtered_iris
```

**Output:**
```
Building filtered_iris from my_first_expr.py
Written 'filtered_iris' to builds/8a2f1c5e9d3b
builds/8a2f1c5e9d3b
```

You get a different hash: `8a2f1c5e9d3b`. Different expression = different hash. This content-addressed hashing means builds are automatically versioned by their computational content.

Save the new hash as an environment variable for the remaining steps:

::: {.panel-tabset}

### macOS/Linux
```bash
export BUILD_HASH=8a2f1c5e9d3b
```

Replace `8a2f1c5e9d3b` with your actual hash from the output.


### Windows (CMD)
```cmd
set BUILD_HASH=8a2f1c5e9d3b
```

Replace `8a2f1c5e9d3b` with your actual hash from the output.

:::


## Run a build

Once you have a build, you can execute it with `xorq run`:

::: {.callout-warning}
### Windows users
Skip the first tab and go straight to "Save to file." Running without an output file might cause issues on Windows.
:::

::: {.panel-tabset}

### View results (macOS/Linux)
```bash
xorq run builds/$BUILD_HASH
```

Runs the build and displays results in your terminal.

### Save to file (macOS/Linux)
```bash
xorq run builds/$BUILD_HASH -o results.parquet
```

Runs the build and saves results to `results.parquet`.


### Save to file (Windows CMD)
```cmd
xorq run builds\%BUILD_HASH% -o results.parquet
```

Runs the build and saves results to `results.parquet`.

:::

The build executes without needing the original Python file. The manifest contains everything Xorq needs to run the computation.

:::{.callout-note}
### Portable execution
You can copy the build directory to another machine, and `xorq run` works there too (assuming data sources are accessible).
:::

## Add to the catalog

Builds become more discoverable when you add them to the catalog. The catalog creates entry IDs for your builds and lets you reference them with human-readable aliases. 

```bash
xorq catalog add builds/$BUILD_HASH --alias my-iris-analysis
```

**Output:**
```
Added build 050dac72b4d8 as entry ce9fe1e5-0004-4087-b668-f67dfbdea6ba revision r1
```

Now you can reference this build by name instead of hash. List catalog entries:

```bash
xorq catalog list
```

**Output:**
```
Aliases:
my-iris-analysis        ce9fe1e5-0004-4087-b668-f67dfbdea6ba    r1
Entries:
ce9fe1e5-0004-4087-b668-f67dfbdea6ba    r1      050dac72b4d8
```

The catalog shows:
- **Aliases**: Human-readable names pointing to entries
- **Entries**: Entry IDs with revision numbers and build hashes


## Build multi-step pipelines

You'll create a more complex build with multiple operations. Create `pipeline.py`:

```python
# pipeline.py
import xorq.api as xo

# <1>
con = xo.connect()
iris = xo.examples.iris.fetch(backend=con)

# <2>
filtered = iris.filter(xo._.sepal_length > 5)

# <3>
with_ratio = filtered.mutate(
    ratio=xo._.sepal_length / xo._.sepal_width
)

# <4>
summary = (
    with_ratio
    .group_by("species")
    .agg(
        avg_ratio=xo._.ratio.mean(),
        count=xo._.species.count()
    )
)
```
1. Connect and load data.
2. Filter rows.
3. Add a calculated column (ratio).
4. Group and aggregate.

Build the final expression:

::: {.panel-tabset}

### macOS/Linux/Windows
```bash
xorq build pipeline.py -e summary
```
:::

The manifest captures the entire pipeline (filter → mutate → aggregate) as a single versioned artifact. When you specify `-e summary`, Xorq includes all expressions that `summary` depends on.

:::{.callout-tip}
### Build scope
Builds capture the expression you specify with `-e` and all its dependencies automatically. You don't need to manually track what goes into a build.
:::

## Complete workflow

Here's the full process from code to cataloged build:

```bash
# 1. Create expression file
cat > analysis.py << 'EOF'
import xorq.api as xo

con = xo.connect()
data = xo.examples.iris.fetch(backend=con)

result = (
    data
    .filter(xo._.sepal_length > 6)
    .group_by("species")
    .agg(avg=xo._.sepal_width.mean())
)
EOF

# 2. Build the expression
xorq build analysis.py -e result
# Output: builds/a1b2c3d4e5f6

# 3. Save the hash
export BUILD_HASH=a1b2c3d4e5f6

# 4. Run the build
xorq run builds/$BUILD_HASH -o output.parquet

# 5. Add to catalog
xorq catalog add builds/$BUILD_HASH --alias iris-analysis
```

## What you learned

You've created your first builds with Xorq. Here's what you accomplished:

- Built expressions into portable artifacts with `xorq build`
- Inspected generated manifests (expr.yaml, manifest.json)
- Understood content-addressed hashing for automatic versioning
- Ran builds without original Python code using `xorq run`
- Added builds to the catalog with human-readable aliases
- Built multi-step pipelines as single artifacts

The key insight? Builds turn expressions into versioned, portable artifacts. Same expression = same hash = same build. This makes it easy to version computations, share them with teammates, and deploy them to production.

## Next steps

Now you understand builds. Continue with domain-specific tutorials:

- [Train your first model](../ml_tutorials/train_your_first_model.qmd) shows you how to build ML pipelines with Xorq
- [Understand deferred execution](../../getting_started/understand_deferred_execution.qmd) explains how Xorq builds computation graphs

## Tutorials — Core tutorials — Build a semantic catalog

This tutorial shows you how to build a semantic model over an FAA-style flights dataset using the **Boring Semantic Layer (BSL)**, query it through Xorq, and store it in the catalog so anyone (or any future build) can recover the model and issue new queries against it.

After completing this tutorial, you understand how to define dimensions and measures with BSL, query them through Xorq, and round-trip the model through the catalog so a downstream consumer can recover it and ask their own questions.

## Prerequisites

You need:

- [`uv`](https://docs.astral.sh/uv/) installed (Xorq's recommended runner)
- Basic familiarity with Xorq expressions ([Your first expression](../../getting_started/your_first_expression.qmd) recommended)
- Completed [Your first build](your_first_build.qmd) so the catalog vocabulary is familiar

## Set up a project directory

`catalog.add(...)` needs a `pyproject.toml` in the working directory (or an ancestor) so it can pin the Xorq version embedded in the entry. Create a fresh project and pull in Xorq with the BSL extra:

```bash
mkdir flights-tutorial && cd flights-tutorial
uv init --bare
uv add "xorq[bsl]"
printf '\n[tool.setuptools]\npy-modules = []\n' >> pyproject.toml
```

The rest of the tutorial assumes commands are run from inside `flights-tutorial/`.

::: {.callout-note}
### Why `--bare`?
Plain `uv init` drops a sample `main.py` next to your `pyproject.toml` and runs `git init`. Both bite you later: setuptools' auto-discovery sees `main.py` + `flights_catalog.py` and refuses to build a wheel for `catalog.add(...)` (multiple top-level modules), and the empty git repo (no `HEAD`) makes Xorq's import-time git probe write `fatal: ambiguous argument 'HEAD'` to stderr on every run. `--bare` skips both—only `pyproject.toml` is created.
:::

::: {.callout-note}
### Why the `py-modules = []` line?
`catalog.add(...)` builds a wheel of your project to embed in the catalog entry as a dep-pinning artifact. With no `[tool.setuptools]` config, setuptools auto-discovers top-level `.py` modules and refuses to build the wheel as soon as it finds more than one—and this tutorial gives you two (`flights_catalog.py` and `recover_flights.py`). `py-modules = []` tells setuptools "no modules in the wheel," so the wheel builds empty. That's fine here—Xorq only needs the wheel for its dependency metadata, not to redistribute your scripts.
:::

::: {.callout-note}
### Activate the project venv
`uv add` set up a `.venv` in `flights-tutorial/`. Activate it once in your shell so plain `python` uses it:

```bash
source .venv/bin/activate
```

The rest of the tutorial assumes the venv is active.
:::

::: {.callout-note}
### Why a project directory?
If you run the script from a directory without a `pyproject.toml`, `catalog.add(...)` raises `cannot locate a pyproject.toml ...`. `uv init` creates one for you; if you're not using uv, an empty `pyproject.toml` next to your script is enough—or pass `project_path=` to `catalog.add(...)` explicitly.
:::

::: {.callout-tip}
### What's BSL?
The [Boring Semantic Layer](https://github.com/boringdata/boring-semantic-layer) is a small, declarative semantic layer that lets you attach **dimensions** (groupings) and **measures** (aggregations) to a table once, then issue many different queries without repeating the SQL or the Python. Xorq integrates with BSL so a `SemanticModel` can be stamped onto an expression and stored in the catalog.
:::

## What you'll build

A reusable semantic model over flights data with:

- Three dimensions: `origin`, `destination`, `carrier`
- Three measures: `flight_count`, `avg_dep_delay`, `total_distance`

You'll query it two different ways, then catalog the model so a colleague can pull it down and ask their own questions—without ever seeing your original Python file.

## Create the flights dataset

Start with a small FAA-style flights table. The columns mirror what you'd find in the FAA On-Time Performance dataset (or `nycflights13`): an origin airport, a destination airport, a carrier code, departure delay in minutes, and route distance in miles.

Create a file called `flights_catalog.py`:

```python
# flights_catalog.py
import xorq.api as xo

# <1>
flights = xo.memtable(
    {
        "origin":      ["JFK", "LAX", "ORD", "JFK", "LAX", "ORD", "JFK", "LAX"],
        "destination": ["LAX", "ORD", "JFK", "ORD", "JFK", "LAX", "LAX", "JFK"],
        "carrier":     ["AA",  "UA",  "AA",  "UA",  "AA",  "UA",  "AA",  "UA"],
        "dep_delay":   [10.0, -5.0,  30.0,  15.0, -2.0,  45.0,   5.0,  20.0],
        "distance":    [2475, 1745,   740,  1300, 2475,  1745,  2475,  2475],
    },
    name="flights",
)
```
1. `xo.memtable` builds a deferred Xorq table from inline data. Nothing executes yet—`flights` is an expression you can pass to BSL.

::: {.callout-note}
### Why an in-memory table?
For the tutorial the data stays inline so you can run the whole thing offline. In production, swap `xo.memtable(...)` for a real source—`con.read_parquet(...)`, a Postgres table, or a pinned dataset. The semantic model on top doesn't change.
:::

## Define the semantic model

A BSL `SemanticModel` is a table plus a vocabulary of dimensions and measures. You build it by wrapping the Xorq expression with `to_semantic_table` and chaining `.with_dimensions(...)` and `.with_measures(...)`.

Append to `flights_catalog.py`:

```python
from boring_semantic_layer import to_semantic_table

# <1>
flights_model = (
    to_semantic_table(flights)
    .with_dimensions(
        origin=lambda t: t.origin,
        destination=lambda t: t.destination,
        carrier=lambda t: t.carrier,
    )
    .with_measures(
        flight_count=lambda t: t.count(),
        avg_dep_delay=lambda t: t.dep_delay.mean(),
        total_distance=lambda t: t.distance.sum(),
    )
)

print("Dimensions:", tuple(flights_model.dimensions))
print("Measures:  ", tuple(flights_model.measures))
```
1. Each dimension and measure is a lambda that takes the table and returns an expression. BSL stores the lambda—it doesn't run it until you query.

::: {.callout-tip}
### Dimensions vs. measures
A **dimension** is a column you can group by (or filter on). A **measure** is an aggregation: counts, means, sums, anything that collapses rows. The split is what lets BSL turn `query(dimensions=..., measures=...)` into the right `group_by(...).agg(...)` for you.
:::

::: {.callout-tip}
### Run as you go
`flights_catalog.py` grows section by section through the rest of the tutorial. After each addition, run `python flights_catalog.py` from your project directory—the output shown beneath each block (dimensions, measures, query results) is what you'll see when you do. The complete script is consolidated at the end in [Putting it all together](#putting-it-all-together).
:::

## Query the model

A model is useless until you ask it questions. `flights_model.query(...)` returns a regular Xorq expression—the same kind you'd get from `table.group_by(...).agg(...)`—so `.execute()` runs it on your backend.

Add a first query: average departure delay by origin airport.

```python
by_origin = flights_model.query(
    dimensions=("origin",),
    measures=("flight_count", "avg_dep_delay"),
).order_by("origin")

print(by_origin.execute())
```

::: {.callout-note}
### The same query, without BSL
Because `query(...)` lowers to ordinary Xorq, the equivalent without the semantic layer is just `group_by` + `agg`:

```python
by_origin_plain = flights.group_by("origin").agg(
    flight_count=flights.count(),
    avg_dep_delay=flights.dep_delay.mean(),
)
```

`by_origin_plain.execute()` returns the same DataFrame. The semantic-layer version pays off as soon as you have a *second* query: `dep_delay.mean()` doesn't have to be re-typed (or kept in sync between callers), and consumers ask for `avg_dep_delay` by name without knowing how it's computed.
:::

You see one row per origin airport, with the count and average delay:

```
  origin  flight_count  avg_dep_delay
0    JFK             3      10.000000
1    LAX             3       4.333333
2    ORD             2      37.500000
```

::: {.callout-note}
### Row order
BSL doesn't sort the result—the row order you see depends on the backend's hash layout. The preceding `.order_by("origin")` is what makes this output reproducible; without it, the rows can come back in any order. Every `query(...)` block below this point is sorted for the same reason.
:::

Now ask a different question—total distance flown by each carrier:

```python
by_carrier = flights_model.query(
    dimensions=("carrier",),
    measures=("flight_count", "total_distance"),
).order_by("carrier")

print(by_carrier.execute())
```

```
  carrier  flight_count  total_distance
0      AA             4            8165
1      UA             4            7265
```

Notice: same model, two completely different queries. The model is the contract—`query(...)` is the conversation.

::: {.callout-tip}
### What if you ask for something that doesn't exist?
Try requesting a dimension or measure the model never registered:

```python
flights_model.query(dimensions=("airport",), measures=("flight_count",)).execute()
```

You get an error immediately, not silently wrong results:

```
XorqTypeError: Column 'airport' is not found in table. Existing columns:
'origin', 'destination', 'carrier', 'dep_delay', 'distance'.
```

This is the second payoff of the semantic layer. Dimensions and measures are a closed vocabulary: `airport` doesn't exist, so the query fails before it touches the data. Without the model, a typo in `group_by("airport")` would give the same error—but a typo in a hand-written measure (say, `dep_delay.mean()` vs. `dep_delay.sum()`) wouldn't fail at all; it would just return a quietly wrong number. By naming the aggregation `avg_dep_delay` once, on the model, every caller gets the right one or none at all.
:::

## Catalog the model

To preserve the model itself—not just one query result—turn it into a Xorq expression that carries the BSL metadata via `to_tagged(flights_model)`, then add that expression to the catalog. The catalog is git-backed, so the directory you point it at becomes a versioned store of every entry you add.

```python
from pathlib import Path

from boring_semantic_layer import to_tagged
from xorq.catalog.catalog import Catalog

# <1>
flights_model_expr = to_tagged(flights_model)

# <2>
catalog_dir = Path("flights-catalog")
catalog = Catalog.from_repo_path(catalog_dir, init=True)

# <3>
catalog.add(flights_model_expr, aliases=("flights-model",), sync=False)

print("Catalog at:", catalog_dir.absolute())
print("Aliases:   ", catalog.list_aliases())
```
1. `to_tagged(flights_model)` serializes the dimensions, measures, and underlying table into a Xorq expression with BSL metadata attached. Bind it to `flights_model_expr` to make the role explicit: it's the *expression form* of the model, ready for the catalog. You're cataloging the model itself, not the result of one of its queries.
2. A stable path inside the project directory. Use anywhere you like—but a real folder (not a temp dir) is what lets the *next* script find the catalog by path. `Catalog.from_repo_path(..., init=True)` initializes a fresh git repo there.
3. The alias `flights-model` is the human-readable handle. Internally each entry has a content-addressed hash; the alias just points at it.

::: {.callout-note}
### Why go through the catalog?
The catalog speaks the language of expressions: schemas, lineage, content hashes, deferred reads. By tagging the model and storing it as a catalog entry, the model travels through the same pipes as everything else—`xorq build`, `xorq run`, lineage tools—and `from_tagged` lets you get the rich Python object back when you need it.
:::

::: {.callout-tip}
### Pointing at a real catalog
For team use, replace the temp directory with a path to a checked-out git repo, push commits with `sync=True`, and your colleagues can clone it. Aliases survive across machines because they live in git.
:::

## Recover the model from a separate script

Here's the payoff. The catalog is now persisted at `./flights-catalog/`—a regular git directory you could commit, push, share, or back up. Switch hats: imagine you're a different person on the team. You have access to that directory, but you've never seen `flights_catalog.py` and don't know how the model was built. All you have is the alias.

Create a *new* file alongside `flights_catalog.py`, called `recover_flights.py`:

```python
# recover_flights.py
from pathlib import Path

from boring_semantic_layer import from_tagged
from xorq.catalog.catalog import Catalog

# <1>
catalog = Catalog.from_repo_path(Path("flights-catalog"), init=False)

# <2>
flights_entry = catalog.get_catalog_entry("flights-model", maybe_alias=True)
flights_model = from_tagged(flights_entry.expr)

print("Recovered type:    ", type(flights_model).__name__)
print("Recovered dims:    ", tuple(flights_model.dimensions))
print("Recovered measures:", tuple(flights_model.measures))

# <3>
by_destination = flights_model.query(
    dimensions=("destination",),
    measures=("flight_count", "total_distance"),
).order_by("destination")

print(by_destination.execute())
```
1. `init=False` opens the existing catalog at the path. Note what's *not* imported: nothing from `flights_catalog.py`, no `to_semantic_table`, no inline `flights` data. The catalog directory is the only handoff.
2. `flights_entry` is the catalog handle—content hash, alias, sidecar metadata, and the cataloged expression on `.expr`. `from_tagged(...)` reads the BSL metadata off that expression and reconstructs a live `SemanticModel`: same dimensions, same measures, same underlying table.
3. A brand-new query that the original Python file never even mentioned. The model's vocabulary is enough.

Run it:

```bash
python recover_flights.py
```

```
Recovered type:     SemanticModel
Recovered dims:     ('origin', 'destination', 'carrier')
Recovered measures: ('flight_count', 'avg_dep_delay', 'total_distance')
  destination  flight_count  total_distance
0         JFK             3            5690
1         LAX             3            6695
2         ORD             2            3045
```

This is the property that makes the catalog interesting: you stored the model, and any consumer with access to the catalog directory can ask anything the model's dimensions and measures are designed to answer—without seeing or running your original code.

## Putting it all together

Two scripts, one shared catalog directory.

`flights_catalog.py`—defines the model, queries it, publishes it:

```python
# flights_catalog.py
from pathlib import Path

from boring_semantic_layer import to_semantic_table, to_tagged

import xorq.api as xo
from xorq.catalog.catalog import Catalog


# 1. Source table
flights = xo.memtable(
    {
        "origin":      ["JFK", "LAX", "ORD", "JFK", "LAX", "ORD", "JFK", "LAX"],
        "destination": ["LAX", "ORD", "JFK", "ORD", "JFK", "LAX", "LAX", "JFK"],
        "carrier":     ["AA",  "UA",  "AA",  "UA",  "AA",  "UA",  "AA",  "UA"],
        "dep_delay":   [10.0, -5.0,  30.0,  15.0, -2.0,  45.0,   5.0,  20.0],
        "distance":    [2475, 1745,   740,  1300, 2475,  1745,  2475,  2475],
    },
    name="flights",
)

# 2. Semantic model
flights_model = (
    to_semantic_table(flights)
    .with_dimensions(
        origin=lambda t: t.origin,
        destination=lambda t: t.destination,
        carrier=lambda t: t.carrier,
    )
    .with_measures(
        flight_count=lambda t: t.count(),
        avg_dep_delay=lambda t: t.dep_delay.mean(),
        total_distance=lambda t: t.distance.sum(),
    )
)

# 3. Query
print(flights_model.query(
    dimensions=("origin",),
    measures=("flight_count", "avg_dep_delay"),
).order_by("origin").execute())

print(flights_model.query(
    dimensions=("carrier",),
    measures=("flight_count", "total_distance"),
).order_by("carrier").execute())

# 4. Tag the model and add it to the catalog
flights_model_expr = to_tagged(flights_model)

catalog_dir = Path("flights-catalog")
catalog = Catalog.from_repo_path(catalog_dir, init=True)
catalog.add(flights_model_expr, aliases=("flights-model",), sync=False)
```

`recover_flights.py`—reads the catalog from scratch, recovers the model, runs a new query:

```python
# recover_flights.py
from pathlib import Path

from boring_semantic_layer import from_tagged
from xorq.catalog.catalog import Catalog


catalog = Catalog.from_repo_path(Path("flights-catalog"), init=False)
flights_entry = catalog.get_catalog_entry("flights-model", maybe_alias=True)
flights_model = from_tagged(flights_entry.expr)

print(flights_model.query(
    dimensions=("destination",),
    measures=("flight_count", "total_distance"),
).order_by("destination").execute())
```

Run them:

```bash
python flights_catalog.py
python recover_flights.py
```

## What you learned

- The Boring Semantic Layer turns a Xorq table into a `SemanticModel` with named dimensions and measures.
- `flights_model.query(dimensions=..., measures=...)` produces an ordinary Xorq expression, so `.execute()` runs on any Xorq backend.
- Asking for a dimension or measure the model didn't register raises an error before any data is touched—typos in measure definitions can't return quietly wrong numbers.
- `to_tagged(flights_model)` produces a catalog-ready expression, and `from_tagged(flights_entry.expr)` reconstructs the live `SemanticModel` on the other side.

The point of the BSL + catalog combination is decoupling: the team that owns the data publishes the model once, and every downstream user gets a typed, queryable object instead of a frozen result set.

## Next steps

- [Your first build](your_first_build.qmd)—package the cataloged model into a portable build artifact.
- [Explore caching](../../getting_started/explore_caching.qmd)—cache the underlying table so repeated BSL queries don't re-read the source.
- [Understand deferred execution](../../getting_started/understand_deferred_execution.qmd)—the foundation that makes `query(...)` cheap and composable.

## Tutorials — Core tutorials — Working with the catalog

In [Build a semantic catalog](build_a_semantic_catalog.qmd) you tagged a BSL flights model and dropped it into a local catalog. That catalog is a regular git repository—every entry, alias, and revision is a git commit. The moment you push it to a remote, anyone with access can clone it, query the model, propose changes, and run the model against their own backend.

This tutorial walks you through the collaboration loop end-to-end, driven from the `xorq` command line (`xorq uv build`, `xorq catalog ...`) rather than the Python API. You'll play both sides:

- **User A** publishes the catalog to GitHub.
- **User B** clones it, recovers the model, and proposes a new entry via pull request.
- **User A** reviews, merges, and pulls the changes back.

You'll also see how to swap the connection profile at recovery time, so a downstream user can run the cataloged expression against their own backend without modifying the entry.

::: {.callout-note}
### Why the command line here
The foundation tutorial cataloged the model with the Python API (`catalog.add(expr)`). This tutorial does the same job from the shell: `xorq uv build` packages a script into a build artifact, and `xorq catalog add <build-dir>` files that artifact into the catalog. The build artifact carries a dependency-pinned wheel of your project, so the command-line flow is what you'd reach for in CI or a Makefile—no Python entry point to maintain. The two Python-only steps that remain (recovering the BSL model, swapping the profile) are called out where they appear.
:::

## Prerequisites

You need:

- Completed [Build a semantic catalog](build_a_semantic_catalog.qmd)—same model definition, same project layout. This tutorial recreates the catalog at a stable path here, since the foundation tutorial used a temp directory.
- User A's `flights-tutorial/` project directory from the foundation tutorial. This tutorial assumes it lives at `~/flights-tutorial/` so it sits as a sibling of User B's `~/flights-tutorial-userb/` (created later in the tutorial). If you put it somewhere else, either move it now (`mv path/to/flights-tutorial ~/flights-tutorial`) or substitute your actual path wherever you see `~/flights-tutorial` below. (User B gets their own project; you'll create it later in the tutorial.)
- The `sqlite` extra installed in that project. The foundation tutorial installed `xorq[bsl,duckdb]`; add `sqlite` here to demonstrate the profile-swap section against a different backend than the one User A built the entry with. From inside `~/flights-tutorial/`:
  ```bash
  uv add "xorq[bsl,duckdb,sqlite]"
  ```
- Git installed locally and authenticated with GitHub (the `gh` command-line tool is convenient but not required).

::: {.callout-note}
### A catalog is a git repo
Xorq's catalog stores entries as files in a git repository: `catalog.yaml` is the index, `aliases/` holds alias pointers, `entries/` holds the entry zips themselves, and `metadata/` holds a sidecar yaml per entry. Every `xorq catalog add` is one git commit. That means GitHub's permission model, branch protection rules, and pull requests Just Work—there's no separate object store, no extra service to provision.
:::

## Build the model into an artifact (User A)

The command line catalogs *build artifacts*, not live Python objects, so the first step is to produce one. Save the model definition below as `flights_model.py` in User A's `~/flights-tutorial/` project. It's the same model as the foundation tutorial, trimmed to just the definition and a final `to_tagged(...)` assignment to `expr`, the variable `xorq uv build` looks for:

```python
# flights_model.py
from boring_semantic_layer import to_semantic_table, to_tagged
import xorq.api as xo

flights = xo.memtable(
    {
        "origin":      ["JFK", "LAX", "ORD", "JFK", "LAX", "ORD", "JFK", "LAX"],
        "destination": ["LAX", "ORD", "JFK", "ORD", "JFK", "LAX", "LAX", "JFK"],
        "carrier":     ["AA",  "UA",  "AA",  "UA",  "AA",  "UA",  "AA",  "UA"],
        "dep_delay":   [10.0, -5.0,  30.0,  15.0, -2.0,  45.0,   5.0,  20.0],
        "distance":    [2475, 1745,   740,  1300, 2475,  1745,  2475,  2475],
    },
    name="flights",
)
flights_model = (
    to_semantic_table(flights)
    .with_dimensions(
        origin=lambda t: t.origin,
        destination=lambda t: t.destination,
        carrier=lambda t: t.carrier,
    )
    .with_measures(
        flight_count=lambda t: t.count(),
        avg_dep_delay=lambda t: t.dep_delay.mean(),
        total_distance=lambda t: t.distance.sum(),
    )
)
expr = to_tagged(flights_model)
```

Build it from inside the project directory. `-e expr` names the variable to build; `xorq uv build` resolves the project root by searching upward for `pyproject.toml`, so the cwd matters:

```bash
cd ~/flights-tutorial
uv run xorq uv build flights_model.py -e expr
# prints the build directory, e.g. builds/a1b2c3d4e5f6
```

The command prints the build directory it created (under `builds/`) as the last line—you'll pass that path to `catalog add` next. To capture it in a shell variable instead of copying by hand:

```bash
BUILD_A=$(cd ~/flights-tutorial && uv run xorq uv build flights_model.py -e expr | tail -1)
```

::: {.callout-note}
### Expect wheel-build output on `xorq uv build`
`xorq uv build` packages your project as a wheel and stores it inside the build artifact, so the cataloged expression keeps a frozen record of the dependencies it was built against. You'll see `Building wheel...`, `running egg_info`, and `Successfully built ...whl` in the output, plus a `UserWarning` about local filesystem paths from the inline `memtable`—both are expected and not errors. If OTel console export pollutes stdout and breaks the `tail -1` capture, pass `--emit-build-path-to build_path.txt` and read the path from that file.
:::

## Publish the catalog to GitHub (User A)

::: {.callout-tip}
### Follow along without GitHub
You don't need a GitHub account to work through this tutorial—a local bare git repo behaves the same way for `clone`/`push`/`pull`. Each remote-using step below has a **Local bare repo** tab next to the GitHub one; pick one and stay on it.
:::

Initialize the catalog at a stable path (`~/work/flights-catalog-usera`) so the rest of the tutorial has somewhere persistent to point at, then add the build artifact under the `flights-model` alias. `init` creates the catalog's leaf directory but not its parent, so make sure `~/work/` exists first:

```bash
mkdir -p ~/work
cd ~/flights-tutorial
uv run xorq catalog --path ~/work/flights-catalog-usera init
uv run xorq catalog --path ~/work/flights-catalog-usera add "$BUILD_A" -a flights-model
uv run xorq catalog --path ~/work/flights-catalog-usera list-aliases
# flights-model
```

(`--path` is a group option, so it comes *before* the subcommand. If you didn't capture `$BUILD_A`, substitute the `builds/<hash>` path the build step printed.)

Wire up a remote and push. `xorq catalog set-remote` adds the git remote; the first push still needs raw `git push -u` to set upstream tracking, which `xorq catalog push` doesn't do:

::: {.panel-tabset}

### GitHub
Create an empty repository on GitHub (web UI: "New repository" → leave empty), then point the catalog at it and push:

```bash
uv run xorq catalog --path ~/work/flights-catalog-usera set-remote https://github.com/<you>/flights-catalog.git
git -C ~/work/flights-catalog-usera push -u origin main
```

Or, equivalently, with the `gh` command-line tool from inside the catalog directory—`--source=.` means "create the repo from this working directory," so the `cd` matters. `gh` adds the `origin` remote and pushes for you, so you don't need `set-remote`:

```bash
cd ~/work/flights-catalog-usera
gh repo create <you>/flights-catalog --public --source=. --remote=origin --push
```

### Local bare repo
Initialize a local bare repo to act as the "remote"—same git semantics, no GitHub account needed:

```bash
git init --bare ~/work/flights-catalog-remote.git
```

Then point the catalog at it and push:

```bash
uv run xorq catalog --path ~/work/flights-catalog-usera set-remote "file://$HOME/work/flights-catalog-remote.git"
git -C ~/work/flights-catalog-usera push -u origin main
```

:::

Every subsequent publish can use `xorq catalog push`, which runs `git push` against the configured remote:

```bash
uv run xorq catalog --path ~/work/flights-catalog-usera push
```

Verify the remote sees what you expect:

::: {.panel-tabset}

### GitHub
```bash
gh repo view --web   # opens the repo on GitHub
```

You should see `catalog.yaml` (the index) at the top level, plus `aliases/`, `entries/`, and `metadata/` directories. Click into `aliases/` and you'll see `flights-model.zip`—the alias you just added.

### Local bare repo
A bare repo has no working tree to `ls`, so list its files at `main` directly:

```bash
git -C ~/work/flights-catalog-remote.git ls-tree -r main
```

You should see `catalog.yaml`, `aliases/flights-model.zip`, an `entries/<hash>.zip`, and a matching `metadata/<hash>.zip.metadata.yaml`—the same files that would appear under "Files" on GitHub.

:::

## Set up User B's project

Switch hats. User B is on a different machine (or pretending to be—same laptop, different working directory and venv). Give them their own uv project so they aren't sharing User A's `pyproject.toml` or `.venv`:

```bash
mkdir ~/flights-tutorial-userb && cd ~/flights-tutorial-userb
uv init --bare
uv add "xorq[bsl,duckdb,sqlite]"
printf '\n[tool.setuptools]\npy-modules = []\n' >> pyproject.toml
```

This is the same setup the foundation tutorial walked you through for User A, just under a different directory name. From here on, every User B command runs with `uv run xorq ...` from inside `~/flights-tutorial-userb/`. The `uv run` command picks up that project's `.venv` automatically, so you never have to deactivate User A's venv to run User B's code.

::: {.callout-tip}
### Two projects, no venv switching
With User A's `flights-tutorial/` and User B's `flights-tutorial-userb/`, you have two `.venv`s on disk. `uv run xorq ...` looks at the `pyproject.toml` of the directory you're in and uses *that* venv—so as long as you `cd` into the right project before each command, the right venv is used. No `source`, no `deactivate`.
:::

## Clone the catalog (User B)

User B clones the catalog with one command:

::: {.panel-tabset}

### GitHub
```bash
cd ~/flights-tutorial-userb
uv run xorq catalog clone https://github.com/<you>/flights-catalog.git --path ~/work/flights-catalog-userb
uv run xorq catalog --path ~/work/flights-catalog-userb list-aliases
# flights-model
```

### Local bare repo
```bash
cd ~/flights-tutorial-userb
uv run xorq catalog clone "file://$HOME/work/flights-catalog-remote.git" --path ~/work/flights-catalog-userb
uv run xorq catalog --path ~/work/flights-catalog-userb list-aliases
# flights-model
```

:::

User B never saw User A's `flights_model.py`, never saw the original `to_semantic_table(...)` call. All they have is a git clone—and that's enough.

## Recover and query the model (User B)

Recovering the BSL `SemanticModel` so you can call `.query(...)` is a Python step. `from_tagged` rebuilds the semantic layer from the cataloged expression. Because the catalog is plain git, the entry contents arrived during `clone`, so `from_tagged` can read them immediately. Save the snippet below as `recover_model.py` in `~/flights-tutorial-userb/` and run it with `uv run python recover_model.py`:

```python
# recover_model.py
from pathlib import Path

from boring_semantic_layer import from_tagged
from xorq.catalog.catalog import Catalog

catalog = Catalog.from_repo_path(Path("~/work/flights-catalog-userb").expanduser(), init=False)

flights_entry = catalog.get_catalog_entry("flights-model", maybe_alias=True)
flights_model = from_tagged(flights_entry.expr)

print(
    flights_model.query(
        dimensions=("origin",),
        measures=("flight_count", "avg_dep_delay"),
    ).order_by("origin").execute()
)
```

The recovered `SemanticModel` has the same dimensions and measures User A defined. The data User A used (the inline `xo.memtable(...)`) is serialized inside the entry, so the query runs locally—no shared filesystem, no out-of-band data transfer.

## Propose a change via pull request (User B)

User B wants to publish a refined view: same model, but filtered to American Airlines only. Because the catalog is a git repo, they branch, commit, push, and open a PR—exactly like any other code change.

```bash
git -C ~/work/flights-catalog-userb checkout -b add-aa-only-model
```

Build the new entry the same way User A built the first one. User B has the same flights data—it's the inline `memtable`—so they reconstruct it in a script. Save the snippet below as `aa_flights_model.py` in `~/flights-tutorial-userb/`:

```python
# aa_flights_model.py
from boring_semantic_layer import to_semantic_table, to_tagged
import xorq.api as xo

flights = xo.memtable(
    {
        "origin":      ["JFK", "LAX", "ORD", "JFK", "LAX", "ORD", "JFK", "LAX"],
        "destination": ["LAX", "ORD", "JFK", "ORD", "JFK", "LAX", "LAX", "JFK"],
        "carrier":     ["AA",  "UA",  "AA",  "UA",  "AA",  "UA",  "AA",  "UA"],
        "dep_delay":   [10.0, -5.0,  30.0,  15.0, -2.0,  45.0,   5.0,  20.0],
        "distance":    [2475, 1745,   740,  1300, 2475,  1745,  2475,  2475],
    },
    name="flights",
)

# Same model shape, restricted to AA
aa_flights = flights.filter(flights.carrier == "AA")
aa_model = (
    to_semantic_table(aa_flights)
    .with_dimensions(
        origin=lambda t: t.origin,
        destination=lambda t: t.destination,
    )
    .with_measures(
        flight_count=lambda t: t.count(),
        avg_dep_delay=lambda t: t.dep_delay.mean(),
    )
)
expr = to_tagged(aa_model)
```

Build it from User B's project, then add it under a new alias with `--no-sync`:

```bash
cd ~/flights-tutorial-userb
BUILD_B=$(uv run xorq uv build aa_flights_model.py -e expr | tail -1)
uv run xorq catalog --path ~/work/flights-catalog-userb add "$BUILD_B" -a flights-aa-only --no-sync
uv run xorq catalog --path ~/work/flights-catalog-userb list-aliases
# flights-aa-only
# flights-model
```

That commits a new entry on the `add-aa-only-model` branch in your clone—`--no-sync` deliberately keeps it local so you can review and push the branch yourself in the next step.

::: {.callout-note}
### `--no-sync`
By default `xorq catalog add` pulls then pushes the catalog's git remote after committing; `--no-sync` skips that, leaving the new commit purely local. The default is harmless when no remote is configured yet (User A's first `add` earlier had nothing to pull or push), but User B's clone *does* have an `origin`—so `--no-sync` is what keeps the entry on the local feature branch until you push it yourself in the next step, after reviewing the diff.
:::

Push the feature branch and open the PR:

::: {.panel-tabset}

### GitHub
```bash
cd ~/work/flights-catalog-userb
git log --oneline -3      # confirm: "add: <hash> (aliases flights-aa-only)"
git push -u origin add-aa-only-model
gh pr create --title "Add AA-only flights model" --body "Adds an AA-filtered view of the flights model under alias flights-aa-only."
```

User A reviews the PR on GitHub. Because each `xorq catalog add` is a single commit, the diff is small and readable: a new alias under `aliases/`, a new entry under `entries/`, a new sidecar under `metadata/`, and an update to `catalog.yaml`.

### Local bare repo
```bash
git -C ~/work/flights-catalog-userb log --oneline -3      # confirm: "add: <hash> (aliases flights-aa-only)"
git -C ~/work/flights-catalog-userb push -u origin add-aa-only-model
```

There's no PR—User A reviews the change as a regular fetched branch (see the next section). The diff is the same: a new alias under `aliases/`, a new entry under `entries/`, a new sidecar under `metadata/`, and an update to `catalog.yaml`.

:::

## Merge the PR and pull the changes (User A)

::: {.panel-tabset}

### GitHub
User A reviews the diff, approves the PR, and clicks **Merge pull request** in the GitHub UI. (The `gh` command-line tool equivalent is `gh pr merge --squash <pr-number>` from a clone—but a tutorial reader doing this step manually is the most common path.)

### Local bare repo
There's no PR UI to merge through, so User A pulls the branch into their catalog and merges by hand:

```bash
cd ~/work/flights-catalog-usera
git fetch origin add-aa-only-model
git diff main..origin/add-aa-only-model    # review the change
git merge --no-ff origin/add-aa-only-model -m "Merge: add AA-only flights model"
git push origin main
```

`--no-ff` keeps the merge commit so the history matches what GitHub would have produced from a "Merge pull request" click.

:::

Once main has moved on the remote, User A pulls. `xorq catalog pull` runs `git pull` against the configured remote, fast-forwarding the local main:

```bash
cd ~/flights-tutorial
uv run xorq catalog --path ~/work/flights-catalog-usera pull
uv run xorq catalog --path ~/work/flights-catalog-usera list-aliases
# flights-aa-only
# flights-model
```

::: {.callout-note}
### On the local-bare-repo path, `pull` is effectively a no-op
You merged in your local clone and pushed up—your local `main` is already at the merged tip, so there's nothing for `pull` to fast-forward. The command is still worth running because it's the same one-liner User A would use after a teammate clicked "Merge pull request" on GitHub; here it just confirms the new alias is visible.
:::

The same alias is now visible everywhere the catalog is cloned. Anyone with access can recover and query `flights-aa-only` exactly the way they recover `flights-model`.

## Swap the profile at recovery time

The catalog stores expressions, not connections. When User A built the entry they used the default Xorq backend (a `xorq_datafusion` session); when User B recovers it, they may want to execute against a different profile—perhaps a SQLite database they've configured locally, a Postgres instance with extra resources, or a Snowflake warehouse. Rebinding to a connection is a Python step.

A **profile** in Xorq is a named connection configuration: `con_name` plus connection kwargs, serialized to disk. Save one with `Profile.from_con(con).save(alias=...)`, load it with `Profile.load(...)`.

This section demonstrates the swap by moving to SQLite—a genuinely different backend than the default, no server to provision, and the `adbc-driver-sqlite` connector already came in via the `sqlite` extra in the prereqs. Save the snippet below as `profile_swap.py` in `~/flights-tutorial-userb/` and run it with `uv run python profile_swap.py`:

```python
# profile_swap.py
from pathlib import Path

from xorq.vendor.ibis.backends.profiles import Profile
import xorq.api as xo
from xorq.catalog.catalog import Catalog

catalog = Catalog.from_repo_path(Path("~/work/flights-catalog-userb").expanduser(), init=False)

# Capture User B's preferred connection as a named profile
sqlite_con = xo.sqlite.connect()                 # in-memory SQLite
Profile.from_con(sqlite_con).save(alias="local_dev_sqlite", clobber=True)

# Later—possibly in a different script—load the profile and bind the entry to it
profile = Profile.load("local_dev_sqlite")
con = profile.get_con()
expr = catalog.load("flights-model", con=con)

print("Executing against backend:", con.name)
print(
    expr.group_by("origin")
        .agg(
            flight_count=expr.count(),
            avg_dep_delay=expr.dep_delay.mean(),
        )
        .order_by("origin")
        .execute()
)
```

```
Executing against backend: sqlite
  origin  flight_count  avg_dep_delay
0    JFK             3      10.000000
1    LAX             3       4.333333
2    ORD             2      37.500000
```

The `Executing against backend: sqlite` line is the proof—User A cataloged the entry against `xorq_datafusion`, User B loaded it bound to a SQLite connection, and `.execute()` shipped the work to SQLite. `catalog.load(name, con=...)` returns the underlying Xorq expression—the flights table, in this case—bound to whichever connection you pass; you compose any group-by / aggregation you like on top, and `.execute()` runs it on the chosen backend. The entry on disk is unchanged; the swap is purely a runtime decision.

::: {.callout-note}
### `catalog.load` vs `from_tagged`
`from_tagged(entry.expr)` rebuilds the BSL `SemanticModel` so you can call `.query(...)` against it—that's the right tool when you want the semantic-layer interface back. `catalog.load(name, con=...)` skips the BSL layer and gives you the underlying Xorq expression bound to a connection of your choosing—that's the right tool when you want to redirect execution to a specific backend without touching the entry. They compose: profiles for the connection, BSL for the dimensions and measures.
:::

## What you learned

- A Xorq catalog is a git repository: every `xorq catalog add` is one commit, and the diff is small enough to review on GitHub.
- `xorq uv build <script> -e expr` packages a model script into a dependency-pinned build artifact; `xorq catalog add <build-dir> -a <alias>` files it into the catalog.
- Sharing the catalog is `xorq catalog push` (after a one-time `xorq catalog set-remote` + `git push -u origin main`); cloning uses `xorq catalog clone <url> --path ...`.
- Collaboration uses the GitHub workflow you already know: branch, commit, push, open a PR. The reviewer sees alias and entry files in the diff; merging makes the new alias available everywhere the catalog is cloned.
- `from_tagged(flights_entry.expr)` recovers the BSL model on the consumer side—a Python step, the same call as in the foundation tutorial, regardless of where the entry came from.
- `catalog.load(name, con=...)` rebinds a cataloged expression to a different connection at recovery time. Combined with named `Profile`s, downstream users can pick their own execution backend—SQLite, Postgres, anything Xorq supports—without modifying the entry.

## Next steps

- [Your first build](your_first_build.qmd)—package the cataloged model into a portable build artifact.
- [Switch backends](../../getting_started/switch_backends.qmd)—see what kinds of profiles Xorq supports.
- [Explore caching](../../getting_started/explore_caching.qmd)—keep the recovered query path fast for downstream users.

## Tutorials — ML tutorials — Split data for training

This tutorial teaches you how to split data for model training and evaluation. You'll learn how to create train, test, validation, and holdout splits using Xorq's deterministic splitting functions.

After completing this tutorial, you'll know how to partition data properly for ML workflows.

## Prerequisites

You need:

- Xorq installed: `pip install "xorq[examples]"`
- Basic understanding of train/test splits in ML


## Why split your data?

Here's the problem: if you train and evaluate on the same data, you can't tell if your model learned real patterns or just memorized the training set.

Why this matters: imagine you're building a fraud detection model. You train it, test it on the same data, and get 99% accuracy. Great, right? But then you deploy it, and it performs terribly on new transactions. You've overfit.

The solution: split your data into separate partitions. Train on one portion, evaluate on another. This gives you an honest measure of how your model performs on unseen data.

:::{.callout-tip}
### Deterministic splits
Xorq's splits are deterministic. You get the same partitions every time with the same random seed, which makes your experiments reproducible.
:::

## How to follow along

This tutorial builds code incrementally. Each section provides a code block that you run sequentially.

**Recommended approach**—Open a terminal, run `python` to start an interactive Python shell, then copy and paste each code block in order.

**Alternative approaches**:

- **Jupyter notebook**: Create a new notebook and run each code block in a separate cell

- **Python script**: Combine all code blocks into a single `.py` file and run it

The code blocks build on each other. Variables like `table`, `train`, and `test` are created in earlier blocks and used in later ones.

## Create sample data

Now you'll create some sample data to work with:

```python
# split_data.py
import xorq.api as xo
from xorq.api import memtable

# <1>
N = 100000
table = memtable(
    [(i, f"value_{i}") for i in range(N)], 
    columns=["key1", "val"]
)

# <2>
print(f"Created table with {N} rows")
print(f"Columns: {table.columns}")
print("\nFirst 5 rows:")
print(table.head(5).execute())
```
1. Create a table with 100,000 rows. Each row has a unique key and a value.
2. Preview what you created.

You'll see:

```
Created table with 100000 rows
Columns: ('key1', 'val')

First 5 rows:
   key1      val
0     0  value_0
1     1  value_1
2     2  value_2
3     3  value_3
4     4  value_4
```

This synthetic data shows you how splitting works without loading a real dataset. Once you've got your data, you can move on to splitting it.

## Simple train/test split

Now you'll split your data into training and test sets.

Add this to `split_data.py`:

```python
# <1>
train, test = xo.train_test_splits(
    table,
    unique_key="key1",
    test_sizes=0.25,
    num_buckets=N,
    random_seed=42
)

# <2>
train_count = train.count().execute()
test_count = test.count().execute()
total = train_count + test_count

# <3>
print(f"\nTrain size: {train_count} ({train_count/total:.1%})")
print(f"Test size: {test_count} ({test_count/total:.1%})")
```
1. Split into train (75%) and test (25%). The unique_key determines how rows get assigned.
2. Count rows in each partition.
3. Verify the split ratios.

You'll see:

```
Train size: 75003 (75.0%)
Test size: 24997 (25.0%)
```

**What just happened?** Xorq hashed the `key1` column for each row and assigned it to either train or test based on the hash value. With `test_sizes=0.25`, roughly 25% go to test and 75% to train.

The key insight here: the same row always goes to the same partition with the same random seed. This makes your splits reproducible.

## Understanding the parameters

Here's what you need to know about each parameter:

**unique_key**—The column Xorq hashes to assign rows to partitions. Choose a column with high cardinality (many unique values). In production, this might be a user ID, transaction ID, or timestamp.

**test_sizes**—When you pass a single float (like 0.25), you get two partitions: train and test. The float is the test proportion.

**num_buckets**—The number of hash buckets. Higher values give more precise splits. Use at least as many buckets as your dataset size.

**random_seed**—Makes splits deterministic. Same seed = same split every time.

Why does this matter? In practice, you want your experiments to be reproducible. If your splits change between runs, then you can't compare model performance reliably.

## Multi-partition splits

Sometimes you need more than two partitions. You might want training, validation, test, and holdout sets.

Add this to `split_data.py`:

```python
# <1>
partition_sizes = [0.1, 0.2, 0.3, 0.4]

# <2>
holdout, test, validation, training = xo.train_test_splits(
    table,
    unique_key="key1",
    test_sizes=partition_sizes,
    num_buckets=N,
    random_seed=42
)

# <3>
counts = {
    "holdout": holdout.count().execute(),
    "test": test.count().execute(),
    "validation": validation.count().execute(),
    "training": training.count().execute()
}

total = sum(counts.values())

# <4>
print("\nMulti-partition split:")
for name, count in counts.items():
    print(f"{name.upper()}: {count} ({count/total:.1%})")
```
1. Define partition sizes as a list. These should sum to 1.0 (10% + 20% + 30% + 40%).
2. Create four mutually exclusive partitions. Order matters: first size goes to first return value.
3. Count rows in each partition.
4. Verify the ratios match what you requested.

You'll see:

```
Multi-partition split:
HOLDOUT: 10003 (10.0%)
TEST: 19995 (20.0%)
VALIDATION: 29994 (30.0%)
TRAINING: 40008 (40.0%)
```

Each partition is a separate table expression. You can use them independently for different stages of your ML workflow.

Understanding this pattern helps you set up proper evaluation pipelines. Train on training, tune hyperparameters on validation, evaluate final performance on test, and keep holdout for the very end.

## Split column for manual control

This raises a question: what if you want more control over how you use the splits?

Here's where `calc_split_column` comes in. Instead of returning separate tables, it adds a column that labels which partition each row belongs to.

Add this to `split_data.py`:

```python
# <1>
split_column = xo.calc_split_column(
    table,
    name="partition",
    unique_key="key1",
    test_sizes=[0.1, 0.2, 0.3, 0.4],
    num_buckets=N,
    random_seed=42
)

# <2>
table_with_split = table.mutate(split_column)

# <3>
print("\nSplit column distribution:")
result = (
    table_with_split
    .group_by("partition")
    .agg(count=xo._.partition.count())
    .order_by("partition")
    .execute()
)
print(result)
```
1. Create a column that assigns each row to a partition (0, 1, 2, or 3).
2. Add the split column to your table.
3. Count how many rows are in each partition.

You'll see:

```
Split column distribution:
   partition  count
0          0  10003
1          1  19995
2          2  29994
3          3  40008
```

The partition numbers (0, 1, 2, 3) correspond to your test_sizes list order. Partition 0 gets 10%, partition 1 gets 20%, and so on.

Why use this pattern? You keep all your data in one table with partition labels. You can filter dynamically, pass labels to downstream processing, or group by partition for analysis.

Now that you've seen how split columns work, you can move on to deterministic splitting.

## Deterministic splits with random_seed

Here's the thing: you want your splits to be reproducible. Same code, same data, same splits.

Add this to `split_data.py`:

```python
# <1>
train_a, test_a = xo.train_test_splits(
    table,
    unique_key="key1",
    test_sizes=0.25,
    num_buckets=N,
    random_seed=42
)

# <2>
train_b, test_b = xo.train_test_splits(
    table,
    unique_key="key1",
    test_sizes=0.25,
    num_buckets=N,
    random_seed=42
)

# <3>
print("\nDeterministic splits (same seed):")
print(f"train_a count: {train_a.count().execute()}")
print(f"train_b count: {train_b.count().execute()}")
print("Counts match - splits are identical!")

# <4>
train_c, test_c = xo.train_test_splits(
    table,
    unique_key="key1",
    test_sizes=0.25,
    num_buckets=N,
    random_seed=99
)

print(f"\ntrain_c count (different seed): {train_c.count().execute()}")
print("Different seed produces different split")
```
1. Create a split with random_seed=42.
2. Create another split with the same random_seed=42.
3. The counts are identical because the seed is the same.
4. Change the seed to get a different split.

You'll see:

```
Deterministic splits (same seed):
train_a count: 75003
train_b count: 75003
Counts match - splits are identical!

train_c count (different seed): 74989
Different seed produces different split
```

**What does success look like?** The same seed produces identical splits. Different seeds produce different splits. This gives you reproducibility when you need it and randomness when you want it.

Most teams find that fixing the random seed in production is crucial. Without it, your splits change between runs, making it impossible to compare experiments.

## Complete example

Here's everything in one place. This shows the full workflow:

```python
# split_data.py
import xorq.api as xo
from xorq.api import memtable

# Create sample data
N = 100000
table = memtable(
    [(i, f"value_{i}") for i in range(N)], 
    columns=["key1", "val"]
)

print(f"Created table with {N} rows")

# Simple train/test split
train, test = xo.train_test_splits(
    table,
    unique_key="key1",
    test_sizes=0.25,
    num_buckets=N,
    random_seed=42
)

train_count = train.count().execute()
test_count = test.count().execute()
total = train_count + test_count

print(f"\nSimple split:")
print(f"Train: {train_count} ({train_count/total:.1%})")
print(f"Test: {test_count} ({test_count/total:.1%})")

# Multi-partition split
holdout, test_set, validation, training = xo.train_test_splits(
    table,
    unique_key="key1",
    test_sizes=[0.1, 0.2, 0.3, 0.4],
    num_buckets=N,
    random_seed=42
)

counts = {
    "holdout": holdout.count().execute(),
    "test": test_set.count().execute(),
    "validation": validation.count().execute(),
    "training": training.count().execute()
}

total_multi = sum(counts.values())

print("\nMulti-partition split:")
for name, count in counts.items():
    print(f"{name.upper()}: {count} ({count/total_multi:.1%})")

# Split column approach
split_column = xo.calc_split_column(
    table,
    name="partition",
    unique_key="key1",
    test_sizes=[0.1, 0.2, 0.3, 0.4],
    num_buckets=N,
    random_seed=42
)

table_with_split = table.mutate(split_column)

print("\nSplit column distribution:")
result = (
    table_with_split
    .group_by("partition")
    .agg(count=xo._.partition.count())
    .order_by("partition")
    .execute()
)
print(result)
```

Run this:

```bash
python split_data.py
```

Notice how you created multiple types of splits: simple train/test, multi-partition, and split columns, all with deterministic results.

## What you learned

You've learned how to split data properly for ML workflows. Here's what you accomplished:

- Created simple train/test splits with `train_test_splits()`
- Built multi-partition splits for train/validation/test/holdout
- Used `calc_split_column()` for manual partition control
- Made splits deterministic with random_seed
- Understood how unique_key determines row assignment

The key insight? Proper data splitting is fundamental to honest model evaluation. Train on one portion, evaluate on another, and always use deterministic splits for reproducibility.

## Next steps

Now that you know how to split data, continue learning:

- [Train your first model](train_your_first_model.qmd) shows how to train models with Xorq
- [Explore model evaluation](explore_model_evaluation.qmd) compares multiple models on held-out data
- [Your first build](../core_tutorials/your_first_build.qmd) packages your training pipeline as a portable artifact

## Tutorials — ML tutorials — Train your first model

This tutorial walks you through training your first classification model with Xorq. You'll use the Iris dataset to build a flower species classifier and see how deferred execution works in action.

After completing this tutorial, you'll know how to wrap scikit-learn pipelines with Xorq and make predictions using deferred execution.

## Prerequisites

You need:

- Xorq installed: `pip install "xorq[examples]"`
- Basic familiarity with scikit-learn

## How to follow along

This tutorial builds code incrementally. Each section adds to the same Python file (`train_classifier.py`). You can:

- **Python interactive shell (recommended)**: Open a terminal, run `python`, then copy and paste each code block sequentially
- **Run as a script**: Add each code block to `train_classifier.py`, then run `python train_classifier.py` after each section
- **Jupyter notebook**: Create a new notebook and run each code block in a separate cell

The code blocks build on each other. Variables like `iris`, `xorq_pipeline`, and `fitted_pipeline` are created in earlier blocks and used in later ones.

:::{.callout-tip}
### How deferred execution works
When you call `.fit()` in Xorq, training doesn't happen immediately. Xorq builds a computation graph. Training only runs when you call `.execute()`. This lets Xorq cache trained models and reuse them across runs.
:::

## Load data and define your target

Here's what you need to know: you'll load the Iris dataset and separate features from the target.

```python
# train_classifier.py
import xorq.api as xo

# <1>
iris = xo.examples.iris.fetch()

# <2>
target = "species"
features = tuple(iris.drop(target).schema())

# <3>
print(f"Loaded {iris.count().execute()} rows")
print(f"Target: {target}")
print(f"Features: {len(features)} columns")
```
1. Load the Iris dataset. This returns an expression, not actual data.
2. Define what you're predicting (target) and what the model uses (features).
3. Check what you loaded.


You'll see:

```
Loaded 150 rows
Target: species
Features: 4 columns
```

**What just happened?** The `iris` variable is an expression. It's a description of data, not the data itself. Nothing executes until you call `.execute()`. This is deferred execution in action.

Once you've got your data loaded, you can move on to building the pipeline.

## Build and wrap your pipeline

Now you'll create a scikit-learn pipeline and wrap it with Xorq. Think of it this way: you're taking a standard scikit-learn workflow and adding Xorq's caching layer.

Add this to `train_classifier.py`:

```python
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline as SklearnPipeline
from xorq.expr.ml import Pipeline

# <1>
sklearn_pipeline = SklearnPipeline([
    ('scaler', StandardScaler()),
    ('classifier', KNeighborsClassifier(n_neighbors=5))
])

# <2>
xorq_pipeline = Pipeline.from_instance(sklearn_pipeline)

print("Pipeline wrapped with Xorq!")
```
1. Create a standard scikit-learn pipeline. It normalizes features, then classifies with k-nearest neighbors.
2. Wrap it with `Pipeline.from_instance()`. This adds deferred execution.

**Why normalize first?** K-nearest neighbors is distance-based. If one feature ranges from 0 to 100 and another from 0 to 1, then the first feature dominates. Normalization fixes this.

## Train the model (but don't execute yet)

Here's the thing: when you call `.fit()`, you're not actually training yet. You're describing the training.

Add this to `train_classifier.py`:

```python
# <1>
fitted_pipeline = xorq_pipeline.fit(
    iris,
    features=features,
    target=target
)

print("Training described (but hasn't run yet)!")
```
1. Describe the training operation. Xorq builds a graph node, but doesn't execute.

**The key insight**—You've told Xorq what to do, but it hasn't done it yet. Training only happens when you call `.execute()`.

Understanding this timing helps you optimize workflows. You can describe complex pipelines, then execute once.

## Make predictions and see results

Now for the payoff. You'll make predictions, and that's when everything executes.

Add this to `train_classifier.py`:

```python
# <1>
def as_struct(expr, name=None):
    struct = xo.struct({c: expr[c] for c in expr.columns})
    if name:
        struct = struct.name(name)
    return struct

# <2>
ORIGINAL_ROW = "original_row"
predictions_expr = (
    iris.mutate(as_struct(iris, name=ORIGINAL_ROW))
    .pipe(fitted_pipeline.predict)
    .drop(target)
    .unpack(ORIGINAL_ROW)
)

# <3>
predictions = predictions_expr.execute()

print("\nFirst 10 predictions:")
print(predictions[["species", "predicted"]].head(10))
```
1. Create a helper that packages data into a struct. This preserves original values.
2. Build the prediction expression. Still deferred.
3. Execute. Training and prediction happen now, in one optimized pass.

You'll see:

```
First 10 predictions:
      species predicted
0      setosa    setosa
1      setosa    setosa
2      setosa    setosa
3      setosa    setosa
...
```

**What does success look like?** Your predictions match the actual species. When you called `.execute()`, Xorq trained the pipeline and made predictions in one shot.

Next, you'll check how accurate your model is.

## Check your accuracy

Now check how well your model performed. Add this to `train_classifier.py`:

```python
# <1>
accuracy_expr = (
    predictions_expr
    .mutate(correct=xo._.species == xo._.predicted)
    .agg(
        total=xo._.species.count(),
        correct_count=xo._.correct.sum().cast("int64"),
    )
    .mutate(accuracy=xo._.correct_count / xo._.total)
)

# <2>
result = accuracy_expr.execute()
accuracy = result["accuracy"][0]
correct = result["correct_count"][0]
total = result["total"][0]

print(f"\nAccuracy: {accuracy:.1%}")
print(f"Got {correct} out of {total} correct")
```
1. Build an accuracy calculation. Create a boolean for correct predictions, count them, compute the ratio.
2. Execute and display results.

You'll typically see:

```
Accuracy: 96.7%
Got 145 out of 150 correct
```

**Why this pattern matters**—Evaluation is also deferred. You describe metrics, then execute once. Most teams find that this simplifies evaluation code.

## Complete example

Here's everything in one file. This is what you built:

```python
# train_classifier.py
import xorq.api as xo
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline as SklearnPipeline
from xorq.expr.ml import Pipeline

# Load data
iris = xo.examples.iris.fetch()
target = "species"
features = tuple(iris.drop(target).schema())

# Build and wrap pipeline
sklearn_pipeline = SklearnPipeline([
    ('scaler', StandardScaler()),
    ('classifier', KNeighborsClassifier(n_neighbors=5))
])
xorq_pipeline = Pipeline.from_instance(sklearn_pipeline)

# Train (deferred)
fitted_pipeline = xorq_pipeline.fit(iris, features=features, target=target)

# Predict (deferred)
def as_struct(expr, name=None):
    struct = xo.struct({c: expr[c] for c in expr.columns})
    if name:
        struct = struct.name(name)
    return struct

ORIGINAL_ROW = "original_row"
predictions_expr = (
    iris.mutate(as_struct(iris, name=ORIGINAL_ROW))
    .pipe(fitted_pipeline.predict)
    .drop(target)
    .unpack(ORIGINAL_ROW)
)

# Evaluate (deferred)
accuracy_expr = (
    predictions_expr
    .mutate(correct=xo._.species == xo._.predicted)
    .agg(
        total=xo._.species.count(),
        correct_count=xo._.correct.sum().cast("int64"),
    )
    .mutate(accuracy=xo._.correct_count / xo._.total)
)

# Execute and show results
result = accuracy_expr.execute()
accuracy = result["accuracy"][0]
correct = result["correct_count"][0]
total = result["total"][0]

print(f"Accuracy: {accuracy:.1%}")
print(f"Got {correct} out of {total} correct")

# Show sample predictions
predictions = predictions_expr.execute()
print("\nSample predictions:")
print(predictions[["species", "predicted"]].head(10))
```

Run it:

```bash
python train_classifier.py
```

Notice how you built multiple deferred operations (training, predictions, accuracy), then executed. Xorq optimized the entire graph.

## What you learned

You built a complete ML workflow with deferred execution. Here's what you accomplished:

- Loaded data as expressions (deferred)
- Wrapped a scikit-learn pipeline with Xorq
- Trained a model without immediate execution
- Made predictions using the struct pattern
- Evaluated accuracy with deferred metrics

The key insight? Deferred execution lets you describe complex workflows, then Xorq handles optimization and caching automatically.

## Next steps

Now that you've trained your first model, continue learning:

- [Split data for training](split_data_for_training.qmd) shows how to create proper train/test splits
- [Explore model evaluation](explore_model_evaluation.qmd) walks through comparing multiple candidate models
- [Explore caching](../../getting_started/explore_caching.qmd) shows how to skip recomputation across iterations

## Tutorials — ML tutorials — Compare model performance

This tutorial shows you how to compare multiple machine learning models to find the best performer. You'll learn how to evaluate different classifiers systematically using Xorq's ML workflows.

After completing this tutorial, you'll know how to run experiments that compare model performance and select the best approach for your data.

## Prerequisites

You need:

- Xorq installed: `pip install "xorq[examples]"`
- Basic familiarity with scikit-learn classifiers
- Understanding of train/test splits


## Why compare models?

Here's the problem: different ML algorithms have different strengths. A decision tree might work well on one dataset while k-nearest neighbors performs better on another. You can't know which is best without testing.

Why this matters: imagine you're building a spam classifier. You pick k-nearest neighbors because it's simple, deploy it, and get 70% accuracy. But if you'd compared multiple models first, then you might have found that a random forest gives you 85% accuracy. That 15-point difference is the cost of not evaluating systematically.

The solution: compare multiple classifiers on your data. Train each one, measure performance, pick the winner. Xorq makes this easy because you can wrap any scikit-learn estimator and evaluate it with the same code.

:::{.callout-tip}
### Systematic comparison
Change the estimator, run the pipeline, compare scores. That's the pattern. Xorq's deferred execution lets you build evaluation workflows that work across any scikit-learn model.
:::

## How to follow along

This tutorial builds code incrementally. Each section provides a code block that you run sequentially.

**Recommended approach**—Open a terminal, run `python` to start an interactive Python shell, then copy and paste each code block in order.

**Alternative approaches**:

- **Jupyter notebook**: Create a new notebook and run each code block in a separate cell

- **Python script**: Combine all code blocks into a single `.py` file and run it

The code blocks build on each other. Variables like `X_train`, `train`, `test`, and `features` are created in earlier blocks and used in later ones.

## Create synthetic data

Start by generating a classification dataset. You'll use the "moons" dataset, which has two interleaving half-circles:

```python
import xorq.api as xo
import pandas as pd
import numpy as np
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split

# <1>
X, y = make_moons(noise=0.3, random_state=0)

# <2>
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.4, random_state=42
)

# <3>
print(f"Training samples: {len(X_train)}")
print(f"Test samples: {len(X_test)}")
print(f"Features: {X.shape[1]}")
```
1. Generate a "moons" dataset with 100 samples and some noise.
2. Split into train (60%) and test (40%) sets.
3. Check the sizes.

You'll see:

```
Training samples: 60
Test samples: 40
Features: 2
```

This synthetic data has two classes that aren't linearly separable. It's perfect for comparing how different classifiers handle non-linear boundaries.

Understanding this data helps you interpret results later. The moons shape means linear models struggle while non-linear models perform better.

## Convert to Xorq tables

Now you'll convert the NumPy arrays into Xorq table expressions:

```python
# <1>
def make_xorq_tables(X_train, y_train, X_test, y_test):
    con = xo.connect()
    
    # Create training table
    train = con.register(
        pd.DataFrame(X_train, columns=["feature_0", "feature_1"])
        .assign(target=y_train),
        "train"
    )
    
    # Create test table
    test = con.register(
        pd.DataFrame(X_test, columns=["feature_0", "feature_1"])
        .assign(target=y_test),
        "test"
    )
    
    features = ["feature_0", "feature_1"]
    return train, test, features

# <2>
train, test, features = make_xorq_tables(X_train, y_train, X_test, y_test)

# <3>
print(f"\nXorq tables created")
print(f"Train columns: {train.columns}")
print(f"Features: {features}")
```
1. Create a helper function that converts arrays to Xorq tables using con.register.
2. Convert your train/test data to Xorq expressions.
3. Verify the tables.

The output shows:

```
Xorq tables created
Train columns: ('feature_0', 'feature_1', 'target')
Features: ['feature_0', 'feature_1']
```

**What just happened?** You registered pandas DataFrames as tables in Xorq. Now you can use these tables with Xorq's deferred execution patterns.

Once you've got your tables ready, you can move on to training models.

## Train and evaluate one model

Now you'll train a single classifier and measure its accuracy. Think of it this way: you're establishing a baseline before comparing multiple models.

```python
import sklearn.pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from xorq.expr.ml import Pipeline

# <1>
sklearn_pipeline = sklearn.pipeline.Pipeline([
    ("scaler", StandardScaler()),
    ("knn", KNeighborsClassifier(n_neighbors=3))
])

# <2>
xorq_pipeline = Pipeline.from_instance(sklearn_pipeline)

# <3>
fitted_pipeline = xorq_pipeline.fit(
    train,
    features=features,
    target="target"
)

# <4>
score_expr = fitted_pipeline.score_expr(test)

# <5>
score = score_expr.execute()
print(f"\nK-Nearest Neighbors accuracy: {score:.2%}")
```
1. Create a scikit-learn pipeline with scaling and k-nearest neighbors (k=3).
2. Wrap it with Xorq's Pipeline.from_instance().
3. Fit on the training data (deferred).
4. Create a scoring expression (still deferred).
5. Execute to get the actual accuracy.

You'll typically see:

```
K-Nearest Neighbors accuracy: 90.00%
```

Here's the key insight: `.score_expr()` returns a deferred expression. Nothing executes until you call `.execute()`. This lets you build complex evaluation workflows before running anything.

Why use `.score_expr()` instead of immediate scoring? You can compose it with other operations, cache results, and optimize execution across multiple evaluations.

## Compare multiple classifiers

This raises a question: how do you compare several models efficiently?

Here's where the pattern shines. You define your models, wrap each in a pipeline, and evaluate them all:

```python
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

# <1>
classifiers = {
    "K-Nearest Neighbors": KNeighborsClassifier(n_neighbors=3),
    "Linear SVM": SVC(kernel="linear", C=0.025, random_state=42),
    "Decision Tree": DecisionTreeClassifier(max_depth=5, random_state=42),
    "Random Forest": RandomForestClassifier(
        max_depth=5, n_estimators=10, max_features=1, random_state=42
    ),
}

# <2>
results = {}
for name, clf in classifiers.items():
    # Wrap in sklearn pipeline with scaling
    sklearn_pipe = sklearn.pipeline.Pipeline([
        ("scaler", StandardScaler()),
        ("classifier", clf)
    ])
    
    # Convert to Xorq and fit
    xorq_pipe = Pipeline.from_instance(sklearn_pipe)
    fitted = xorq_pipe.fit(train, features=features, target="target")
    
    # Evaluate
    score = fitted.score_expr(test).execute()
    results[name] = score
    
    print(f"{name}: {score:.2%}")

# <3>
best_model = max(results, key=results.get)
best_score = results[best_model]

print(f"\nBest model: {best_model}")
print(f"Best accuracy: {best_score:.2%}")
```
1. Define four classifiers to compare.
2. Loop through each: wrap, fit, score.
3. Find the best performer.

You'll see output like:

```
K-Nearest Neighbors: 90.00%
Linear SVM: 85.00%
Decision Tree: 87.50%
Random Forest: 92.50%

Best model: Random Forest
Best accuracy: 92.50%
```

**What does success look like?** You've compared four different classifiers and identified that Random Forest performs best on this dataset. The non-linear decision boundary of Random Forest handles the moons shape better than linear models.

Most teams find that this pattern simplifies model selection. Define models, evaluate them, pick the winner.

## Verify against scikit-learn

Now you'll verify that Xorq's scores match scikit-learn's scores exactly. This builds confidence that Xorq's wrapper doesn't change the underlying algorithms.

```python
# <1>
def verify_score(train, test, features, target, sklearn_pipeline):
    # Xorq evaluation
    xorq_pipe = Pipeline.from_instance(sklearn_pipeline)
    fitted = xorq_pipe.fit(train, features=features, target=target)
    xorq_score = fitted.score_expr(test).execute()
    
    # sklearn evaluation
    train_df = train.execute()
    test_df = test.execute()
    sklearn_pipeline.fit(train_df[features], train_df[target])
    sklearn_score = sklearn_pipeline.score(test_df[features], test_df[target])
    
    return xorq_score, sklearn_score

# <2>
sklearn_pipe = sklearn.pipeline.Pipeline([
    ("scaler", StandardScaler()),
    ("knn", KNeighborsClassifier(n_neighbors=5))
])

xorq_score, sklearn_score = verify_score(
    train, test, features, "target", sklearn_pipe
)

# <3>
print(f"\nVerification:")
print(f"Xorq score: {xorq_score:.4f}")
print(f"sklearn score: {sklearn_score:.4f}")
print(f"Match: {np.isclose(xorq_score, sklearn_score)}")
```
1. Create a helper that evaluates with both Xorq and scikit-learn.
2. Test with a k-nearest neighbors classifier.
3. Verify the scores match.

You'll see:

```
Verification:
Xorq score: 0.9000
sklearn score: 0.9000
Match: True
```

This confirms that Xorq produces identical results to scikit-learn. The only difference is deferred execution and caching. The algorithms themselves are unchanged.

Understanding this gives you confidence to use Xorq in production. You're not changing how models work, just how they execute.

## Complete example

Here's the full workflow in one place. If you started with the Python shell, you've already run all of this. If you want to create a script, here's everything combined:

```python
import xorq.api as xo
import pandas as pd
import numpy as np
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
import sklearn.pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from xorq.expr.ml import Pipeline

# Generate synthetic data
X, y = make_moons(noise=0.3, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.4, random_state=42
)

# Convert to Xorq tables
def make_xorq_tables(X_train, y_train, X_test, y_test):
    con = xo.connect()
    train = con.register(
        pd.DataFrame(X_train, columns=["feature_0", "feature_1"])
        .assign(target=y_train),
        "train"
    )
    test = con.register(
        pd.DataFrame(X_test, columns=["feature_0", "feature_1"])
        .assign(target=y_test),
        "test"
    )
    features = ["feature_0", "feature_1"]
    return train, test, features

train, test, features = make_xorq_tables(X_train, y_train, X_test, y_test)

# Define classifiers
classifiers = {
    "K-Nearest Neighbors": KNeighborsClassifier(n_neighbors=3),
    "Linear SVM": SVC(kernel="linear", C=0.025, random_state=42),
    "Decision Tree": DecisionTreeClassifier(max_depth=5, random_state=42),
    "Random Forest": RandomForestClassifier(
        max_depth=5, n_estimators=10, max_features=1, random_state=42
    ),
}

# Evaluate each classifier
results = {}
for name, clf in classifiers.items():
    sklearn_pipe = sklearn.pipeline.Pipeline([
        ("scaler", StandardScaler()),
        ("classifier", clf)
    ])
    xorq_pipe = Pipeline.from_instance(sklearn_pipe)
    fitted = xorq_pipe.fit(train, features=features, target="target")
    score = fitted.score_expr(test).execute()
    results[name] = score
    print(f"{name}: {score:.2%}")

# Select best model
best_model = max(results, key=results.get)
print(f"\nBest model: {best_model} ({results[best_model]:.2%})")
```

Notice how you compared four classifiers with minimal code. The pattern is consistent: wrap, fit, score, compare.

## What you learned

You've learned how to evaluate multiple models systematically. Here's what you accomplished:

- Created synthetic classification data with make_moons
- Converted NumPy arrays to Xorq table expressions
- Trained and scored individual classifiers
- Compared multiple models to find the best performer
- Verified that Xorq matches scikit-learn exactly

The key insight? Model comparison is systematic with Xorq. Define your candidates, evaluate them all, pick the winner. The deferred execution pattern works across any scikit-learn estimator.

## Next steps

Now that you know how to compare models, continue learning:

- [Train your first model](train_your_first_model.qmd) covers the basics of model training with Xorq
- [Split data for training](split_data_for_training.qmd) shows proper train/test/validation splits
- [Your first build](../core_tutorials/your_first_build.qmd) shows how to package the winning model as a reproducible artifact

## Concepts — Fundamentals — What is Xorq?

Xorq is an open source compute catalog for AI. It helps teams catalog, compose, reuse, and observe transformations, features, models, and pipelines across multiple engines.

Think of it this way—Apache Iceberg standardized data. Xorq standardizes compute.

## What's Xorq?

Xorq is an open source compute catalog for reusing, shipping, and observing ML pipelines across multiple engines.

::: {.callout-note}
Data has standards like Iceberg and Delta. Xorq is the missing analog to Apache Iceberg, but for compute.
:::

## ML pipeline problems

Data infrastructure has standards. Iceberg and Delta solved how teams store and version data. But compute remains fragmented.

ML pipelines suffer from four problems:

- **Leakage prone**: Feature generation on historical data causes silent temporal leakage. Models train on information they shouldn't have access to at prediction time.
- **Disjointed**: Pipelines communicate through runtime artifacts. Monitoring gets bolted on as an afterthought rather than built in from the start.
- **Research-only**: Features trapped in notebooks require complete rewrites for batch and online serving. What works in Jupyter fails in production.
- **Undecipherable**: ML pipelines become black boxes with no lineage or computation visibility. When something breaks, there is no way to trace back to the root cause.

These problems compound over time. A single invisible imputation in opaque preprocessing can run undetected for months, breaking model assumptions and invalidating predictions.

## The solution: Xorq

Traditional approaches force a choice between the expressiveness of pandas and the scalability of SQL engines. This creates SQL-pandas impedance mismatches, wasteful recomputation, and pipelines that work in notebooks but fail in production.

Xorq provides a compute catalog with five core capabilities:

- **Multi-engine workflows**: Combine Snowflake, DuckDB, and Python in a single declarative pipeline. No engine-specific rewrites.
- **Portable UDFs**: Define user-defined functions once and run them across supported engines. Xorq serializes logic to YAML artifacts for reproducibility.
- **Automatic caching**: Cache intermediate results to avoid recomputing expensive joins after every change.
- **Dev-to-prod consistency**: A pipeline that works locally runs in production without code changes. Compile-time validation catches errors early.
- **Built-in lineage**: Automatic column-level lineage tracking and fail-fast execution provide visibility for production systems.


## Core features

Xorq's architecture provides six core capabilities that work together with:

- **Deferred execution**: Builds expression graphs before computation. Execution happens when explicitly triggered, allowing optimization of the full pipeline.
- **Automatic caching**: Caches intermediate results and invalidates cache when source data changes. Avoids unnecessary recomputation.
- **Multi-engine execution**: A single typed plan (YAML manifest) executes on DuckDB, Snowflake, DataFusion, BigQuery, Pandas, PostgreSQL, PyIceberg, and Trino.
- **Content-addressed artifacts**: Features, models, and pipelines are versioned by content hash. Identical computation produces identical hashes.
- **Composable catalog**: Catalog entries act as contracts between teams. Teams discover existing computations, compose on previous work, and share validated logic.
- **Built-in governance**: Column-level lineage tracking and compile-time schema validation. Policies enforce through compiled plans, not runtime checks.


## Where Xorq fits

Xorq sits between your data sources and your ML applications. It's not a replacement for your data warehouse or your model serving infrastructure, it's the layer that connects them.

The architecture image here illustrates this positioning. Expressions and UDFs flow through the manifest layer into stateless, Arrow-native execution across any supported engine.

![](/images/introduction/architecture-dark.png){fig-alt="Architecture diagram showing expressions and UDFs flowing through manifest to stateless Arrow-native execution"}

Teams use Xorq for feature stores, semantic layers, ML pipelines, batch scoring, and data CI. In each case, the value comes from the same properties: reproducibility, multi-engine portability, and full lineage.


## Who Xorq is for

Xorq is designed for data scientists, data engineers, and ML engineers who:

- Work with SQL-heavy pipelines but need Python for ML.
- Run pipelines across multiple engines such as DuckDB locally and Snowflake in production.
- Need reproducibility and lineage for compliance or debugging.
- Want to share and reuse validated computations across teams.

If you are comfortable with pandas and SQL, then the Xorq API feels familiar.

## Built on open standards

Xorq builds on battle-tested foundations:

- **Apache Arrow Flight**: Zero-copy data transfer between processes.
- **Ibis**: Cross-engine expression trees serialized to YAML.
- **Apache DataFusion**: Embedded engine for stateless Arrow-native execution.
- **uv**: Reproducible Python environment management.

The core expression and build logic remains open source.

## Next steps

Now that you understand what Xorq is and why it exists, choose your path:

- [Quickstart](/getting_started/quickstart.qmd) to build your first pipeline in under five minutes
- [GitHub](https://github.com/xorq-labs/xorq) to star the repo and contribute
- [Discord](https://discord.gg/8Kma9DhcJG) to join the community

## Concepts — Fundamentals — Deferred Execution

Deferred execution is the architectural choice that everything else in Xorq
builds on. When you chain operations on a Xorq expression, nothing runs. Xorq
records *what you asked for* as a graph of operations and waits. Computation
happens only when you explicitly ask for results—`.execute()`, `.to_pandas()`,
`.to_pyarrow()`. The same approach drives [Polars
Lazy](https://docs.pola.rs/user-guide/concepts/lazy-api/) and [Dask
Delayed](https://docs.dask.org/en/stable/delayed.html), but Xorq applies it to
ML pipelines that span multiple engines.

This page explains why the deferral is there and what you get for it. For a
hands-on walkthrough of when computation fires, see the
[Defer query execution](../../getting_started/understand_deferred_execution.qmd)
tutorial.

## What "deferred" means here

A Xorq expression is a description, not a result. `iris.filter(...)` returns
another expression; `.group_by(...).agg(...)` returns another still. Each call
adds a node to an **expression graph**—a tree of operations rooted at a source
table—and returns immediately without touching data. The graph is a value you
can inspect, hash, serialize, and pass around. It's not the answer; it's the
plan for computing the answer.

Because the plan is a first-class object, Xorq can look at the *whole* pipeline
before it runs any of it. An eager system sees one operation at a time and must
commit to a result before it knows what comes next. A deferred system sees the
filter, the join, the aggregation, and the cache boundary all at once, and gets
to decide how to run them together.

## Why it matters for ML pipelines

ML pipelines are where the payoff shows up, for three reasons.

**Full-pipeline optimization.** With the entire graph in hand, Xorq can prune
columns that nothing downstream reads, push filters closer to the source so less
data moves, and collapse redundant steps. Run the operations eagerly and each
one materializes its result before the next is even known, so none of these
rewrites are possible.

**Cross-engine routing.** A real pipeline rarely lives in one engine. You might
read from Postgres, do a heavy join in DuckDB, and run a model in Python. Because
the graph names every backend boundary explicitly, Xorq can route each part to
the engine that suits it and stream the data across the boundary—without you
hand-coding the transfers. The graph is what makes
[multi-engine execution](multi_engine_execution.qmd) automatic instead of manual.

**A portable artifact.** A deferred graph can be serialized to a manifest and
re-run elsewhere. That's what lets a build become a catalog entry: the plan
travels, and a different machine reproduces the computation from it.

## Eager versus deferred

The difference is concrete. Build an expression and it stays an expression—no
data has moved yet:

```{python}
import xorq.api as xo

con = xo.connect()
iris = xo.examples.iris.fetch(backend=con)

# Deferred: still an expression; the full plan is visible to Xorq.
deferred = (
    iris.filter(xo._.sepal_length > 6)
    .group_by("species")
    .agg(avg_width=xo._.sepal_width.mean())
)

print(f"deferred is still an expression: {type(deferred).__module__}.{type(deferred).__name__}")
```

Force execution after the first step instead, and you hold materialized data—a
table—that Xorq can no longer optimize or chain:

```{python}
# Eager: execute() runs now and returns data, so optimization stops here.
immediate = iris.filter(xo._.sepal_length > 6).execute()

print(f"immediate is materialized data: {type(immediate).__module__}.{type(immediate).__name__}")
print(f"rows: {len(immediate)}")
```

In the eager line, the filter runs against the full table before the group-by
exists, so Xorq can't fuse the two or push the predicate down. In the deferred
graph, the filter, group-by, and aggregation are one plan; Xorq optimizes and
runs them as a unit when you finally call `.execute()`:

```{python}
result = deferred.execute()
print(result)
```

## The tradeoff

Deferral isn't free. The result you want isn't there until you ask for it, and
an expression that looks complete may still be hiding an error that surfaces only
at execution time. The graph adds a layer of indirection between your code and
your data, and debugging sometimes means reasoning about a plan rather than
inspecting a value.

The fix is to execute deliberately. Call `.execute()` early when you genuinely
need materialized data—an exploratory peek at a sample, a value you have to
branch on in Python, a result you'll hand to a library that doesn't speak Xorq.
The rule of thumb: defer while you're still describing the pipeline, materialize
when you've crossed into needing the data itself. Executing too early just throws
away optimization Xorq would otherwise have done for you.

## Connection to caching

Deferred execution is what makes Xorq's [caching](intelligent_caching_system.qmd)
possible at all. Because the pipeline is a known graph before it runs, Xorq can
attach a cache to any node and compute a stable key for the computation that node
represents. `.cache()` doesn't run anything—it inserts a `CachedNode` into the
graph. On execution Xorq walks the graph, and at each cached node either reuses a
stored result or computes and stores it.

That only works because the structure is known up front. An eager system has no
graph to attach a cache to and no way to ask whether this exact computation has
run before—the computation is already gone by the time you'd ask. A deferred graph
gives every node a stable identity, which is the same property that powers
[content-addressed artifacts](why_content_addressed_artifacts.qmd).

## See also

- [Defer query execution](/getting_started/understand_deferred_execution.qmd)—a hands-on walkthrough of building a graph and running it.
- [Get started with Xorq](/tutorials/core_tutorials/get-started-with-xorq-init.qmd)—build a deferred pipeline end to end with `xorq build` / `xorq run`.
- [Expression types](/concepts/understanding_xorq/expression_types.qmd)—the kinds of nodes a deferred graph is made of.
- [`execute`](/reference/execute.qmd)—the API call that materializes a deferred expression.

## Concepts — Fundamentals — Content-addressed artifacts

Most systems name things by *where* or *when*: a path, a timestamp, a
version number someone bumped by hand. Xorq names things by *what they contain*. An
artifact's name is a deterministic hash of the computation that produces it, so
identical computations get identical names and different computations get
different names—automatically, with no bookkeeping. This is **content
addressing**, and it's the mechanism underneath both Xorq's
[cache](intelligent_caching_system.qmd) and its
[catalog](catalog_as_executable_memory.qmd).

This page explains what that hash is, why Xorq is built around it, and what it
buys you in practice.

## What gets hashed

The hash is computed over the [expression
graph](why_deferred_execution.qmd)—the tree of operations you built—not over the
data the graph produces and not over the wall-clock time you built it. Two
expressions that describe the same computation hash to the same value; change any
operation in the graph and the hash changes.

You can see this directly. The same pipeline built twice produces the same hash;
widen the filter and you get a different one:

```{python}
import xorq.api as xo

con = xo.connect()
iris = xo.examples.iris.fetch(backend=con)

def pipeline(threshold):
    return (
        iris.filter(xo._.sepal_length > threshold)
        .group_by("species")
        .agg(n=xo._.species.count())
    )

a = pipeline(6)
b = pipeline(6)   # same computation
c = pipeline(5)   # one operand changed

print(f"a: {a.ls.tokenized}")
print(f"b: {b.ls.tokenized}")
print(f"c: {c.ls.tokenized}")
print(f"a == b (same pipeline):     {a.ls.tokenized == b.ls.tokenized}")
print(f"a == c (filter changed):    {a.ls.tokenized == c.ls.tokenized}")
```

The hash is deterministic and stable across processes and machines: it depends
only on the structure of the computation, so the same pipeline run tomorrow on
another host produces the same name. Note that it fingerprints the
*specification*, not the bytes of the output—a run can still differ in timestamps
or floating-point rounding, but its identity as a computation is fixed.

## When the hash moves

A build's name answers "was *this* pipeline, under *these* rules, built?", so
two things move it: the expression graph, and the set of normalization rules
Xorq hashed it under. The second is why a Xorq upgrade can change build hashes
without you editing a pipeline—the rule set in force is folded into the build
hash, so a build produced under a different set of rules is honestly a
different artifact.

Build hashes and cache keys are separate. Cache keys are computed by the
[cache strategies](intelligent_caching_system.qmd) and are deliberately
neutral to *which rules are registered*, so adding, removing or replacing a
rule moves build hashes and leaves cache keys alone. That is the case below,
and it is the common one for an upgrade.

It is not a blanket guarantee: cache keys are computed *by* those rules, so an
upgrade that changes what an existing normalizer actually does moves both.

::: {.callout-note}
### Build hashes change once in this release

Xorq now folds the identity-bearing normalization rule set into the build hash.
Concretely:

- **Build directory names change**, and catalog entries keyed by build hash
  move with them. **Rebuild your pipelines** so catalog entries are re-keyed
  and aliases point at hashes produced under the new regime.
- The `xorq:expr_hash` value recorded in cached-Parquet provenance metadata
  changes for the same reason.
- **Cache keys do not change.** Existing cache entries still hit, and nothing
  recomputes.
- **Existing builds still load and run.** A build's hash is read from its
  directory name and never recompared, so artifacts built before this release
  are unaffected.
:::

## Why Xorq uses it

Three properties fall out of naming by content, and Xorq leans on all three.

**Reproducibility.** A name that's a function of the computation is a name that
can't drift. `final_v2_fixed.parquet` tells you nothing about what's inside;
`8b4472fbeb97` *is* the computation. If you have the hash, you can ask for
exactly that artifact and know you got the right one.

**Cache validity.** A cache is only useful if you can answer whether this exact
computation has run before. Content addressing makes that a dictionary lookup: hash the
expression, check whether the key exists. Xorq's
[`ModificationTimeStrategy`](intelligent_caching_system.qmd) folds the source's
change metadata into the hash, so when upstream data changes the key changes, the
old key misses, and the result recomputes. Invalidation isn't a timer or a
watcher—it's a key that stops matching when its inputs change.

**Artifact identity.** When two artifacts share a hash they *are* the same
computation; when the hashes differ, something in the graph or its inputs
changed. The name can't lie about what the artifact does. That honesty is what
lets a catalog dedupe, share, and reuse entries without trusting a human-chosen
label.

## What it means in practice

The practical payoff is that *reuse is automatic*. Build the same pipeline twice
and the second build is a no-op—the artifact already exists under that name, so
there is nothing to recompute and nothing to ask. You never write "did someone
already produce this?" logic; identical computations collapse to one artifact by
construction.

This is the opposite of prose or vector memory, which has to dedupe explicitly—
comparing embeddings or running similarity heuristics—because two records of the
same fact look different on disk. Content addressing makes that question
disappear.

## Connection to the catalog

A [Xorq catalog](catalog_as_executable_memory.qmd) is a git repository of build
artifacts, and content addressing is how it names them. Each entry is a zipped
build named by its content hash:

```
git-catalogs/penguins
├── aliases
│   └── penguins-agg.zip -> ../entries/fa2122f6a9e9.zip
├── entries
│   └── fa2122f6a9e9.zip
└── metadata
    └── fa2122f6a9e9.zip.metadata.yaml
```

The hash is the identity; an **alias** is a human-readable symlink—
`penguins-agg`—pointing at whatever hash is current. As the pipeline evolves the
hash moves and the alias follows. The hash answers *what is this*; the alias
answers *which one do I want*. Because the name is derived from the computation,
the catalog gets deduplication, honest provenance, and conflict-free sharing for
free—the same git repo two people clone can never disagree about what an entry
contains.

::: {.callout-tip}
### Where to go next
- [Deferred execution](why_deferred_execution.qmd)—the graph that gets hashed.
- [Caching](intelligent_caching_system.qmd)—how the hash becomes a cache key and drives invalidation.
- [The catalog as executable memory](catalog_as_executable_memory.qmd)—how content-addressed entries become shareable, runnable memory.
:::

## Concepts — Fundamentals — The catalog as executable memory

Most "memory" an agent accumulates is prose: Markdown notes, a `MEMORY.md`
index, chat history. Prose memory has to be reread and reinterpreted by
whatever model picks it up next, and it has no way to *run*. A Xorq catalog is
the other kind of memory—**executable**. Each entry is a content-addressed
pipeline plus the environment to run it, so a future agent or human doesn't
re-derive the computation from a description; it executes the artifact and gets
the same answer.

The same instinct drives teams to keep agent memory as Markdown in a git repo
rather than in a vector database: the files stay the source of truth, and the
data never gets trapped behind a proprietary schema or query language. A Xorq
catalog takes that instinct one step further. Where a Markdown note still has to
be read and acted on, a catalog entry can be executed directly.

This page explains what that means and why the catalog is built that way.
For the hands-on version, see
[Working with the catalog](/tutorials/core_tutorials/working_with_the_catalog.qmd).

## What a memory entry is

In a prose memory system the unit is a snippet of text. In Xorq the unit is an
**entry**: a built [expression](/concepts/understanding_xorq/expression_types.qmd)
together with its pinned Python environment. An entry is enough to reproduce a
result on a machine that has never seen the code that produced it—the wheel
and `requirements.txt` travel with it.

You reason about an entry the way you reason about a matrix—by its invariants,
not its cells:

- **Schema**—what columns it produces, and of what type.
- **Lineage**—which sources and transforms it depends on.
- **Content hash**—a deterministic fingerprint of the computation. Same
  pipeline, same hash; change a filter and the hash changes.
- **Deterministic execution**—the hash fingerprints the pipeline
  *specification*, and the pinned environment makes "run it again" reproduce the
  same computation on the same code path. (It doesn't promise byte-identical
  output: timestamps, aggregation order, or floating-point rounding can still
  vary at runtime.)

These are knowable *before* execution. An agent can decide whether an entry is
the one it needs by reading metadata, without running anything.

## Why a git repo, not a service

A Xorq catalog is a git repository of build artifacts on the filesystem—
nothing more. There is no server to call and no API to learn. That choice is
deliberate:

- **Discovery is file operations.** Metadata lives in `metadata/*.yaml` sidecars
  next to the zipped entries. Listing, filtering, and lookup-by-alias or by-hash
  are `ls`, `grep`, and `git`—tools every agent already has. An agent that
  clones the repo can find everything without learning anything new.
- **History is the audit log.** Every catalog operation is a commit. `git log`
  and `git reflog` tell you exactly when an entry was added, aliased, or
  removed, and by whom—provenance you get for free from git rather than
  building separately.
- **Sharing is `git push` / `git clone`.** Collaboration is the pull-request
  workflow teams already run. A teammate clones the catalog, adds an entry on a
  branch, opens a PR; the merge is ordinary. A catalog supports at most
  one git remote, which keeps "where's the source of truth" unambiguous. Large
  files are handled by an opt-in [git-annex](https://git-annex.branchable.com/)
  remote rather than bloating the repo.

- **No vendor lock-in.** A vector-database memory store keeps your data inside a
  proprietary schema; getting it out means an export job and a migration plan.
  A catalog is files, so migrating is `git clone` or an `rsync` of the
  directory. Nothing is trapped behind an API.

The catalog is portable for the same reason any git repo is: it's just files.

## Indexes are derived, not the source of truth

A vector store inverts the usual relationship: the index *is* the memory, and the
original text is something you hope was preserved alongside the embeddings. A
catalog keeps the source of truth on disk—the entries and their YAML
metadata—and treats any richer lookup as a layer built on top. The plain-text
sidecars already make keyword search a `grep`; nothing stops you from building a
BM25 or vector index over the same metadata when you want fuzzy recall. Because
those indexes are derived, you can rebuild or throw them away without losing
anything, and a stale index can never disagree with the artifact it points to.

## What's on disk

::: {.callout-important}
A catalog uses symlinks for aliases, so it expects a POSIX filesystem that
preserves them. On Windows, run catalog operations under
[WSL](https://learn.microsoft.com/windows/wsl/)—a native Windows filesystem
without symlink support produces a broken catalog.
:::

After `catalog init` the repo holds a single manifest:

```yaml
# catalog.yaml
entries: []
aliases: []
```

`catalog add` packages a build directory—the manifest (`expr.yaml` plus
`*_metadata.json`) and the uv-pinned environment—into an entry and records it.
A populated catalog looks like:

```
git-catalogs/penguins
├── aliases
│   └── penguins-agg.zip -> ../entries/fa2122f6a9e9.zip
├── entries
│   └── fa2122f6a9e9.zip
├── metadata
│   └── fa2122f6a9e9.zip.metadata.yaml
└── catalog.yaml
```

Entries are zipped builds named by their content hash. Aliases are symlinks
pointing at an entry—a human-readable name (`penguins-agg`) for a hash
(`fa2122f6a9e9`). The symlink on disk carries the `.zip` suffix of its target
(`penguins-agg.zip`), so a catalog expects a POSIX filesystem that preserves
symlinks. Metadata sidecars are plain YAML, which is what makes discovery a
`grep`:

```bash
# Entries that emit an 'avg_bill_length' column
grep -l 'avg_bill_length' git-catalogs/penguins/metadata/*.yaml

# Entries running on DataFusion
grep -l 'xorq_datafusion' git-catalogs/penguins/metadata/*.yaml
```

## Why content-addressing matters

Entries are named by a hash of the computation, not by a timestamp or a
user-chosen filename. Two consequences fall out of that:

1. **Reuse is automatic.** Building the same pipeline twice produces the same
   hash, so the second build is a no-op—the entry already exists. There is no
   "did someone already compute this?" question to answer by hand. Prose and
   vector memory have to dedupe explicitly—comparing embeddings, running
   similarity heuristics—because two notes describing the same fact look
   different on disk. Content-addressing makes that question disappear: identical
   computations collapse to one entry by construction.
2. **Identity is honest.** If two entries have the same hash they *are* the same
   computation; if the hashes differ, something in the graph or its inputs
   changed. The name can't lie about what the artifact does, the way a filename
   like `final_v2_fixed.py` can.

Aliases sit on top of this: they give a stable name to "whatever the current
best version is" while the underlying hash moves as the pipeline evolves. The
hash is identity; the alias is intent.

## Memory you can run

A prose note says *what someone once did*. A catalog entry *is the thing*, ready
to run, with its schema, lineage, and environment attached. That's the
difference between memory you have to trust and memory you can execute—and
it's why the catalog, not a pile of scripts, is where Xorq keeps what an agent
learns.

## See also

- [Working with the catalog](/tutorials/core_tutorials/working_with_the_catalog.qmd)—the publish / clone / pull loop, hands-on
- [Expression types](/concepts/understanding_xorq/expression_types.qmd)—what gets stored in an entry

## Concepts — Fundamentals — Expression types

Every operation in Xorq produces an expression --- a lazy, immutable description of a computation. Before anything executes, Xorq classifies each expression into an `ExprKind`. This classification drives how expressions are hashed, cached, serialized, and executed across backends.

## The expression kinds

Three of the kinds are **structural** --- they fall out of the shape of the expression graph itself:

| Kind | Meaning | Has input schema? |
|------|---------|-------------------|
| **Source** | A leaf node --- raw data with no transformations | No |
| **Expr** | Bound data with transformations applied | No |
| **UnboundExpr** | A template expression not tied to any backend | Yes |

Two more are **tag-driven** --- they're recognized by domain-specific tags
that higher-level tools (the ML pipeline API, the catalog, semantic layers)
attach to the outermost node:

| Kind | Meaning | Produced by |
|------|---------|-------------|
| **ExprBuilder** | Carries builder metadata that can recover a domain object (for example, a fitted ML pipeline) | `xo.Pipeline` fit/predict, semantic-layer queries |
| **Composed** | Carries a catalog tag, marking a reference to another catalog entry | `catalog add` / composition |

You can inspect the kind of any expression using the `.ls` accessor:

```{python}
import xorq.api as xo

# Source — just data, no transforms
t = xo.memtable({"a": [1, 2, 3], "b": ["x", "y", "z"]})
print(t.ls.kind)  # ExprKind.Source

# Expr — transforms applied to bound data
filtered = t.filter(xo._.a > 1)
print(filtered.ls.kind)  # ExprKind.Expr

# UnboundExpr — schema template, no concrete data
template = xo.table(schema={"a": "int64", "b": "string"})
print(template.ls.kind)  # ExprKind.UnboundExpr
```

## Source expressions

A **Source** is a leaf node in the expression graph --- it represents data without any transformations. Sources are the starting points of every pipeline.

### In-memory tables

The simplest source. Data lives in the current process and is wrapped in an `InMemoryTable` operation.

```{python}
import xorq.api as xo

t = xo.memtable({"name": ["alice", "bob"], "score": [95, 87]})
print(t.ls.kind)  # ExprKind.Source
```

### Database tables

A table registered in a backend (DuckDB, DataFusion, Postgres, etc.). Backed by a `DatabaseTable` operation that holds a reference to the backend connection.

```{python}
import xorq.api as xo

con = xo.duckdb.connect()
batting = xo.examples.batting.fetch(backend=con)
print(batting.ls.kind)  # ExprKind.Source
```

### Deferred reads

A lazy pointer to a file (Parquet, CSV). The file isn't read until execution. Backed by a `Read` operation.

```{python}
import xorq.api as xo

# A local Parquet file, fetched once via pins
path = xo.options.pins.get_path("batting")

# With an explicit schema the file isn't read until execute()
expr = xo.deferred_read_parquet(
    path,
    schema={
        "playerID": "string",
        "yearID": "int64",
        "teamID": "string",
        "G": "int64",
        "AB": "float64",
        "H": "float64",
    },
)
print(expr.ls.kind)  # ExprKind.Source
```

### Cached nodes

When you call `.cache()` on an expression, Xorq wraps it in a `CachedNode`. The cache itself is a source --- it acts as a materialization boundary that breaks the expression graph into independent segments.

```{python}
import xorq.api as xo
from xorq.caching import ParquetCache

con = xo.connect()
ddb = xo.duckdb.connect()
cache = ParquetCache.from_kwargs(source=con)

# The cached expression is a Source, even though it wraps transforms
cached = (
    xo.examples.batting.fetch(backend=ddb)
    .filter(xo._.yearID > 2010)
    .cache(cache=cache)
)
print(cached.ls.kind)       # ExprKind.Source
print(cached.ls.is_cached)  # True
```

This is an important property: caching resets the expression kind back to Source. Downstream transforms on a cached expression produce a new Expr that depends on the cache, not on the original upstream query.

### Remote tables

Created by `.into_backend()`, a `RemoteTable` transfers data between backends via Arrow record batches. Like a cache, it acts as a source boundary.

```{python}
import xorq.api as xo

ddb = xo.duckdb.connect()
con = xo.connect()  # DataFusion

# Move filtered DuckDB data into DataFusion
t = (
    xo.examples.batting.fetch(backend=ddb)
    .filter(xo._.yearID == 2015)
    .into_backend(con, "local_batting")
)
print(t.ls.kind)  # ExprKind.Source
```

## Expr --- transformed expressions

An **Expr** is any expression that applies one or more transformations to bound data. It has at least one source upstream and at least one operation (filter, project, aggregate, join, etc.) on top.

```{python}
import xorq.api as xo

t = xo.memtable({"a": [1, 2, 3], "b": [10, 20, 30]})

# Each of these produces an Expr
filtered = t.filter(xo._.a > 1)
projected = t.select("a")
aggregated = t.aggregate(total=t.b.sum())
sorted_t = t.order_by("a")
limited = t.limit(10)

# All are ExprKind.Expr
for expr in [filtered, projected, aggregated, sorted_t, limited]:
    print(expr.ls.kind)  # ExprKind.Expr
```

Transforms compose naturally by chaining. Each method returns a new immutable expression:

```{python}
import xorq.api as xo

ddb = xo.duckdb.connect()

result = (
    xo.examples.batting.fetch(backend=ddb)
    .filter(xo._.yearID > 2010)
    .select("playerID", "yearID", "teamID", "H", "AB")
    .mutate(avg=xo._.H / xo._.AB)
    .order_by(xo._.avg.desc())
    .limit(20)
)
print(result.ls.kind)  # ExprKind.Expr
```

### Available transform operations

| Method | Operation | Description |
|--------|-----------|-------------|
| `.filter()` | Filter | Select rows matching boolean predicates |
| `.select()` | Project | Choose or compute columns |
| `.mutate()` | Project | Add or replace columns |
| `.aggregate()` | Aggregate | Group-by and reduce |
| `.order_by()` | Sort | Sort rows |
| `.limit()` | Limit | Take first N rows |
| `.join()` | JoinChain | Combine tables |
| `.union()` | Union | Stack rows from two tables |
| `.distinct()` | Distinct | Deduplicate rows |
| `.drop()` | DropColumns | Remove columns |
| `.fill_null()` | FillNull | Replace nulls |
| `.dropna()` | DropNull | Remove rows with nulls |
| `.sample()` | Sample | Random subset |
| `.unnest()` | TableUnnest | Flatten array column |

## `UnboundExpr` --- template expressions

An **UnboundExpr** is an expression built on an `UnboundTable` instead of a concrete data source. It defines a transformation as a reusable template --- a schema in, a schema out, and the operations in between.

### Creating unbound expressions

Use `xo.table()` with a schema to create a template:

```{python}
import xorq.api as xo

# Define a template: takes int64 column "a", filters, adds a computed column
template = xo.table(schema={"a": "int64"})
transform = template.filter(template.a > 0).mutate(doubled=template.a * 2)

print(transform.ls.kind)  # ExprKind.UnboundExpr
```

### Unbinding bound expressions

You can convert any bound expression into an unbound one with `.unbind()`. This strips the backend connection and replaces `DatabaseTable` nodes with `UnboundTable` nodes:

```{python}
import xorq.api as xo

ddb = xo.duckdb.connect()
bound = xo.examples.batting.fetch(backend=ddb).filter(xo._.yearID > 2010)

unbound = bound.unbind()
print(bound.ls.kind)    # ExprKind.Expr
print(unbound.ls.kind)  # ExprKind.UnboundExpr

# Schema is preserved
assert unbound.schema() == bound.schema()
```

### Why unbound expressions matter

Unbound expressions are the foundation of several Xorq features:

- **Flight UDXFs**: Define a transformation template that runs on an Arrow Flight server. The input data is streamed as record batches, and the unbound expression describes what to do with it.
- **Serialization**: The expression YAML format stores unbound expressions so they can be loaded and bound to a different backend later.
- **Schema validation**: `ExprMetadata` exposes `schema_in` (the UnboundTable's schema) and `schema_out` (the result schema), enabling compile-time checks before execution.

The `.ls.metadata` accessor returns the `ExprMetadata` for any expression:

```{python}
import xorq.api as xo

template = xo.table(schema={"a": "int64"})
transform = template.filter(template.a > 0)

metadata = transform.ls.metadata
print(metadata.kind)       # ExprKind.UnboundExpr
print(metadata.schema_in)  # Schema with column "a" (int64)
print(metadata.schema_out) # Schema with column "a" (int64)
```

## Tags: the mechanism behind tag-driven kinds

Both tag-driven kinds come from the same primitive --- a **tag** node wrapped
around an expression. A tag is metadata (a name plus arbitrary keyword values)
that rides on top of the expression without changing the rows it produces.

Any table exposes two methods to attach one:

| Method | Node | Affects content hash? |
|--------|------|-----------------------|
| `.tag(name, **kwargs)` | `Tag` | No --- stripped before hashing |
| `.hashing_tag(name, **kwargs)` | `HashingTag` | Yes --- preserved during hashing |

```{python}
import xorq.api as xo

t = xo.memtable({"a": [1, 2, 3]})

tagged = t.tag("my_marker", note="anything")
print(tagged.ls.metadata.root_tag)  # "my_marker"
```

The difference is what happens at hash time. A plain `Tag` is invisible to the
content hash --- two expressions that differ only by a `Tag` cache to the same
key. A `HashingTag` is folded into the hash, so different tag metadata produces
distinct hashes (and distinct cache entries):

```{python}
import xorq.api as xo

t = xo.memtable({"a": [1, 2, 3]})

# Plain Tag: hash unchanged
print(t.ls.tokenized == t.tag("x").ls.tokenized)          # True

# HashingTag: hash changes with the metadata
print(t.ls.tokenized == t.hashing_tag("x").ls.tokenized)  # False
```

When Xorq classifies an expression, it walks the outermost chain of `Tag` /
`HashingTag` nodes and looks the tag name up in two registries: catalog tags
yield **Composed**, builder tags (registered by the ML pipeline API, semantic
layers, or third-party plugins) yield **ExprBuilder**. An unrecognized tag is
transparent --- the expression falls through to its structural kind.

## `ExprBuilder` --- expressions that carry a domain object

An **ExprBuilder** is an expression whose outermost node carries a builder tag
--- domain-specific metadata that lets Xorq recover the higher-level object the
expression came from. The expression still executes like any other; the tag is
extra information riding alongside it.

The most common source is the ML pipeline API. Fitting a pipeline and calling
`.predict()` produces an expression tagged with the fitted pipeline's steps,
features, and target:

```{python}
import xorq.api as xo
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression

train = xo.memtable(
    {
        "feature_0": [1.0, 2.0, 3.0, 4.0],
        "feature_1": [4.0, 5.0, 6.0, 7.0],
        "target": [0.0, 1.0, 0.0, 1.0],
    },
    name="train",
)

sk_pipe = make_pipeline(StandardScaler(), LinearRegression())
fitted = xo.Pipeline.from_instance(sk_pipe).fit(
    train, features=("feature_0", "feature_1"), target="target"
)
predict = fitted.predict(train)

print(predict.ls.kind)  # ExprKind.ExprBuilder
```

The recovered metadata lives on `.ls.metadata.builders`:

```{python}
for builder in predict.ls.metadata.builders:
    print(builder["type"], builder["steps"], "->", builder["target"])
```

This is what lets the catalog persist a fitted pipeline as an artifact and
rebuild it later: the tag describes how to reconstruct the domain object, not
just the rows it produces. Semantic-layer queries (boring-semantic-layer)
produce `ExprBuilder` expressions the same way.

## `Composed` --- references to catalog entries

A **Composed** expression carries a *catalog tag*. When you add an expression to
a catalog and then reference it from another expression, the reference is tagged
so the catalog can resolve it back to the stored entry instead of inlining the
whole subgraph. Composition is a catalog-level concern --- see the catalog
documentation for the full workflow.

## Inspecting expressions with `.ls`

Every expression has an `.ls` accessor (the LETSQL accessor) that exposes introspection properties:

| Property | Type | Description |
|----------|------|-------------|
| `.ls.kind` | `ExprKind` | Source, Expr, or UnboundExpr |
| `.ls.metadata` | `ExprMetadata` | Full metadata (kind, schema_in, schema_out) |
| `.ls.unwrapped` | `ops.Node` | Underlying op with any wrapper layers stripped |
| `.ls.backends` | `tuple` | All backend connections used in the expression graph |
| `.ls.is_multiengine` | `bool` | Whether the expression spans multiple backends (`False` for memtable / unbound---no backend) |
| `.ls.is_cached` | `bool` | Whether the root op is a CachedNode |
| `.ls.has_cached` | `bool` | Whether any CachedNode exists in the graph |
| `.ls.cached_nodes` | `tuple` | All CachedNode operations in the graph |
| `.ls.cache` | `Cache` | The cache object if `is_cached`, else None |
| `.ls.uncached` | `Expr` | Expression with all cache nodes removed |
| `.ls.tokenized` | `str` | Content hash of the expression |
| `.ls.cache_exists()` | `bool \| None` | Whether the cache is materialized (`None` if not cached) |

```{python}
import xorq.api as xo
from xorq.caching import ParquetCache

ddb = xo.duckdb.connect()
con = xo.connect()
cache = ParquetCache.from_kwargs(source=con)

expr = (
    xo.examples.batting.fetch(backend=ddb)
    .filter(xo._.yearID > 2010)
    .into_backend(con, "batting_local")
    .cache(cache=cache)
)

print(expr.ls.kind)            # ExprKind.Source (cache is a source)
print(expr.ls.is_cached)       # True
print(expr.ls.is_multiengine)  # True (DuckDB + DataFusion)
print(len(expr.ls.backends))   # 2
print(expr.ls.cache_exists())  # False (not yet executed)
```

## How kind affects the system

The expression kind isn't just a label --- it drives behavior across Xorq:

| Concern | Source | Expr | UnboundExpr |
|---------|--------|------|-------------|
| **Hashing** | Hash of the data reference (table name, path, etc.) | Hash of the full operation graph | Hash includes UnboundTable schema |
| **Serialization** | Stored as a data source reference in `expr.yaml` | Full operation tree serialized | Stored with `schema_in` for rebinding |
| **Caching** | Can be the *result* of a cache | Can be cached (wrapping in CachedNode) | Can't be cached (no data to materialize) |
| **Execution** | Read data from backend | Execute operation graph | Must bind to data first |
| **Build system** | Leaf node in the build graph | Intermediate node | Template --- not directly buildable |

The tag-driven kinds (**ExprBuilder**, **Composed**) execute, hash, and
serialize like the structural expression underneath them; the tag adds the
metadata the ML pipeline API and catalog use to recover or resolve the
higher-level object.

## See also

- [Get started with Xorq](/tutorials/core_tutorials/get-started-with-xorq-init.qmd)—construct Source and Expr nodes and build them into an artifact.
- [Deferred execution](/concepts/understanding_xorq/why_deferred_execution.qmd)—why expressions stay unevaluated until you ask for results.
- [`connect`](/reference/connect.qmd) and [`memtable`](/reference/memtable.qmd)—the API entry points that produce Source expressions.
- [`execute`](/reference/execute.qmd)—materialize any structural expression kind.

## Concepts — Execution and backends — Multi-Engine

Move data between different engines within a single expression using `into_backend()`.

```{python}
import xorq.api as xo

# Connect to engines
pg = xo.postgres.connect_env()
db = xo.duckdb.connect()

# Load data from different sources
batting = pg.table("batting")
awards = xo.examples.awards_players.fetch(backend=db)

# Filter in respective engines
recent = batting.filter(batting.yearID == 2015)
nl_awards = awards.filter(awards.lgID == "NL")

# Move data to postgres for join
result = recent.join(
    nl_awards.into_backend(pg),
    ["playerID"]
)

result.execute()
```

### Key features

* **Zero-copy transfers** using Apache Arrow
* **Automatic optimization** of data placement
* **Transparent movement** between any supported engines

### When to use

* **Leverage engine strengths** - Use DuckDB for local analysis, Postgres for joins
* **Combine data sources** - Join tables from different databases
* **Optimize performance** - Move small tables to where large tables live

## See also

- [Route a step to a specific backend](/how_to/switch_backends.qmd)—use `into_backend()` to place each step on the right engine.
- [Switch between backends](/getting_started/switch_backends.qmd)—a tutorial introduction to the same machinery.
- [`Table.into_backend`](/reference/Table.qmd#xorq.vendor.ibis.expr.types.relations.Table.into_backend)—the API that moves a step across engines.
- [Caching](/concepts/understanding_xorq/intelligent_caching_system.qmd)—cache the transferred result so the move doesn't repeat.

## Concepts — Execution and backends — Caching

Xorq's caching system stores intermediate results so iterative ML pipelines
don't recompute work they've already done. A cache can invalidate automatically
when upstream data changes, or freeze a result so it never recomputes—and it
can live inside a database backend or on disk. This page explains the model
behind those choices. For step-by-step usage see the
[Explore caching](../../getting_started/explore_caching.qmd) tutorial and the
[Cache results by backend](../../how_to/cache_by_backend.qmd) how-to; for the
full class matrix and how to build each cache see
[Cache API overview](../../api_reference/cache_api_overview.qmd).

## Lazy caching

Xorq operations are **lazy**—nothing runs until you call `.execute()`.
Attaching `.cache(...)` to an expression adds a `CachedNode` to the expression
graph but computes nothing on its own.

::: {.callout-note}
The lazy `.cache` is a deviation from Ibis
[cache](https://ibis-project.org/reference/expression-tables.html#ibis.expr.types.relations.Table.cache),
where the method eagerly executes the expression.
:::

On `.execute()`, Xorq walks the graph bottom-up looking for `CachedNode`s. For
each one it computes a cache key, checks storage for that key, and then either:

- **Cache miss**: materializes the result, writes it to storage, and proceeds as
  if it were a hit.
- **Cache hit**: returns a reference to the stored table (backend storage) or a
  deferred read of the stored file (Parquet storage).

The `CachedNode` is replaced with the storage result and execution continues on
the transformed graph. Hits and misses emit
[OpenTelemetry](https://opentelemetry.io/) trace events keyed by the cache key,
so a pipeline run shows exactly what was reused.

## Why multiple cache classes? Strategy versus storage

Every cache is the combination of two independent choices:

- A **strategy** decides *how the cache key is computed*—which in turn decides
  *when the cache invalidates*.
- A **storage** decides *where the cached data lives*.

The classes you import from `xorq.caching` (`SourceCache`, `ParquetCache`,
`SourceSnapshotCache`, …) are named combinations of these two axes. Picking a
class is really picking one strategy and one storage.

### Strategy: when the cache invalidates

A **cache key** is a hex string that identifies a cached result, computed by
hashing the expression. The two strategies differ in what goes into that hash:

- **`ModificationTimeStrategy`** folds backend-specific change metadata (a row
  count estimate, a last-altered timestamp, file mtime, a snapshot id) into the
  key. When the source data changes, the metadata changes, the key changes, the
  old key is no longer found, and the expression re-executes. This is what
  "automatic invalidation" means in Xorq—not a timer or a watcher, but a key
  that changes when its inputs change.
- **`SnapshotStrategy`** keys on the expression's structure only—table name,
  path, schema. The key is stable even when the underlying data changes, so the
  first cached result is served until you delete it. Use it when you'd rather
  serve a fixed result than pay for recomputation, like model-training inputs or
  an expensive backfill.

The exact change signal per backend (and its caveats) is tabulated in
[Cache API overview](../../api_reference/cache_api_overview.qmd#backend-invalidation-signals).

### Storage: where the data lives

- **`SourceStorage`** stores the result as a table inside whatever backend you
  pass as `source`. If that backend is in-memory (DuckDB, DataFusion, Xorq) the
  cache is an in-memory table; if it's a remote database (Postgres, Snowflake,
  Trino) the cache is a table there.
- **`ParquetStorage`** writes the result as a `.parquet` file on local disk
  (default `~/.cache/xorq/parquet/`, configurable via `XORQ_CACHE_DIR`). For
  Parquet storage the `source` backend isn't where the cache lives—it only
  writes the file on a miss and reads it back on a hit—so the source can be
  any backend.
- **`ParquetTTLStorage`** extends Parquet storage with a time-to-live: a file
  older than the TTL is treated as expired.
- **`GCStorage`** stores the Parquet files in a GCS bucket instead of local
  disk.

Calling `.cache()` with no arguments builds a `SourceCache` whose source is the
expression's own backend.

## Record batches and streaming

Data moves through the cache as Arrow **record batches**—chunks of rows with a
fixed schema, streamed one at a time by a `RecordBatchReader` rather than loaded
whole into memory. Writing to Parquet storage streams batches from the source
expression into the file; cross-engine transfer via `into_backend()` moves
batches over [Arrow Flight](https://arrow.apache.org/docs/format/Flight.html);
ingestion into a database streams batches in over
[ADBC](https://arrow.apache.org/adbc/). The pattern is always the same: produce
batches from one backend, consume them in another, without temp files or
whole-table loads.

## Chaining caches across engines

Because storage and source are decoupled, you can cache at each engine boundary
in a multi-engine pipeline—read and filter in one backend, move the result to
another with `into_backend()`, and cache the transferred result so the transfer
doesn't repeat on the next run. Each `.cache()` is an independent
strategy/storage choice, so a single pipeline can mix backend caches and
on-disk Parquet caches.

::: {.callout-tip}
### Where to go next
- [Explore caching](../../getting_started/explore_caching.qmd)—a hands-on walkthrough of hits, misses, and invalidation.
- [Cache results by backend](../../how_to/cache_by_backend.qmd)—pick the right cache class for your source and target.
- [Cache API overview](../../api_reference/cache_api_overview.qmd)—the full class matrix, how to build each cache, and per-backend invalidation signals.
:::

## Concepts — Advanced capabilities — The UDF system

User-defined functions (UDFs) are how you run your own Python inside a Xorq
expression. A UDF looks like any other column operation—it binds to columns,
returns a typed value, and stays deferred until execution—but the body is
arbitrary Python you write. This lets you fold pandas, PyArrow, scikit-learn, or
any library into a pipeline without leaving the expression graph.

This page explains the mental model behind UDFs and the kinds Xorq provides. For
exact signatures, see the [UDF system reference](/reference/make_pandas_udf.qmd).

## Why UDFs exist

SQL engines give you a fixed vocabulary of built-in functions. The moment you
need a custom transformation—a domain-specific calculation, a trained model's
prediction, a call into a Python library—that vocabulary runs out.

Xorq UDFs close that gap while preserving the three properties that make the
rest of the system work:

- **Deferred**: a UDF is a node in the expression graph, not an eager call. It
  runs only when you execute the expression, so the optimizer sees the whole
  pipeline first.
- **Typed**: every UDF declares an input schema and a return type. Schema
  mismatches are caught at build time, not mid-run.
- **Portable**: the UDF body is serialized (via `cloudpickle`) into the YAML
  artifact alongside the rest of the expression. The same pipeline runs on a
  different machine or backend without a code rewrite.

That last property is the point of difference. A UDF in Xorq isn't glue code
bolted onto a query—it's part of the content-addressed artifact, hashed, and
versioned like everything else.

## The mental model

Every UDF follows the same shape, regardless of kind:

1. You write a plain Python function.
2. You wrap it with a **factory** (`make_pandas_udf`, `agg.pyarrow`, and so on),
   passing a schema and a return type.
3. The factory returns a **constructor**. Calling `.on_expr(table)` binds the
   constructor to columns and produces an expression you drop into `mutate`,
   `aggregate`, or `over`.

```python
import xorq.api as xo
import xorq.expr.datatypes as dt

penguins = xo.examples.penguins.fetch(backend=xo.connect())

def bill_ratio(df):
    return df["bill_length_mm"] / df["bill_depth_mm"]

# 1. factory wraps the function with a schema and return type
bill_ratio_udf = xo.make_pandas_udf(
    fn=bill_ratio,
    schema=penguins.select("bill_length_mm", "bill_depth_mm").schema(),
    return_type=dt.float64,
    name="bill_ratio",
)

# 2. bind to columns and use like any other expression
result = penguins.mutate(ratio=bill_ratio_udf.on_expr(penguins)).execute()
```

Two details are worth internalizing because they recur in every kind:

**The schema is the contract.** The `schema` you pass names the columns your
function receives and their types. `.on_expr(table)` pulls exactly those columns
out of `table` by name and feeds them in. The `return_type` tells Xorq what
comes back. Together they let the build step validate the UDF before any data
moves.

**Data crosses a PyArrow boundary.** Internally, columns arrive as PyArrow
arrays. The pandas-based factories convert them to a `DataFrame` for your
function and convert the result back to a PyArrow array. The PyArrow-native
factories skip that conversion—you work with arrays directly. This is why the
pandas variants are convenient and the PyArrow variants are faster.

## The kinds of UDF

Xorq groups UDFs by *what shape of computation* they express—one row at a time,
a whole group, an ordered window, or a remote exchange.

| Kind | Factory | Input → output | Reach for it when |
|------|---------|----------------|-------------------|
| Scalar (pandas) | `make_pandas_udf` | rows → one value per row | you want pandas ergonomics for a per-row transform |
| Scalar (PyArrow) | `scalar` | rows → one value per row | the transform is numeric and performance matters |
| Expression scalar | `make_pandas_expr_udf` | a pre-computed value + rows → one value per row | a prediction needs a model trained earlier in the same pipeline |
| Aggregate | `agg.pyarrow`, `agg.pandas_df` | many rows → one value per group | you reduce a group to a scalar, struct, or trained model |
| Window | `pyarrow_udwf` | an ordered partition → one value per row | the result depends on row order or a sliding frame |
| Exchange | `flight_udxf` | a table → a table, out of process | the work belongs in a separate process or service |

### Scalar UDFs

A scalar UDF maps input columns to one output value per row. `make_pandas_udf`
hands your function a `DataFrame` and expects a `Series` back; the vendored
`scalar` factory works on PyArrow arrays directly when you want to avoid the
pandas round-trip. Use scalar UDFs for row-wise feature engineering—ratios,
classifications, string manipulation, anything that doesn't need to see other
rows.

### Expression scalar UDFs

`make_pandas_expr_udf` is a scalar UDF with one extra input: a value computed by
*another* expression, evaluated once and passed into every invocation. The
canonical use is inference. An aggregate UDF trains a model and serializes it;
the expression scalar UDF takes that serialized model as its `computed_kwargs_expr`
and applies it row by row. Training and prediction live in the same deferred
pipeline, so the model is never an out-of-band file.

```python
from xorq.expr.udf import make_pandas_expr_udf, agg

# an aggregate UDF trains and serializes a model
model_udaf = agg.pandas_df(
    fn=train_model, schema=train_schema, return_type=dt.binary, name="train",
)

# the expression scalar UDF consumes that model to predict
predict_udf = make_pandas_expr_udf(
    computed_kwargs_expr=model_udaf.on_expr(train_data),
    fn=predict,
    schema=test_schema,
    return_type=dt.string,
    name="predict",
)
```

### Aggregate UDFs (UDAFs)

An aggregate UDF reduces many rows to a single result, typically inside a
`group_by(...).agg(...)`. `agg.pyarrow` is the high-performance path—your
function receives PyArrow arrays and uses PyArrow compute. `agg.pandas_df` gives
each group to your function as a `DataFrame`, which is the right tool when you
need the pandas or scikit-learn ecosystem. The return type can be a scalar, a
`Struct` of several statistics, or `binary` for a serialized model.

### Window UDFs (UDWFs)

`pyarrow_udwf` creates a window function—the result for each row depends on an
ordered partition of surrounding rows. Configuration flags (`uses_window_frame`,
`include_rank`) change the signature your function must implement, from
whole-partition processing to frame-bounded scalars to rank-aware evaluation.
Custom parameters you pass to the factory become attributes on `self` inside the
function. Apply the result with `.over(...)`. Reach for a UDWF when order
matters—running totals, smoothing, rank scores.

### Exchange UDFs (UDXFs)

`flight_udxf` runs a whole-table transformation in a separate process behind an
[Apache Arrow Flight](https://arrow.apache.org/docs/format/Flight.html) server.
Unlike the other kinds, which execute in-process, a UDXF streams the input table
to a Flight server, runs your `process_df(df) -> df` function there, and streams
the result back. That isolation is what makes it suited to external API calls,
heavy model inference, microservice deployments, or running memory-intensive
code outside the main process. See [User-Defined Exchange
Functions](/api_reference/user_defined_exchange_functions.qmd) for the full
treatment.

## Choosing a UDF

Work down two questions:

1. **What's the output shape?** One value per row → scalar or window. One value
   per group → aggregate. A whole table → exchange. If a row's result depends on
   other rows in an order, that's a window; if it depends on a value computed
   elsewhere in the pipeline, that's an expression scalar UDF.
2. **Where should it run?** In-process is the default and the fastest. Choose an
   exchange UDF only when isolation or a separate service is the actual
   requirement—it adds a network hop.

Between the pandas and PyArrow variants of a given kind, prefer PyArrow when the
work is numeric and the dataset is large; prefer pandas when you want its API or
a library that speaks `DataFrame`.

## How UDFs fit the rest of Xorq

Because a UDF is a typed, deferred node, it inherits the system's guarantees for
free. It participates in [content-addressed
hashing](/concepts/understanding_xorq/why_content_addressed_artifacts.qmd), so an
unchanged UDF over unchanged data resolves to a cached result. It serializes into
the [expression format](/api_reference/expression_format.qmd), so a pipeline with
custom Python is still a portable artifact. And it sits in the
[expression graph](/concepts/understanding_xorq/expression_types.qmd) like any
other operation, so lineage and validation see straight through it.

## See also

- [`make_pandas_udf`](/reference/make_pandas_udf.qmd), [`make_pandas_expr_udf`](/reference/make_pandas_expr_udf.qmd), [`pyarrow_udwf`](/reference/pyarrow_udwf.qmd)—scalar and window factory reference.
- [`agg.pyarrow`](/reference/agg.pyarrow.qmd) and [`agg.pandas_df`](/reference/agg.pandas_df.qmd)—aggregate factory reference.
- [User-Defined Exchange Functions](/api_reference/user_defined_exchange_functions.qmd)—the exchange UDF concept and `flight_udxf` usage.
- [Expression types](/concepts/understanding_xorq/expression_types.qmd)—the graph a UDF becomes part of.
- [Why content-addressed artifacts](/concepts/understanding_xorq/why_content_addressed_artifacts.qmd)—why a portable UDF matters.

## Concepts — Advanced capabilities — Pins: versioned data artifacts

Xorq uses [pins](https://rstudio.github.io/pins-python/) to provide named, versioned access to shared datasets, trained models, and code modules. Instead of shipping large files in the repository or requiring manual downloads, artifacts are stored in a public cloud bucket and accessed by name.

## Overview

The pins system in Xorq allows you to:

- **Access shared datasets by name** without managing file paths or URLs
- **Load pre-trained ML models** for use in pipelines
- **Pin specific versions of artifacts** for reproducible pipelines
- **Cache downloads locally** so repeated access is fast

## Quick start

```{python}
import xorq.api as xo

# Fetch an example dataset as a table expression
t = xo.examples.iris.fetch()
```

`xo.examples.<name>.fetch()` downloads the artifact via pins, reads it into a table expression on the default backend, and caches the download locally. Subsequent calls use the cached copy.

## How it works

Xorq wraps the [pins Python library](https://rstudio.github.io/pins-python/) with a preconfigured connection to a public GCS bucket (`letsql-pins`). No authentication is required.

The configuration lives in `xo.options.pins`:

| Setting | Default | Description |
|---|---|---|
| `protocol` | `"gcs"` | Storage protocol |
| `path` | `"letsql-pins"` | GCS bucket name |
| `storage_options` | `{"cache_timeout": 0, "token": "anon"}` | Anonymous access; always check remote freshness |

### `get_path(name, board=None, **kwargs)`

Downloads the named pin and returns a local path. Extra keyword arguments
(such as `version`) are forwarded to the underlying pins library via
`board.pin_download`.

```{python}
# Get the latest version
path = xo.options.pins.get_path("diamonds")

# Pin a specific version for reproducibility
path = xo.options.pins.get_path("hackernews_lib", version="20250820T111457Z-1d66a")
```

Once you have a local path, hand it to any backend reader and execute:

```{python}
con = xo.connect()
t = con.read_parquet(xo.options.pins.get_path("batting"))
t.execute()
```

### `get_board()`

Returns the underlying `pins.board` object for advanced operations like listing available pins or reading metadata.

```{python}
board = xo.options.pins.get_board()
board.pin_list()        # list all available pins
board.pin_meta("iris")  # get metadata for a pin
```

## Available pins

| Pin | Format | Description |
|---|---|---|
| `"iris"` | CSV | Classic iris dataset |
| `"diamonds"` | Parquet | Diamonds pricing dataset |
| `"penguins"` | Parquet | Palmer penguins dataset |
| `"batting"` | Parquet | Baseball batting statistics |
| `"lending-club"` | Parquet | Lending Club loan data |
| `"bank-marketing"` | CSV | Bank marketing dataset |
| `"hn-fetcher-input-small.parquet"` | Parquet | HackerNews sample data |
| `"hn_tfidf_fitted_model"` | Binary | Pre-trained TF-IDF model |
| `"hn_sentiment_reg"` | Binary | Pre-trained XGBoost sentiment model |
| `"hackernews_lib"` | Python module | HackerNews pipeline code (versioned) |
| `"diamonds-model"` | JSON | XGBoost model for predicting diamond price |

## Common patterns

### Loading datasets

The preferred way to load example datasets is `xo.examples.<name>.fetch()`:

```{python}
import xorq.api as xo

# Fetch with default backend
t = xo.examples.diamonds.fetch()

# Fetch with a specific backend
con = xo.connect()
t = xo.examples.diamonds.fetch(backend=con)
```

This calls `xo.options.pins.get_path("diamonds")` under the hood, reads the file with the appropriate method based on format, and returns a table expression.

### Loading ML models

Pre-trained models are pinned as binary artifacts and loaded by path:

```{python}
import pathlib

TFIDF_MODEL_PATH = pathlib.Path(
    xo.options.pins.get_path("hn_tfidf_fitted_model")
)
XGB_MODEL_PATH = pathlib.Path(
    xo.options.pins.get_path("hn_sentiment_reg")
)
```

### Loading versioned code modules

Python modules can be pinned and loaded with a specific version, ensuring pipeline reproducibility:

```{python}
from xorq.common.utils.import_utils import import_python

m = import_python(
    xo.options.pins.get_path("hackernews_lib", version="20250820T111457Z-1d66a")
)
# m is now a module with functions defined in the pinned file

# Everything the pinned module brought into the namespace:
print([name for name in dir(m) if not name.startswith("_")])
```

## Caching behavior

The pins library caches downloads automatically in a platform-specific cache directory (typically `~/.cache/pins/` on Linux). The `cache_timeout: 0` setting in Xorq's default configuration means the library always checks if the remote version has changed, but serves from cache if it hasn't.

## See also

- [The catalog as executable memory](/concepts/understanding_xorq/catalog_as_executable_memory.qmd)—the catalog versions whole expressions the way pins version data artifacts.
- [Caching](/concepts/understanding_xorq/intelligent_caching_system.qmd)—how Xorq decouples stored results from their source.

## Overview — Backends — Overview

## Overview

Xorq supports multiple execution engines (backends) for running your data and ML workloads. Configure and use different backends to execute your pipelines.

### Configuration

|                                              |                                                                                   |
|----------------------------------------------|-----------------------------------------------------------------------------------|
| [Overview](overview.qmd)                     | Introduction to backend configuration and how to choose between backends.          |
| [Profiles](profiles_api.qmd)                 | The Profiles API securely manages database connections via environment variables. |
| [Environment Variables](env_variables.qmd)   | The available environment variables for backend configuration.                   |
| [Supported Backends](supported_backends.qmd) | The supported backends and how to choose between them.                            |

## Overview — Backends — Profiles

The Profiles API provides a secure way to manage database connection parameters through environment variable references, allowing you to create, save, load, and use database connections while keeping sensitive information protected.

## Profile

A `Profile` is an immutable representation of database connection settings that can be serialized to disk.

### Properties

| Property | Type | Description |
|----------|------|-------------|
| `con_name` | `str` | Backend name (for example, 'postgres', 'snowflake') |
| `kwargs_tuple` | `tuple` | Connection parameters as key-value pairs |
| `kwargs_dict` | `dict` | Connection parameters as a dictionary |
| `hash_name` | `str` | Unique hash identifier for the profile |
| `idx` | `int` | Unique numeric identifier |

### Creation methods

```{python}
#| eval: false

import xorq.api as xo 
from xorq.vendor.ibis.backends.profiles import Profile

# Create a new profile
profile = Profile(
    con_name="postgres",
    kwargs_tuple=(
        ("host", "${POSTGRES_HOST}"),
        ("port", 5432),
        ("database", "postgres"),
        ("user", "${POSTGRES_USER}"),
        ("password", "${POSTGRES_PASSWORD}"),
    ),
)

# Create from an existing connection
profile = Profile.from_con(xo.connect())

# Load from disk by alias
profile = Profile.load("postgres_example")
```

### Instance methods

```{python}
#| eval: false

# Create a connection
connection = profile.get_con()

# Create a modified copy
modified = profile.clone(**{"connect_timeout": 10})

# Convert to dictionary
profile_dict = profile.as_dict()

# Serialize as JSON
json_str = profile.as_json()

# Serialize as YAML
yaml_str = profile.as_yaml()

# Save to disk with optional alias
path = profile.save(alias="postgres_example", clobber=True)
```

## Profiles

A collection manager for working with multiple saved profiles.

```{python}
#| eval: false

from xorq.vendor.ibis.backends.profiles import Profiles

# Create a profiles manager
profiles = Profiles()
```

### Methods

```{python}
#| eval: false

# List all available profiles
available = profiles.list()

# Get profile by name
profile = profiles.get("postgres_example")

# Access profiles as attributes
profile = profiles.postgres_example

# Access profiles as dictionary keys
profile = profiles["postgres_example"]
```

## Environment variable handling

Profiles store references to environment variables, not their values:

```{python}
#| eval: false

import os

# Set environment variables
os.environ["POSTGRES_HOST"] = "examples.letsql.com"
os.environ["POSTGRES_USER"] = "letsql"
os.environ["POSTGRES_PASSWORD"] = "letsql"

# Create profile with environment variable references
profile = Profile(
    con_name="postgres",
    kwargs_tuple=(
        ("host", "${POSTGRES_HOST}"),
        ("port", 5432),
        ("database", "postgres"),
        ("user", "${POSTGRES_USER}"),
        ("password", "${POSTGRES_PASSWORD}"),
    ),
)

# Profile stores references (${POSTGRES_PASSWORD}), not actual values
print(profile.kwargs_dict["password"])  

# Connection resolves environment variables when created
conn = profile.get_con()
```

## Common patterns

### Creating and using profiles

```{python}
#| eval: false

# Create profile
profile = Profile(
    con_name="postgres",
    kwargs_tuple=(
        ("host", "${POSTGRES_HOST}"),
        ("port", 5432),
        ("database", "postgres"),
        ("user", "${POSTGRES_USER}"),
        ("password", "${POSTGRES_PASSWORD}"),
    ),
)

# Save profile
profile.save(alias="postgres_example", clobber=True)

# Load profile
loaded_profile = Profile.load("postgres_example")

# Create connection
connection = loaded_profile.get_con()

# Use connection
tables = connection.list_tables()
```

### Cloning and modifying profiles

```{python}
#| eval: false

# Clone with modifications
cloned_profile = profile.clone(**{"connect_timeout": 10})

# Save modified profile
cloned_profile.save(alias="postgres_other_db", clobber=True)
```

### Working with multiple profiles

```{python}
#| eval: false

# Get profiles collection
profiles = Profiles()

# List all profiles
all_profiles = profiles.list()

# Access by name
my_profile = profiles.get("postgres_example")
```

## Security considerations

- Profiles never store actual values of environment variables, only references
- Sensitive information is resolved only when connections are created
- Profiles can be safely committed to version control
- Always use environment variables for passwords, API keys, and tokens

## Overview — Backends — Environment variables

This document provides a comprehensive reference for all environment variables used in the project's `.template` files. These template files serve as blueprints for configuring various services and connections in the Xorq ecosystem.

## Overview

Xorq uses several `.env.{service}.template` files to define environment variables for different services. These templates help ensure consistent configuration across different environments while keeping sensitive credentials secure through environment variable references.

## Template files

### PostgreSQL configuration (`.env.postgres.template`)

PostgreSQL database connection parameters.

| Variable | Description | Required | 
|----------|-------------|----------|
| `POSTGRES_USER` | Database username | Yes | 
| `POSTGRES_PASSWORD` | Database password | Yes |
| `POSTGRES_HOST` | Database host address | Yes |
| `POSTGRES_PORT` | Database port number | Yes | 
| `POSTGRES_DATABASE` | Database name | Yes |


### AWS configuration (`.env.aws.template`)

Amazon Web Services authentication and configuration.

| Variable | Description | Required |  
|----------|-------------|----------|
| `AWS_ACCESS_KEY_ID` | AWS access key identifier | Yes |  
| `AWS_SECRET_ACCESS_KEY` | AWS secret access key | Yes |  
| `AWS_SESSION_TOKEN` | AWS session token (for temporary credentials) | No |  
| `AWS_ALLOW_HTTP` | Allow HTTP connections (not recommended for production) | No |  
| `AWS_REGION` | AWS region for services | Yes | 

**Security Note**—Never commit actual AWS credentials to version control. Use IAM roles or temporary credentials when possible.

### Snowflake configuration (`.env.snowflake.template`)

Snowflake data warehouse connection parameters.

| Variable | Description | Required | 
|----------|-------------|----------|
| `SNOWFLAKE_USER` | Snowflake username | Yes | 
| `SNOWFLAKE_PASSWORD` | Snowflake password | Yes | 
| `SNOWFLAKE_ACCOUNT` | Snowflake account identifier | Yes | 
| `SNOWFLAKE_ROLE` | Snowflake role to assume | Yes | 
| `SNOWFLAKE_WAREHOUSE` | Snowflake warehouse to use | Yes | 

**Note**—Snowflake accounts can be identified by their account locator or account URL format.

### Weather API configuration (`.env.weather.template`)

OpenWeatherMap API integration settings.

| Variable | Description | Required | Default | 
|----------|-------------|----------|---------|
| `OPENWEATHER_API_KEY` | OpenWeatherMap API key | Yes | - | 
| `WEATHER_FEATURES_PORT` | Port for weather features service | No | `8819` | 
| `WEATHER_API_URL` | OpenWeatherMap API base URL | No | `https://api.openweathermap.org/data/2.5/weather` |

**API Key**—Obtain from [OpenWeatherMap](https://openweathermap.org/api) after creating an account.

### PyIceberg configuration (`.env.pyiceberg.template`)

Apache Iceberg table format configuration for PyIceberg.

| Variable | Description | Required | Default | 
|----------|-------------|----------|---------|
| `ICEBERG_WAREHOUSE_PATH` | Path to Iceberg warehouse | No | `warehouse` | 
| `ICEBERG_NAMESPACE` | Default namespace for tables | No | `default` | 
| `ICEBERG_CATALOG_NAME` | Name of the Iceberg catalog | No | `default` | 
| `ICEBERG_CATALOG_TYPE` | Type of catalog backend | No | `sql` | 
| `ICEBERG_URI` | Connection URI for the catalog | Yes | - | 

**Catalog Types:**
- `sql`: SQL-based catalog (requires database connection)
- `hive`: Hive Metastore catalog
- `glue`: AWS Glue Data Catalog

### Xorq configuration (`.env.xorq.template`)

Core Xorq app settings.

| Variable | Description | Required | Default | Example |
|----------|-------------|----------|---------|---------|
| `XORQ_CACHE_DIR` | Directory for caching data | No | `~/.cache/xorq` | 
| `XORQ_DEFAULT_RELATIVE_PATH` | Default relative path for data | No | `parquet` | 
| `XORQ_PROFILE_DIR` | Directory for storing profiles | No | `~/.config/xorq/profiles` | 
| `XORQ_DEBUG` | Enable debug mode | No | `False` | 
| `XORQ_CACHE_KEY_PREFIX` | Prefix for cache keys | No | `letsql_cache-` |
| `XORQ_LOG_LEVEL` | Log level for file logging (`DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`, or `OFF` to turn off) | No | `INFO` |

**Note**—These variables use `export` statements in the template, indicating they should be set as shell environment variables.

## Overview — Backends — Supported backends

Xorq currently supports:

1. **In-Process Engines**
    - DuckDB
    - DataFusion
    - Pandas
2. **Distributed Engines**
    - Trino
    - Snowflake
    - BigQuery
3. **Arrow Flight SQL Engines**
    - GizmoSQL (DuckDB over Arrow Flight SQL)

### Backend selection guidelines

Choose engines based on their strengths:

1. **DuckDB**: Local processing, AsOf joins, efficient file formats
2. **DataFusion**: Custom UDFs, streaming processing
3. **Trino**: Distributed queries, federation, security
4. **Snowflake/BigQuery**: Managed infrastructure, scalability
5. **GizmoSQL**: DuckDB over Arrow Flight SQL, client-server analytics

### Data transfer

Data movement between engines is handled through:

1. **Arrow Flight**: Zero-copy data transfer protocol
2. **Memory Management**: Automatic spilling to disk
3. **Batching**: Efficient chunk-based processing

## Overview — Serialization format — Expression Format

Xorq uses a YAML-based serialization format for storing expression artifacts. This human-readable format enables:

- **Version Control:** Store expression definitions alongside your code with meaningful diffs
- **Lineage Tracking:** Complete schemas and dependency information preserved in YAML
- **Debugging:** Debug outputs for SQL and metadata included in artifacts
- **Roundtrip Compatibility:** Expressions can roundtrip back and forth from Ibis expressions

When you build an expression, Xorq generates several YAML files:

- expr.yaml - Complete expression definition with schemas and lineage
- deferred_reads.yaml - Information about deferred data reads
- profiles.yaml - Backend profile configurations
- sql.yaml - Generated SQL for debugging

## Overview — UDF system — User-Defined Exchange Functions

## What are UDXFs?

User-Defined Exchange Functions (UDXFs) are a specialized type of user-defined function in Xorq that enable **distributed data processing** using Apache Arrow Flight protocol. Unlike traditional UDFs that operate within a single process, UDXFs execute custom Python logic in separate processes or even remote services, making them ideal for:

- **External API integrations** (calling REST APIs, databases, or third-party services)
- **Resource-intensive computations** (ML model inference, heavy transformations)
- **Microservice architectures** (deploying models as standalone services)
- **Process isolation** (running untrusted or memory-intensive code safely)

### Key components

1. **Process Function**: Your custom Python logic that transforms pandas DataFrames
2. **Schema Validation**: Input/output schema specifications for type safety
3. **Flight Server**: Hosts your processing function as a network service
4. **Flight Client**: Transfers data to/from the server automatically

## Creating UDXFs

### Basic syntax

```python
import xorq.api as xo
from xorq.expr.relations import flight_udxf

# Define your processing function
@curry
def my_transform(df: pd.DataFrame, param1, param2):
    # Your custom logic here
    return transformed_df

# Create the UDXF
my_udxf = flight_udxf(
    process_df=my_transform(param1=value1, param2=value2),
    maybe_schema_in=input_schema.to_pyarrow(),
    maybe_schema_out=output_schema.to_pyarrow(),
    name="MyTransformation"
)

# Apply to data
result = input_expr.pipe(my_udxf)
```

### Schema specifications

UDXFs require explicit schema definitions for type safety and optimization:

```python
# Define input requirements
input_schema = xo.schema({"text": "string", "id": "int64"})

# Define output schema  
output_schema = xo.schema({
    "text": "string", 
    "id": "int64", 
    "sentiment": "string"
})

# Schema validation functions
maybe_schema_in = schema_contains(input_schema)  # Validates required columns
maybe_schema_out = schema_concat(output_schema)  # Adds new columns
```

## Overview — Caching — Cache API overview

The classes exported from `xorq.caching` are named combinations of a **strategy**
(how the cache key is computed) and a **storage** (where the cached data lives).
For the conceptual treatment of those two axes, see
[Caching](../concepts/understanding_xorq/intelligent_caching_system.qmd).

## Class matrix

| Cache class | Strategy | Storage |
|---|---|---|
| [`SourceCache`](../reference/SourceCache.qmd) | `ModificationTimeStrategy` | `SourceStorage` |
| [`SourceSnapshotCache`](../reference/SourceSnapshotCache.qmd) | `SnapshotStrategy` | `SourceStorage` |
| [`ParquetCache`](../reference/ParquetCache.qmd) | `ModificationTimeStrategy` | `ParquetStorage` |
| [`ParquetSnapshotCache`](../reference/ParquetSnapshotCache.qmd) | `SnapshotStrategy` | `ParquetStorage` |
| `ParquetTTLSnapshotCache` | `SnapshotStrategy` | `ParquetTTLStorage` |
| `GCSCache` | `ModificationTimeStrategy` | `GCStorage` |

- `ModificationTimeStrategy` folds backend-specific change metadata into the key,
  so the cache invalidates when the source data changes.
- `SnapshotStrategy` keys on expression structure only (table name, path,
  schema), so the first cached result is served until you delete it.

## Building a cache

Build any cache with `from_kwargs(...)` and attach it with `.cache(...)`. The
keyword arguments are forwarded to the storage; the accepted arguments and their
defaults are documented on each class's reference page (linked in the matrix
above). The example below caches a small real dataset into the embedded
(local DataFusion) backend with `SourceCache`, then to a local Parquet file with
`ParquetCache`:

```{python}
import xorq.api as xo
from xorq.caching import SourceCache, ParquetCache

con = xo.connect()  # embedded DataFusion backend, runs locally
penguins = xo.examples.penguins.fetch(backend=con)
expr = penguins.group_by("species").agg(n=xo._.count())

# Cache as a table in the source backend
expr.cache(SourceCache.from_kwargs(source=con)).execute()
```

```{python}
# Cache to a local Parquet file (default ~/.cache/xorq/parquet/)
expr.cache(ParquetCache.from_kwargs(source=con)).execute()
```

`GCSCache` follows the same shape but writes to a Google Cloud Storage bucket,
so it needs a bucket name and isn't runnable locally:

```python
from xorq.caching import GCSCache

# Requires a GCS bucket reachable via gcsfs
expr.cache(GCSCache.from_kwargs(bucket_name="my-bucket", source=con))
```

`SourceStorage` checks existence with `key in source.tables` and writes the
result as a table named after the key. `ParquetStorage` checks whether the
`.parquet` file exists, writing to a `.tmp` file first and atomically renaming
on success. `ParquetTTLStorage` additionally treats a file older than its `ttl`
as expired.

## Backend invalidation signals

`ModificationTimeStrategy` reads a per-backend change signal into the key hash:

| Backend | Invalidation signal | ADBC ingestion | Notes |
|---|---|---|---|
| Postgres | `reltuples` from `pg_class` | Yes (`PgADBC`) | Estimate; may lag writes until `ANALYZE` runs |
| Snowflake | `LAST_ALTERED` timestamp | Yes (`SnowflakeADBC`) | Updates on any DDL/DML |
| BigQuery | `last_modified_time` from `__TABLES__` | — | |
| DuckDB | File metadata / data bytes | No (direct register) | In-memory hashes data |
| DataFusion / Xorq | Data bytes (in-memory) | No (direct register) | |
| SQLite | `COUNT(*)` and `MAX(id)` (on-disk) / data bytes (in-memory) | Yes (`SQLiteADBC`) | On-disk requires an `id` column |
| PyIceberg | Snapshot IDs | No (uses `create_table`) | Tied to Iceberg's snapshot model |
| Deferred file reads (parquet, CSV) | inode mtime, size, number | — | |
| Deferred URL reads (S3/GCS) | `last_modified`, `size`, `e_tag` | — | |
| Deferred URL reads (HTTP) | `Last-Modified`, `Content-Length` | — | |

For a frozen result regardless of source changes, use a snapshot variant
(`SourceSnapshotCache`, `ParquetSnapshotCache`).