CLI

The xorq CLI provides two powerful commands, build and run, that help you separate the definition of data transformations from their execution. This separation enables:

Replicability: Define transformations once and execute them consistently across different environments with guaranteed identical results.
Serialization: Store complex queries as compiled artifacts that can be shared, versioned, and executed without the original code
Performance optimization: Pre-compile expressions to avoid repeated parsing and optimization at runtime

Prerequisites

Before starting, make sure you have xorq installed:

pip install xorq

Building Expressions

The build command compiles an Ibis expression into a reusable artifact that can be executed later.

Basic Usage

The basic syntax for the build command is:

xorq build <script_path> -e <expression_name> --builds-dir <output_directory>

Where: - <script_path> is the path to your Python script containing the expression - <expression_name> is the name of the variable holding the Ibis expression - <builds-dir> is where the artifacts will be generated (defaults to “builds”)

Example

Let’s create a simple script that defines an Ibis expression:

import xorq as xo
from xorq.common.utils.defer_utils import deferred_read_parquet
from xorq.expr.relations import into_backend


pg = xo.postgres.connect_env()
db = xo.duckdb.connect()

batting = pg.table("batting")

integer = 1

backend = xo.duckdb.connect()
awards_players = deferred_read_parquet(
    backend,
    xo.config.options.pins.get_path("awards_players"),
    table_name="award_players",
)
left = batting.filter(batting.yearID == 2015)
right = awards_players.filter(awards_players.lgID == "NL").drop("yearID", "lgID")
expr = left.join(
    into_backend(right, pg, "pg-filtered-table"), ["playerID"], how="semi"
)[["yearID", "stint"]]

Now, let’s build this expression using the CLI:

xorq build pipeline.py -e expr --builds-dir artifacts

This command will: 1. Load the pipeline.py script 2. Find the expr variable 3. Generate artifacts based on the expression 4. Save them to the artifacts directory

You should see output similar to:

Building expr from pipeline.py
Written 'expr' to artifacts/3350466c8fcd

Running Expressions

Once you’ve built an expression, you can execute it with the run command.

Basic Usage

The basic syntax for the run command is:

xorq run <build_path> --output-path <output_file> --format <output_format>

Where: - <build_path> is the path to the built expression - --output-path argument specifies where to write the results (defaults to discarding output, writing to os.devnull) - --format argument specifies the output format, which can be “csv”, “json”, or “parquet” (defaults to “parquet”)

Example

To run the expression we built earlier and save the results to a parquet file:

xorq run artifacts/2a3b46d8d3d0 --output-path results.parquet

To save the results as CSV instead:

xorq run artifacts/2a3b46d8d3d0 --output-path results.csv --format csv

Error Handling

The CLI will provide helpful error messages if:

The script doesn’t exist
The expression variable isn’t found in the script
The variable isn’t an Ibis expression
No expression name is provided for the build command
The build path doesn’t exist or isn’t a valid xorq artifact
The output format isn’t supported