Multi-Engine

Move data between different engines within a single expression using into_backend().

import xorq.api as xo

# Connect to engines
pg = xo.postgres.connect_env()
db = xo.duckdb.connect()

# Load data from different sources
batting = pg.table("batting")
awards = xo.examples.awards_players.fetch(backend=db)

# Filter in respective engines
recent = batting.filter(batting.yearID == 2015)
nl_awards = awards.filter(awards.lgID == "NL")

# Move data to postgres for join
result = recent.join(
    nl_awards.into_backend(pg),
    ["playerID"]
)

result.execute()

Key features

  • Zero-copy transfers using Apache Arrow
  • Automatic optimization of data placement
  • Transparent movement between any supported engines

When to use

  • Leverage engine strengths - Use DuckDB for local analysis, Postgres for joins
  • Combine data sources - Join tables from different databases
  • Optimize performance - Move small tables to where large tables live