Multi-Engine

Move data between different engines within a single expression using into_backend().

import xorq.api as xo

# Connect to engines
pg = xo.postgres.connect_env()
db = xo.duckdb.connect()

# Load data from different sources
batting = pg.table("batting")
awards = xo.examples.awards_players.fetch(backend=db)

# Filter in respective engines
recent = batting.filter(batting.yearID == 2015)
nl_awards = awards.filter(awards.lgID == "NL")

# Move data to postgres for join
result = recent.join(
    nl_awards.into_backend(pg),
    ["playerID"]
)

result.execute()

	playerID	yearID	stint	teamID	lgID	G	AB	R	H	X2B	...	IBB	HBP	SH	SF	GIDP	awardID	yearID_right	lgID_right	tie	notes
0	alvarpe01	2015	1	PIT	NL	150	437	60	106	18	...	9	2	0	4	6	Silver Slugger	2013	NL		3B
1	arenano01	2015	1	COL	NL	157	616	97	177	43	...	13	4	0	11	17	Silver Slugger	2015	NL		3B
2	arenano01	2015	1	COL	NL	157	616	97	177	43	...	13	4	0	11	17	Gold Glove	2015	NL		3B
3	arenano01	2015	1	COL	NL	157	616	97	177	43	...	13	4	0	11	17	Gold Glove	2014	NL		3B
4	arenano01	2015	1	COL	NL	157	616	97	177	43	...	13	4	0	11	17	Gold Glove	2013	NL		3B
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
273	wrighda03	2015	1	NYN	NL	38	152	24	44	7	...	0	0	0	0	4	TSN All-Star	2006	NL		3B
274	yelicch01	2015	1	MIA	NL	126	476	63	143	30	...	2	2	0	0	13	Gold Glove	2014	NL		LF
275	zimmery01	2015	1	WAS	NL	95	346	43	86	25	...	0	1	0	10	13	Silver Slugger	2010	NL		3B
276	zimmery01	2015	1	WAS	NL	95	346	43	86	25	...	0	1	0	10	13	Silver Slugger	2009	NL		3B
277	zimmery01	2015	1	WAS	NL	95	346	43	86	25	...	0	1	0	10	13	Gold Glove	2009	NL		3B

278 rows × 27 columns

Key Features

• Zero-copy transfers using Apache Arrow
• Automatic optimization of data placement
• Transparent movement between any supported engines

When to Use

• Leverage engine strengths - Use DuckDB for local analysis, Postgres for joins
• Combine data sources - Join tables from different databases
• Optimize performance - Move small tables to where large tables live