catalog rebase

Re-derive an entry over its live sources and catalog it as a new entry.

Run it once xorq catalog check-sources reports a changed source. The recorded expression is rebuilt over the schemas the sources have now; the old entry is never edited or removed. The new entry keeps the old one’s wheels and requirements, and records the old one as its ancestor, which xorq catalog show prints.

–dry-run writes nothing to the catalog (it may fetch the entry’s archive, as any rebase does): it prints the hash a rebase would produce, says what it would do to the aliases, and exits 3. It does not run the sync’s pull, so a synced rebase can still refuse an alias the pull moved. –no-add writes the rebased build to BUILDS_DIR/, bundle and ancestor included, and prints its path instead of cataloging it, and exits 3; xorq catalog add on that directory yields the same hash, as long as the directory keeps its name, which add takes as the entry’s. A directory already at that path is never replaced: the rebase is refused before building. It does not sync, and takes no –alias, –move-aliases or –only-alias, since there is no entry to name. An interrupted –no-add can leave a hidden BUILDS_DIR/..* stage, safe to delete. With no drift, or no source that can be probed, neither writes anything and both print the entry’s own name.

No alias moves unless asked. –move-aliases moves every alias the old entry has once the sync’s pull is in, and names any the pull took off it; –only-alias moves just the ones it names, each of which must be on the old entry or, after the pull, already on the new one (a rerun after a failed push). –alias registers another, and is refused if it points at any entry but the old or the new one; one already on the old entry moves. Aliases left on the old entry are named on stderr. Alias options apply only to a new entry: with no drift nothing is written and they are ignored, which stderr says, though –only-alias must still name the old entry’s aliases.

Prints the resulting entry name on stdout, and the detail on stderr. With no drift that name is the entry’s own, and nothing is committed. An entry none of whose sources can be probed is left alone, since nothing can be refreshed: its own name is printed, and stderr says drift was not ruled out.

A renamed column is a removed one to rebase, unless –rename says which live column it is now: --rename t a b reads t’s recorded column a from its live column b, and every op over t still sees a. SOURCE is a source’s name as check-sources prints it; OLD is a column it recorded, and a later rebase of the result keeps following the rename. Rebase never guesses a rename; a conflict over a lost column lists what each changed source lost and gained. A –rename is refused, every one in a single run, when SOURCE names no source, or sources on several connections; when OLD is not recorded, or is still live (so also with no drift); when NEW is not live; or when no source can be probed. One that renames a column of its source twice, maps two to one, or renames one to itself is an invalid option. A NEW the source had recorded too is allowed, with a warning: rebase can’t tell a mistyped NEW from a column dropped and replaced by the renamed one.

Exit codes: 0 no drift, drift not ruled out (no source could be probed), or the rebase succeeded 1 refused, or failed: the name does not resolve, the catalog does not open, the entry is pinned, some but not all of its sources cannot be probed, the entry was built on another Python minor or records none (–ignore-venv-mismatch overrides it, with a warning), –only-alias names an alias on neither the old entry nor the new one (an unregistered one before the sweep), –alias names an alias on another entry than the old or new one (once there is a new entry), ENTRY is an alias the pull moved or removed, a –rename is refused (see above), a probed source matched no source of the loaded expression, –no-add found its BUILDS_DIR/ taken, or a write failed (the new entry and its alias moves are rolled back locally; if that fails too, the error names what is left, and –json says rollback-failed) 2 a source was unreachable or unreadable, its database is missing, or its reads disagree on its live schema; the record is unreadable or lacks its wheel or requirements; the sync’s pull failed; or the options were invalid (–move-aliases with –only-alias, –no-add with –dry-run or an alias option, –builds-dir without –no-add, a –rename that repeats a column of its source or renames one to itself); nothing written 3 a rebase was derived but not cataloged: –dry-run printed its hash, or –no-add its build directory 4 conflict: an op no longer fits its new inputs, or a source’s table is gone; stderr names the op (if one failed) and the sources involved, with their recorded and live schemas, and any columns each changed source lost and gained; nothing written 5 rebased locally, but the sync’s push failed: the new entry is printed and its alias moves are kept; xorq catalog sync (pull, then push) finishes it

With –json, stdout is one document, printed once the rebase has ended and nothing else; its state and exit_code sit at the root, and the exit code is the process’s. Every outcome of rebasing a resolved entry prints one, a failed write included; a warning the rebase raises goes in its warnings, not on stderr. As with check-sources --json, a rebase that never started prints none: a name that does not resolve or a catalog that does not open exits 1, and invalid options exit 2, with the error on stderr. Its shape is documented in xorq.catalog.rebase.

Usage

xorq catalog rebase ENTRY [OPTIONS]

Arguments

  • ENTRY—An entry name or alias.

Options

Option Default Description
--sync / --no-sync --sync Push the catalog to its remotes after the operation.
--cache-dir $XORQ_CACHE_DIR or ~/.cache/xorq Directory for parquet cache files.
-a, --alias none Register this alias for the new entry.
--move-aliases off Move all of the source entry’s aliases onto the new entry.
--only-alias none Move only this alias of the source entry onto the new entry (repeatable).
--ignore-venv-mismatch off Proceed even if the entries’ wheel requirements or Python minor differ.
--dry-run off Report the hash a rebase would produce, and catalog nothing.
--add / --no-add --add Catalog the rebased build; –no-add only writes its build directory.
--builds-dir none Where –no-add writes the build directory (default: builds); only with –no-add.
--rename none Read SOURCE’s recorded column OLD from its live column NEW, keeping the name OLD (repeatable).
--json off Output as JSON.

Examples

xorq catalog rebase prod-matches
xorq catalog rebase prod-matches --move-aliases
xorq catalog rebase prod-matches --only-alias prod -a matches-v2
xorq catalog rebase prod-matches --json
xorq catalog rebase prod-matches --dry-run
xorq catalog rebase prod-matches --no-add --builds-dir builds
xorq catalog rebase prod-matches --rename matches team_id team

See also