How catladder pipelines work

All pipeline files are generated from catladder.ts — fix pipeline problems in the config (see catladder-config skill), never in the generated YAML.

Triggers and environments

Trigger	Runs on	Deploys to (default)
mainBranch	push to the main branch	dev
mr	merge/pull requests	review (one app per MR/PR)
taggedRelease	git tags	stage, prod
{ branch: "<name>" }	push to that branch	envs declaring it via on

The trigger of each env is its on in the top-level environments config (project-wide — components cannot diverge), defaulting from the env type as in the table. An env with on: { branch: "next" } gets its own pipeline (gitlab: rules on $CI_COMMIT_BRANCH; github: a catladder-branch-<name>.yml workflow) and deploys as one stable, branch-tracking environment — unlike review apps, it is not per-instance and is stopped via the manual stop job/workflow. on: false keeps an env out of every pipeline but the env still exists (setup, secrets, catenv); environments: { <name>: false } removes it entirely.

Generated layout:

• GitLab: .gitlab-ci.yml includes generated files from .catladder-generated/gitlab/ (one per trigger).
• GitHub: workflows in .github/workflows/ (generated ones carry a "generated by catladder" header; others are unmanaged). Manual actions are per-kind workflow_dispatch workflows, visually grouped by their emoji prefix in the Actions sidebar: 🚀 catladder create release, ▶️ catladder deploy (manual-gated deploys), 🛑 catladder stop, ↩️ catladder rollback — when all of a kind's tasks target one env the name carries it (e.g. ▶️ catladder deploy prod); otherwise pick the task from the workflow's dropdown. Review apps are stopped automatically when their pull request closes (🛑 catladder stop review app); the same workflow can be dispatched with a PR number to tear one down while the pull request is still open (gitlab runs these as manual jobs inside the pipeline instead).

Stages: setup → test → build → post-build (postBuildTests against the build output, block the deploy) → deploy → verify (post-deploy checks), plus stop jobs for review-app teardown.

Superseded pipelines are cancelled

Pushing a new commit cancels the pipeline of the previous one, so the runners are not busy with results nobody will read.

Pipeline	On a new commit
mr	cancelled
mainBranch (dev)	cancelled, except a release already running
taggedRelease	runs through — never cancelled
agent runs (trigger)	runs through — never cancelled

• GitHub: the generated MR workflow sets concurrency with cancel-in-progress: true.
• GitLab: generated workflow:auto_cancel:on_new_commit: interruptible, with per-rule none for tags and agent runs. Every job carries an explicit interruptible, jobs that must finish (release jobs, agent jobs) carry interruptible: false.

GitLab requires the project setting too. The generated YAML only tunes how redundant pipelines are cancelled — it does not switch the feature on. If old pipelines keep running, check Settings > CI/CD > General pipelines > Auto-cancel redundant pipelines; with that checkbox off nothing is ever cancelled. (catladder project-doctor does not check this.)

Adding a hand-written GitLab job? Set interruptible explicitly. GitLab defaults it to false, and a non-interruptible job that has started keeps itself alive — under gitlab's own default (conservative, which catladder overrides) it would shield the whole pipeline from cancellation.

When review apps deploy (reviewApps)

By default every merge/pull request deploys its review apps. Projects with many MRs/PRs (e.g. opened by agents) can put the review delivery chains — docker image → deploy → verify, for all components — behind ONE switch per MR/PR with the top-level reviewApps config. Tests, lint and audit always run; only the delivery waits.

reviewApps: {
  deploy: "optIn",   // auto (default) | manual | optIn | optOut
  // label: "catladder:review-app",       // optIn label (default)
  // skipLabel: "catladder:no-review-app", // optOut label (default)
  drafts: "checksOnly", // likeReady (default) | checksOnly (no review apps) | skip (no pipeline)
},

• optIn: deploys only while the MR/PR carries label.
• optOut: deploys unless it carries skipLabel.
• manual: never on its own — only on request.
• drafts: "checksOnly": drafts run tests, lint and audit only, the review apps deploy once the MR/PR is marked ready; "skip": drafts run nothing.
• Per-component deploy.when of the review env is ignored then.

GitHub — like gitlab, the review workflow decides at its start from the PR's labels and draft state: when the policy says deploy, the chains run in the same run (one app build). Whatever changes later goes through the ▶️ catladder deploy review workflow (catladder-deploy-review.yml, which rebuilds the app):

• when the review workflow is green but didn't deploy, its last job (▶️ trigger review deploy) reads the PR's labels/draft state again and dispatches the deploy if the policy now says so — a label set while CI ran counts. The guard trusts that dispatch (the CI is green by construction; it only verifies the PR is open and still on that head) instead of re-reading the check, which raced the checks api. Meanwhile the deploy workflow's guard of the label event defers (the CI isn't finished) and sets the commit status catladder review app to pending ("waiting for the CI"), so the PR never shows no status;
• switching the label on (opt-in: add label, opt-out: remove skipLabel) deploys right away if the PR's catladder ✅ is green; switching it off stops the review apps (🛑 catladder stop review app), and so does closing the PR — the stop first waits for a review run of the PR that may still be deploying (on close it cancels it), so a late deploy can't outlive the teardown;
• on request, any time (agents included): gh workflow run catladder-deploy-review.yml --ref <pr-branch> -f pr=<number> — explicit requests always deploy, but need the PR open and its CI green (the run reuses the green CI instead of re-running tests);
• merge gates wait for catladder review app ✅ (stable name, independent of the components), the last job of whichever run deploys. It also sets the commit status catladder review app on the PR head (success/failure; the deploy workflow marks it pending first) — dispatched runs are attached to the commit but not listed on the pull request, the commit status is. A run that doesn't deploy, or is cancelled (superseded, stopped), reports nothing — a gate must treat a missing or pending status on the current head as "wait". In the review run a skipped chain doesn't fail catladder ✅, a failed one does;
• gh pr edit <number> --add-label catladder:review-app is the agent-friendly opt-in. Labels set with the workflow's own GITHUB_TOKEN (from inside a workflow) trigger nothing — use a user or app token;
• the dispatch only works once the workflow exists on the default branch (github rule) — the PR introducing reviewApps can't dispatch yet.

A single component with env.review.deploy.when: "manual" (and no reviewApps) uses the same workflow for just that component, without the automatic trigger.

GitLab — a 🚀 deploy review job in the MR pipeline is the switch: the docker and deploy jobs of all components need it. The policy decides whether it starts on its own (label set / opt-out not set) or waits as a manual job; playing it any time deploys (early clicks wait for builds and tests). It has allow_failure: true, so an unplayed switch leaves the pipeline green. Label changes don't start a pipeline on gitlab — use yarn catladder mr review-app-on (sets the label and triggers a pipeline) / mr review-app-off. Drafts are detected by their title prefix (Draft:, [Draft], (Draft)); with drafts: "skip" marking an MR ready starts no pipeline, the next push does.

Review-app auto-stop and pinning (GitLab)

GitLab review environments stop automatically after 1 week, dev environments after 4 weeks. Configure with top-level autoStop in catladder.ts ({ review, dev }, gitlab natural language like "3 days" or "never"). Merging or closing the MR always stops its review apps, independent of the timer.

To keep an MR's review apps alive longer, pin the MR:

• yarn catladder mr pin — adds the pin label (default catladder::pin-review, configurable via autoStop.pinLabel) to the current branch's open MR and triggers a pipeline so the pin takes effect immediately. While the label is set, ALL components' review apps deploy with auto_stop_in: never.
• yarn catladder mr unpin — removes the label; the auto-stop timer re-arms with the next deploy.

The pin lives on the MR (a label), not on the environment, so it survives redeploys — unlike gitlab's per-environment pin button, which the next successful deploy resets. Adding/removing the label by hand works too; a label change takes effect from the MR's next pipeline.

GitHub has no auto-stop: review apps run until the pull request is closed (pinning does not apply there).

Job images and catci

Jobs run in catladder-provided images (🐳 catladder image <name> build jobs). Their definitions are materialized into .catladder-generated/images/ and built in the project's own registry under catladder/ (content-hashed, rebuilt only on change). Each image dir carries an IMAGE_TAG file with the current tag, so a tag change always shows up as a changed file. If jobs fail with an image pull error (…/catladder/<name>:<tag>: not found), check that the generated files are committed, and that the 🐳 catladder image <name> job ran. .catladder-generated/catci/ holds a small bundled CI companion used by generated jobs (e.g. the release security audit) plus a package.json pinning the folder to CommonJS (needed when the project's root package.json has "type": "module") — all generated, never edit.

Projects can declare their own job images under images in catladder.ts (🐳 image <name> build jobs, pushed to job-images/ in the registry) and reference them in any jobImage field via { image: "<name>" } — see the catladder-builds skill. A declared Dockerfile directory is used in place; an inline dockerfile is materialized into .catladder-generated/images/project/<name>/. Either way the build job is skipped when the content hash already exists in the registry.

GHCR image paths are lowercased (GitHub)

Image repository names may not contain uppercase characters, but GitHub's ${{ github.repository }} context interpolates the display casing of the owner and repo. For a mixed-case repository (AcmeCorp/Nautilus) catladder therefore resolves the repository from the git remote at generation time and writes the lowercased literal (ghcr.io/acmecorp/nautilus/...) into the workflows — GHCR serves that org under the lowercased namespace, so it is the correct address. An all-lowercase repository keeps the ${{ github.repository }} expression, which stays correct in forks.

If a docker job fails with invalid tag ...: repository name must be lowercase, the committed workflows were generated before this was fixed (or hand-edited): regenerate with yarn catenv and commit. When generation cannot see the git remote, pin it explicitly:

pipelines: { github: { repository: "AcmeCorp/Nautilus" } },

yarn catladder project doctor reports a mixed-case repository whose generated workflows still carry the expression.

Artifacts between jobs (GitHub)

A job's artifacts.paths are handed to the jobs that need them as one tar archive (catladder-artifacts-<job-id>.tar): an "Archive artifacts" step tars them after the job, upload-artifact uploads the archive under the job id, and the consumers get a matching "Extract artifacts" step after download-artifact. The tar keeps symlinks, file modes and repo-relative paths, as GitLab artifacts do. Plain upload-artifact loses them, which breaks Next.js/Turbopack builds in pnpm workspaces: .next/node_modules/<pkg>-<hash> are symlinks into the pnpm store, and as copied directories they fail at runtime with ERR_MODULE_NOT_FOUND. If a deployed app shows that error, check that the committed workflows contain the archive steps and regenerate if they don't.

Caching

Cache configuration is generated per build type. yarn node builds cache node_modules and the .yarn zip cache; GitLab uses mutable caches with fallback keys (review branches fall back to the dev cache), GitHub immutable lockfile-keyed caches where only the build job writes. pnpm builds cache nothing: measured on a 3800-package monorepo, moving the store or node_modules through a CI cache costs more than pnpm's from-registry install. Caching behavior is changed via the build config in catladder.ts, not in the YAML.

Releases

With releases configured, tagged releases are created by a release job (method: semantic-release or changesets). The release job also runs a dependency security audit that gates the release. See the catladder-releases skill for the full flow.

Debugging CI

Useful CLI commands (see the catladder-cli skill for invocation):

• yarn catladder project ci job-log / project ci job-open — fetch or open a CI job
• yarn catladder project doctor — detect drift between catladder.ts and the actually provisioned infrastructure (missing IAM roles, stale secrets, ...). Run this when a deploy/teardown fails with permission errors even though the config looks right; fix findings with yarn catladder project setup [component].
• yarn catladder project logs open, project k8s list-pods, project k8s get-shell — inspect the running deployments

Related skills

• catladder-config — catladder.ts structure and regeneration
• catladder-builds — build jobs and Docker images
• catladder-deploys — deploy jobs, environments and review apps
• catladder-releases — the taggedRelease flow and security-audit gate
• catladder-secrets — env vars and secret management
• catladder-cli — non-interactive CLI usage and command reference
• catladder-migrate-ci-backend — moving between GitLab CI and GitHub Actions (running both in parallel, then cutting over)