> ## Documentation Index > Fetch the complete documentation index at: https://miles.radixark.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Docker build > The Dockerfiles, the build script, rolling and release build workflows, and how to build and push manually. GPU CI runs inside `radixark/miles`. This doc maps which Dockerfiles exist, the script that builds them, the PR-side build check, the rolling and versioned release workflows, and manual build and push behavior. ## Dockerfiles | Path | Builds | Wired into | | - | - | - | | `docker/Dockerfile` | `radixark/miles` (CUDA) | `docker-build.yml`, `release-docker.yml` | | `docker/Dockerfile.rubin` | Vera Rubin (aarch64/SM107) | `docker-build.yml` (`rubin` manual variant) | | `docker/Dockerfile.tpu` | TPU v7x through torch\_tpu (x86\_64) | `docker-build.yml` (`tpu` manual variant) | | `docker/Dockerfile.rocm` | AMD ROCm (MI35x, MI30x) | `docker-build.yml` (`rocm*-mi35x`, `rocm10-mi30x` variants) | | `docker/Dockerfile.cu12` | Deprecated CUDA 12.9 (x86\_64), unmaintained | Manual builds only | CUDA 12 image builds are retired, following [SGLang v0.5.20](https://github.com/sgl-project/sglang/releases/tag/v0.5.20). Existing CUDA 12 image tags remain available, but are no longer rebuilt, retagged, or pruned by this workflow. The old recipe is preserved in `docker/Dockerfile.cu12` with the former `cu12-x86` defaults (`v0.5.19-cu129`, `sglang-miles-v0.5.19-final`, and `cu129-x86_64`); it is deprecated and no longer maintained. ### `docker/Dockerfile` — inputs & output The Dockerfile is the build recipe: it provides the cu13 defaults and emits one image. `build.py` owns the variant → build-arg overrides (see Build script), including the ROCm base and wheels releases. The CUDA 13 defaults use `lmsysorg/sglang:v0.5.22` and the stable branch `sglang-miles`. `SGLANG_COMMIT` remains empty unless the caller pins a revision. From that checkout it installs `python/` and `radix_transfer_plugins/`, the output store package SGLang imports, both editable, and the build fails unless that package imports; its Mooncake backend is left out because `mooncake.store` needs the GPU driver. ROCm variants keep their separate base and wheels pins in `docker/build.py`. **Inputs (build-args)** | Arg | Meaning | | - | - | | `SGLANG_IMAGE_TAG` | base `lmsysorg/sglang` image tag | | `WHEELS_REPO` | prebuilt-wheels GitHub repo (`radixark/miles-wheels`) | | `WHEELS_TAG_X86` / `WHEELS_TAG_ARM64` | the two **complete** wheels release tags selected by `TARGETARCH` and installed **verbatim**. cu13 uses the rolling `cu130-torch214-x86_64` / `cu130-torch214-aarch64` releases | | `SGLANG_BRANCH` / `SGLANG_COMMIT`, `MEGATRON_REPO` / `MEGATRON_BRANCH` / `MEGATRON_COMMIT`, `MILES_COMMIT`, `SGL_ROUTER_*` | source pins for the layered repos; empty commit args follow their branch, while release builds provide exact SHAs | | `TARGETARCH` | set by buildx; also the one argument `docker/install-kube-tools.sh` takes | **Output** — one `radixark/miles` image for the platform buildx targets: the SGLang base, then the Python dependencies declared in `requirements.txt`, Megatron-LM at its branch default or caller-pinned commit, Miles, and the prebuilt wheels (`sgl-router` among them). It also carries `kubectl`, `helm` and `tmux`, which the k8s-native launch path drives the cluster with. A multi-arch build is one `buildx` run executed once per platform — `TARGETARCH` differs each time, so each arch installs its own wheels — and buildx pushes the two as a single manifest. `docker/Dockerfile.rocm` is the ROCm counterpart (build-args `GPU_ARCH`, a ROCm `SGLANG_IMAGE_TAG`, and a `WHEELS_TAG_ROCM` release from `XinyuJiangCMU/miles-wheels-rocm`). All ROCm variants build on the `lmsysorg/sglang` v0.5.22 ROCm releases pinned in `docker/build.py`, so `docker/Dockerfile.rocm` fetches the matching stable branch `sglang-miles`. `rocm724-mi35x` uses the Python 3.12 base and ROCm 7.2.4 wheels from the `rocm724-gfx950-v0.5.20` release. It sets `APPLY_ROCR_VMMFIX=1` to install the point-release-matched ROCr VMM-pause fix; ROCm 10 has the fix upstream. The MI300X / MI325X (`gfx942`) variant `rocm10-mi30x` uses the `rocm10-gfx942-v0.5.20` release. That release holds Transformer Engine, flash-attn and apex built for `gfx942` from the same sources as the `gfx950` shelf, so the variant installs every wheel prebuilt, like `rocm10-mi35x`. It also sets `AITER_PREBUILD_JIT=1`. `docker/amd/prebuild_aiter_jit.py` then builds the bf16 aiter attention kernels that sglang otherwise JIT-compiles on first use: the `mha_batch_prefill` extend modules, and `pa_ragged` decode for GQA ratios 1–16, head sizes 64 and 128, and contexts up to 64K. It builds them for `GPU_ARCH`, because the build host has no GPU. ### `docker/install-kube-tools.sh` `kubectl` and `helm` are not optional extras of a training image: the workbench installs releases with them and the launcher drives the run with them, so **every** image `docker/build.py` can build must carry them. The pinned versions and their checksum verification therefore live in one script that each training Dockerfile copies and runs, rather than in a `RUN` block per Dockerfile that only one of them ever got. Bumping either client is an edit to `docker/install-kube-tools.sh` alone. ## Build script `docker/build.py` builds and pushes the images. Select a build with `--variant` and a tag mode with `--image-tag {dev,latest,custom}`. The `VARIANTS` table is the source of truth for each variant's image, target platforms, Dockerfile, and default build-args. Repeatable `--build-arg KEY=VALUE` options are appended after those defaults, so an explicit caller override wins. | `--variant` | Tag (`--image-tag dev`) | Platforms | Notes | | - | - | - | - | | `cu13` | `radixark/miles:dev` | `linux/amd64` + `linux/arm64` | **multi-arch**, one manifest — the daily image | | `cu13-x86` | `radixark/miles:dev` | `linux/amd64` | x86-only build of the same image | | `cu13-aarch64` | `radixark/miles:dev` | `linux/arm64` | arm64-only build of the same image | | `rubin` | `radixark/miles:dev-rubin` | `linux/arm64` | Vera Rubin — `docker/Dockerfile.rubin` | | `tpu` | `radixark/miles:dev-tpu` | `linux/amd64` | TPU v7x (torch\_tpu) — `docker/Dockerfile.tpu` | | `rocm724-mi35x` | `rocm/sgl-dev:miles-rocm724-mi35x` | native | AMD MI35x (`gfx950`, ROCm 7.2.4) — `docker/Dockerfile.rocm` | | `rocm10-mi35x` | `rocm/sgl-dev:miles-rocm10-mi35x` | native | AMD MI35x (`gfx950`, ROCm 10, ROCm SDK as pip wheels) — `docker/Dockerfile.rocm` | | `rocm10-mi30x` | `rocm/sgl-dev:miles-rocm10-mi30x` | native | AMD MI300X / MI325X (`gfx942`, ROCm 10) — `docker/Dockerfile.rocm` | The cu13 variants share one multi-arch CUDA base image and differ only in platforms. `cu13` runs a single `buildx --platform linux/amd64,linux/arm64` — buildx builds both arches and pushes them as one manifest in a single shot, with the Dockerfile picking each layer's wheels by `TARGETARCH` (see Dockerfile inputs), so `docker pull` auto-selects by host arch. The **Tag** column is for `--image-tag dev`, which also pushes a timestamped `dev-` sibling; `latest` swaps the prefix to `latest`, `custom` uses `--custom-tag`. `cu13` / `cu13-x86` / `cu13-aarch64` intentionally share `radixark/miles:dev` — the daily build runs `cu13` (multi-arch), while a single-arch variant overwrites `dev` with one arch when run alone. A multi-arch build (`cu13`) needs Buildx's `docker-container` driver. Use `--push` to publish directly, or `--output type=oci,dest=/tmp/image,tar=false` to export a local OCI layout for later publication; it cannot load into the local image store. Use `cu13-x86` / `cu13-aarch64` (single-platform; the arm64 one cross-builds via QEMU on an x86 host) for local single-arch iteration. Other flags: `--push`, `--dry-run`, `--dockerfile`, `--custom-tag`, and repeatable `--build-arg KEY=VALUE`. `--context ` selects a separate build context while keeping the driver and hash reader in their original checkout. The `rubin` variant uses the existing `docker-build` runner and targets `linux/arm64`, like `cu13-aarch64`. It is not included in scheduled builds, automatic tag promotion/pruning, regular CUDA PR images or versioned releases. The `tpu` variant builds from sglang-torchtpu's release image (`SGLANG_TPU_IMAGE_TAG`): it swaps in sglang-torchtpu at `SGLANG_TPU_COMMIT` (the release's torch\_tpu, libtpu and jax), moves the one Python environment to `TORCH_VERSION` (the CUDA image's torch; the torchtitan backend needs >= 2.13), and adds torchtitan at `TORCHTITAN_COMMIT` and miles. Like `rubin`, it is manual only. ## PR build check (in `pr-test.yml`) Dockerfile changes can be build-tested on the PR itself, before merge, by selecting CUDA tests, for example with `run-ci-image`. A Dockerfile change alone does not request GPU execution or a PR image build. `pr-test.yml` calls `_build-pr-ci-image.yml` only when its final stage selection contains CUDA tests and `stage-a-cpu` satisfies its success/bypass gate. CPU-only PRs skip both the build call and its downstream `resolve-ci-image`. Both CPU stages run without waiting for images. An eligible PR keeps **one** image tag, `radixark/miles:pr-`, for its whole life, and rebuilds it only when the content that feeds it changes: | Job | What it does | | - | - | | `docker-decide` | hashes the build inputs (`docker/image_inputs.py`) and compares. Inputs equal to the base branch → no PR image, suites run on `dev`. Otherwise it reads the `miles.image-inputs` label off the published `pr-` tag: same hash → reuse it, different or absent → rebuild. Forks inspect the public tag without repository secrets | | `docker-build` | builds `cu13` for `linux/amd64` and `linux/arm64` and pushes `pr-`, stamped with the inputs hash. Same-repository rebuilds use a `docker-build` node. Fork rebuilds wait on a hosted runner for `Build Fork CI Image` to return success; otherwise this job is a hosted-runner no-op | | `resolve-ci-image` | resolves the CI image to `pr-` whenever that tag is current — whether this run built it or an earlier one did — so **every GPU suite runs inside the PR's image**; a failed build stops the matrix instead of testing a stale image. A PR image outranks a `ci-image-tag:` PR-body directive, which applies only when the PR has no image (PRs whose image inputs match the base) | | `delete-pr-tag` (`docker-pr-tag-cleanup.yml`) | removes the `pr-` tag when the PR closes; the tag stays available for re-runs while the PR is open | So a rerun, or a push that touches only source files, reuses the image the PR already has instead of rebuilding an identical one. Non-docker PRs are untouched: no PR image, matrix on `dev`, as before. `docker/image_inputs.py` is the single source of truth for what counts as an input (`docker/Dockerfile`, `docker/Dockerfile.rubin`, `docker/build.py`, `docker/install-kube-tools.sh`, `docker/verify_transformer_engine.py`, `docker/patch/**`, `requirements.txt`). `Dockerfile.rocm` is deliberately excluded — it feeds the `rocm/sgl-dev` images, not the `cu13` image built here. To rebuild an eligible PR image when its inputs did not change — a moved base image, a floating dependency, a corrupt push — add the **`rebuild-ci-image`** label alongside a CUDA test request. The label does not select tests or make a PR whose build inputs match the base eligible. Once consumed by a build, it is removed so later runs can reuse the image. ### Fork PR publication Fork `pull_request` runs do not receive Docker Hub secrets. Their image job uses the same CUDA selection, CPU A success/bypass gate, hash decision, and output contract as same-repository PRs. When a rebuild is needed, it submits an attempt-bound request and waits for trusted publication. Success continues to `resolve-ci-image` and GPU tests in the same CI attempt; a publication failure fails the image job. CPU tests are not rerun. `build-fork-ci-image.yml` starts on `PR Test`'s `workflow_run.in_progress` event and waits for the request artifact, which is readable before the source run finishes. Only a request allocates a builder. The caller waits for the matching publisher run and its build job to succeed, so an existing tag cannot satisfy a forced rebuild prematurely. The hosted handoff has a 360-minute limit covering queue and build time; the builder retains its 180-minute limit. The trusted workflow binds the request to its source run and attempt, the open fork PR and current head, the latest matching PR Test run, and the original merge SHA recorded by GitHub for `_build-pr-ci-image.yml`. It resolves selection and fast-fail policy from the request's original PR-event labels, matching the caller even after labels are removed. These labels are policy input from the source workflow, not proof of publishing authority; credential isolation is described below. It checks the CPU gate, including successful jobs retained by partial reruns, and uses trusted code to parse the PR's test registrations without importing them. A later main advance does not change the frozen merge. Host orchestration comes from the default branch. Only the fork Dockerfile and build context enter an isolated BuildKit instance, with no secret forwarding or insecure entitlements. Both architectures are exported as a local OCI layout. BuildKit is removed before Docker Hub login; the publisher copies the layout to the fixed `pr-` tag without running the image and verifies both architectures' inputs hashes. Fork edits to the host build driver, hash reader, or workflow take effect after merge. A new request cancels an older build for the same PR. During building, the trusted helper checks every 30 seconds that the source attempt remains active and stops its build process group if it ends or is superseded. The publisher rechecks the live PR and source before publication. Builder and local-output cleanup also run on failure or cancellation. A forced build consumes its original one-shot label, with the same removal-failure warning as same-repository builds. This path requires `build-fork-ci-image.yml` and its trusted helpers on the default branch. It cannot complete while introduced only on a PR branch. Existing `Approve Trusted CI` handles GitHub's contributor approval hold independently. ## Rolling Docker build (`docker-build.yml`) This workflow owns the rolling `dev` and `latest` image families. It has three jobs: * **`check-upstream`** (schedule / `simulate_schedule` only) — polls the inputs the image bakes: the HEAD SHA of sglang `sglang-miles` (`sgl-project/sglang`) and Megatron-LM `miles-main` (`radixark/Megatron-LM`) — the source branches it builds — plus a fingerprint of the selected `radixark/miles-wheels` rolling release, so a rebuilt sgl-router or other wheel also triggers a build (re-uploads to the same tag are caught by fingerprint, not commit SHA). It compares against the values cached from the last build and sets `should_build=true` if any moved. `miles` itself is intentionally not polled — that would rebuild far too often. The cron polls every 10 minutes (off the top of the hour), so an upstream move starts a build within about 30 minutes, and this check is what stops those polls from rebuilding an unchanged image, with one staleness bound: because the image also bakes a `miles` checkout, `should_build` is forced to `true` once the last triggered build is **12h** old, so `dev` never drifts more than half a day behind the `miles` repo even when sglang / Megatron / wheels are quiet. A lookup that still fails after three attempts, or returns a malformed value, fails that poll (the run goes red) without building or saving state, and the next poll retries, so a lookup that keeps failing shows up as red runs rather than silently stopping rebuilds. (The cache file's last line records the epoch of the last triggered build; it is only re-saved when a build fires.) * **`build-and-push`** (self-hosted `docker-build` runner) — calls `docker/build.py` to build + push, then conditionally points `latest` at the new `dev` and prunes old timestamped tags. * **`notify-build-failure`** — when an automatic build (schedule or push) fails, posts one card to the Miles CI Lark group with the failed job and step (`lark_notify.py docker-build-failure`). A failed manual dispatch is not reported. `build-and-push` runs when `check-upstream` was skipped, or ran and reported `should_build=true`. Automatic builds (schedule, push) share one concurrency group, so they run one at a time and at most one waits; a newer waiting build replaces an older one, and it still builds the newest heads because the Dockerfile clones them when the build starts. Manual dispatches stay outside that group. ### Triggers: automatic vs manual * **Automatic** (no human) — the **schedule** (10-minute poll, gated by `check-upstream`) and any **push to `main` that touches `docker/Dockerfile`, `docker/install-kube-tools.sh`, `docker/verify_transformer_engine.py`, or `requirements.txt`**. Both leave `--variant` empty and build `cu13` → `radixark/miles` (multi-arch). * **Manual** — `workflow_dispatch` (pick one variant — see Trigger a build yourself below) or running `docker/build.py` locally. The `rubin` and ROCm images (`rocm*-mi35x`, `rocm10-mi30x`) have **no automatic path in this repo** (ROCm nightlies live in sgl-project/sglang) (`cu13-x86` / `cu13-aarch64` just rebuild the same `dev` image single-arch). `docker/build.py` and `docker/patch/**` participate in PR image validation but are not `main`-push triggers; [Release a Version](https://miles.radixark.com/docs/developer/ci/04-release) treats them as manual preflight cases. | Trigger | `check-upstream` | builds | `latest` move | prune | | - | - | - | - | - | | schedule (10-minute poll) | runs; build if upstream moved or last build ≥ 12h ago | `cu13` | yes | yes | | push to `main` touching `docker/Dockerfile`, `docker/install-kube-tools.sh`, `docker/verify_transformer_engine.py`, or `requirements.txt` | skipped | `cu13` | no | no | | `workflow_dispatch` | skipped | the one input variant | no | no | | `workflow_dispatch` + `simulate_schedule` | runs | the one input variant | no | no | ### Tags & where it pushes All images push to **Docker Hub**. CUDA variants → `radixark/miles`; ROCm variants → `rocm/sgl-dev` (a separate namespace). The tag a build writes depends only on `--image-tag` and the variant's postfix: | variant(s) | `--image-tag dev` writes | `latest` mode writes | | - | - | - | | `cu13` / `cu13-x86` / `cu13-aarch64` | `radixark/miles:dev` + `radixark/miles:dev-` | `radixark/miles:latest` | | `rubin` | `radixark/miles:dev-rubin` (+ timestamped sibling) | `radixark/miles:latest-rubin` | | `rocm724-mi35x` / `rocm10-mi35x` | `rocm/sgl-dev:miles-rocm*-mi35x` (+ timestamped sibling) | `rocm/sgl-dev:latest-rocm*-mi35x` | | `rocm10-mi30x` | `rocm/sgl-dev:miles-rocm10-mi30x` (+ timestamped sibling) | `rocm/sgl-dev:latest-rocm10-mi30x` | What **moves a shared tag**: `--image-tag dev` overwrites `:dev` and adds a timestamped sibling; on a **scheduled** run `latest`→`dev` advances; pruning likewise runs **only on schedule**, keeping the newest 20 `dev-` tags and never deleting a tag younger than 14 days. Any `workflow_dispatch` — **including** `simulate_schedule` — writes its own tag(s) but never moves `latest` or prunes; only the real cron mutates published tags. See the trigger table above. ### Trigger a build yourself Manual builds run through `workflow_dispatch` — by default the image is built straight from the inputs you pass; with `simulate_schedule`, `check-upstream` runs first for signal but does not gate the manual build (see the `workflow_dispatch` rows in the table above for how this differs from schedule). Start one two ways: * **Web UI** — Actions → "Docker Build & Push" → **Run workflow**, then fill the inputs below. * **CLI** — `gh` dispatches on the repo's default branch; pass `--ref ` to build another branch's workflow. ```bash theme={null} gh workflow run docker-build.yml -f variant=cu13 -f image_tag=dev # custom tag instead of dev/latest: gh workflow run docker-build.yml -f variant=cu13-x86 -f image_tag=custom -f custom_tag=my-tag ``` | input | required | values / default | | - | - | - | | `variant` | yes | `cu13` / `cu13-x86` / `cu13-aarch64` / `rubin` / `rocm724-mi35x` / `rocm10-mi35x` / `rocm10-mi30x` | | `image_tag` | yes | `dev` / `latest` / `custom` | | `custom_tag` | no | tag name; required when `image_tag=custom` | | `dockerfile` | no | path to Dockerfile (default `docker/Dockerfile`) | | `simulate_schedule` | no | `true` runs the `check-upstream` poll first (default `false`) | ### Steps (`build-and-push`) 1. checkout → Buildx → install Python + typer → Docker Hub login. 2. **Build + push** via `build.py` — automatic runs build `cu13`; a manual dispatch builds only the one variant you picked. 3. **schedule only** — point `latest`→`dev`. 4. **schedule only** — prune the `dev-` series to the newest 20, keeping every tag younger than 14 days. ### Push auth & permissions Pushes use a Docker Hub credential, not your identity: * **Remote (CI)** — the workflow logs in with repo secrets `DOCKERHUB_USERNAME` / `DOCKERHUB_TOKEN`, so you don't hold the key — you just trigger the run, which needs repo **Write** access. No approval gate, but `build-and-push` runs on a `self-hosted` runner, so it only fires when one is online. * **Local** — `build.py --push` uses your own `docker login`; you need push rights to the target namespace (`radixark/miles`, or `rocm/sgl-dev` for ROCm). ## Wheels build (`build-wheels.yml`) The independent [Wheels rebuild workflow](https://miles.radixark.com/docs/developer/ci/06-build-wheels.md) publishes the prebuilt assets this image consumes. Its scheduling, manual inputs, build stages, and publishing credentials are documented separately. ## Versioned release build (`release-docker.yml`) `release-docker.yml` checks out the supplied release ref, requires its committed `release-lock.json`, and passes the locked SGLang and Megatron-LM commits plus the checked-out Miles SHA as final `--build-arg` overrides. Rolling `docker-build.yml` keeps the branch defaults. Published tags and the guarded dispatch and retry procedure are documented in [Release a Version](https://miles.radixark.com/docs/developer/ci/04-release). ## Image retention (open) `docker-build.yml` prunes `dev-`, keeping the newest 20 plus every tag younger than 14 days; `dev` / `latest` move forward. Ordinary PR, nightly, and weekly CI therefore has no durable image record. A release branch cut instead retags the newest timestamped `dev` image, or mutable `dev` when none exists, as prune-exempt `release-vX.Y.Z-ci` and records that tag in `release-lock.json`. The guarded preflight is documented in [Release a Version](https://miles.radixark.com/docs/developer/ci/04-release). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.