GLM-5.3-Flash

RL recipe for GLM-5.3-Flash, a KDA + DSA hybrid MoE with mHC hyper-connections and NoPE MLA.

Implementation: radixark/miles#2786. It goes with the SGLang sglang-miles-glm53next branch and radixark/Megatron-LM#89; the image in section 3 pins all three.

1. Model Introduction

GLM-5.3-Flash (model_type: glm5_next) is a 45-layer KDA + DSA hybrid MoE. It is a different architecture from the 744 B GLM5 and GLM5.2 flagships, not a smaller cut of them.

• 45 layers, hybrid: 34 KDA linear-attention layers + 11 DSA sparse-attention layers.
• 288-expert MoE, sigmoid routing at top-8; the first 3 layers are dense.
• mHC hyper-connections at every block.
• NoPE MLA — multi-latent attention with the positional half of the QK head empty.
• kpool-compressed lightning indexer picks which keys the DSA layers attend.
• Hidden 4096, FFN 12288, 64 attention heads, vocab 154880, rotary base 800000.
• MTP is dropped for training.

2. Supported Variants

Variant	--model-name	Layers
Full	GLM-5.3-Flash	45
4-layer slice	GLM-5.3-Flash-4layer	4 (launcher default)

3. Environment Setup

Use docker.io/radixark/miles:glm53next — the rolling radixark/miles:dev image with the three moving parts checked out at the versions this recipe was built against, multi-arch so the same tag serves GB300 and x86 nodes.

Component	Pinned at
miles	#2786 1cd14c00
SGLang	sglang-miles-glm53next 9a26e749
Megatron-LM	#89 e8f57451

hf download zai-org/GLM-5.3-Flash --local-dir /root/models/GLM-5.3-Flash
hf download --repo-type dataset zhuzilin/dapo-math-17k --local-dir /root/datasets/dapo-math-17k

The reference checkpoint has to be converted first — --ref-load resolves to <--ckpt-dir>/<megatron_model_type>_torch_dist:

cd /root/miles
MODEL_ARGS_LINE="$(python3 miles/utils/external_utils/model_args_utils.py glm5.3-flash)" || exit 1
read -ra MODEL_ARGS <<< "${MODEL_ARGS_LINE}"
CONVERT_KEEP_PP1=1 CUDA_DEVICE_MAX_CONNECTIONS=1 PYTHONPATH=/root/Megatron-LM \
  torchrun --nproc-per-node 8 \
    tools/convert_hf_to_torch_dist.py "${MODEL_ARGS[@]}" \
    --hf-checkpoint /root/models/GLM-5.3-Flash \
    --save          /root/ckpt/glm5.3-flash_torch_dist

4. Launch

Bring up a ray cluster across the nodes, export MILES_SCRIPT_EXTERNAL_RAY=1, then on the head node:

cd /root/miles
python scripts/run_glm5_3_flash.py train \
  --model-name GLM-5.3-Flash \
  --num-nodes 16 --num-gpus-per-node 4 \
  --num-rollout 20 --rollout-max-response-len 4096

Smoke slice on one node:

python scripts/run_glm5_3_flash.py train --num-nodes 1 --num-gpus-per-node 8

Shape	TP	PP	EP	Rollout engine
16 × 4 (full)	8	4	16	8 GPUs, SGLang TP 8 / EP 8
8 × 4 (full)	8	4	8	8 GPUs, SGLang TP 8 / EP 8
2 × 4 or 1 × 8 (4-layer slice)	2	2	2	4 GPUs, SGLang TP 4 / EP 4

The PP-4 shapes run 11 / 11 / 11 / 12 layers per stage, since 45 does not divide by 4. GRPO on DAPO-Math-17k, Adam at lr 1e-6, max_tokens_per_gpu 8192, full uniform recompute. Rollout is colocated, with the trainer offloaded to disk; both DSA paths run on tilelang and the KV cache is BF16. Routing replay turns on with --enable-r3.

5. What a Healthy Run Looks Like

From the validation run in #2786 — 16 nodes × 4 GB300, DAPO on DAPO-Math-17k:

Metric	Observed
train/train_rollout_logprob_abs_diff	0.0068 – 0.0106 across the first 11 rollouts
rollout/raw_reward	0.5 → 0.94 within 10 rollouts
train/ppo_kl	~2.6e-4
train/grad_norm	0.31 – 0.49

train/train_rollout_logprob_abs_diff is the one to read first on a fresh bring-up: it covers the KDA, DSA and hyper-connection paths at once.

6. LoRA RL

scripts/run_glm5_3_flash_lora.py trains a LoRA adapter through native LoRA (--megatron-to-hf-mode raw): the trainer is the glm5_next model of section 4, loaded from the torch_dist base of section 3, and miles.backends.megatron_utils.lora.native attaches the adapters. With --lora-base-cpu-backup SGLang keeps the frozen base across colocated offload and onload, so each step syncs only the adapter.

Layer type	Adapted HF projections
KDA (34 layers)	q/k/v/o_proj, b_proj, forget_gate.f_a/f_b_proj, g_a/g_b_proj
DSA (11 layers)	q_a/q_b_proj, kv_a_proj_with_mqa, o_proj
MLP	dense and shared-expert gate/up/down_proj

The 288 routed experts stay frozen, since native LoRA has no per-expert adapter. kv_b_proj stays frozen too: the DSA attention absorbs its weight into the query and output without calling the module, so an adapter on it would never run. The launcher’s --target-modules attn,mlp --exclude-modules "*.mlp.experts.*,*.self_attn.kv_b_proj" selects exactly the table, and selecting a frozen projection fails at argument validation. The KDA conv1d, A_log, dt_bias and o_norm, the DSA indexer and the mHC parameters are not LoRA targets.

Serving the adapter needs SGLang LoRA on the KDA projections, which sglang-miles does not have yet; it is in sgl-project/sglang#44862.

python scripts/run_glm5_3_flash_lora.py prepare --model-name GLM-5.3-Flash-4layer
python scripts/run_glm5_3_flash_lora.py train --model-name GLM-5.3-Flash-4layer \
  --num-nodes 1 --num-gpus-per-node 8

# full model, 3 x 8 GPUs: TP 8, EP 24, PP 1, one colocated 8-GPU engine per node
export MILES_SCRIPT_EXTERNAL_RAY=1   # after `ray start` on every node
python scripts/run_glm5_3_flash_lora.py train --model-name GLM-5.3-Flash \
  --num-nodes 3 --num-gpus-per-node 8