Qwen3.5 MoE

Launch recipe for Qwen3.5-35B-A3B with MTP training and EAGLE speculative rollout.

1. Model Introduction

Qwen3.5-35B-A3B is the MoE branch of the Qwen3.5 line — 3 B active / 35 B total — combining the gated-attention architecture with a built-in MTP head.

Key highlights:

• Sparse MoE: 3 B active out of 35 B total parameters.
• Attention-output gate: shared with the Qwen3.5 dense series, with FP32-preserved A_log.
• Multi-Token Prediction (MTP): --mtp-num-layers 1 baked into the model config; the recipe trains the MTP head and uses EAGLE speculative decoding at rollout.
• Single-node footprint: full recipe fits on 1 × 8 GPU.

2. Supported Variants

Model	Active / Total	HF ID
Qwen3.5-35B-A3B	3 B / 35 B	Qwen/Qwen3.5-35B-A3B

3. Environment Setup

3.1 Download model + datasets

hf download --repo-type dataset zhuzilin/dapo-math-17k --local-dir /root/datasets/dapo-math-17k
hf download --repo-type dataset zhuzilin/aime-2024     --local-dir /root/datasets/aime-2024

3.2 HF → Megatron torch_dist conversion

cd /root/miles
MODEL_ARGS_LINE="$(python3 miles/utils/external_utils/model_args_utils.py qwen3.5-35B-A3B)" || exit 1
read -ra MODEL_ARGS <<< "${MODEL_ARGS_LINE}"
PYTHONPATH=/root/Megatron-LM torchrun --nproc-per-node 8 \
   tools/convert_hf_to_torch_dist.py \
   ${MODEL_ARGS[@]} \
   --hf-checkpoint /root/models/Qwen3.5-35B-A3B \
   --save          /root/models/Qwen3.5-35B-A3B_torch_dist \
   --mtp-num-layers 1

--mtp-num-layers 1 during conversion preserves the MTP layer so it survives into Megatron format. The launcher below also runs the download and conversion itself before submitting, so this step is optional.

4. Launch

4.1 Quick start

cd /root/miles
python scripts/run_qwen3_5_35b_a3b_mtp.py \
   --parallelism tp1-ep8 \
   --rollout-max-response-len 8192 \
   --sglang-ep-size 8 \
   --sglang-max-running-requests 512 \
   --enable-eval

--parallelism also accepts tp2-cp2-ep8 (the launcher default). Checkpoints are read from --model-dir (default /root/models) and datasets from --data-dir (default /root/datasets); checkpoints are written under --output-dir (default /root/shared_data).

5. Recipe Configuration

5.1 Parallelism

TP	PP	CP	EP	expert-TP	max_tokens_per_gpu	GPUs
1	1	1	8	1	8192	8 (1 × 8)

5.2 Algorithm

GRPO with --eps-clip 0.2 --eps-clip-high 0.28 --use-kl-loss --kl-loss-coef 0.00. Plus MTP training:

--enable-mtp-training
--mtp-num-layers 1
--mtp-loss-scaling-factor 0.2

5.3 Rollout & SGLang

--rollout-num-gpus-per-engine 8
--sglang-mem-fraction-static 0.7
--sglang-ep-size 8
--sglang-cuda-graph-bs 1 2 4 8 16 24 ... 256

# mtp speculative decoding
--sglang-speculative-algorithm EAGLE
--sglang-speculative-num-steps 2
--sglang-speculative-eagle-topk 1
--sglang-speculative-num-draft-tokens 3

--sglang-max-running-requests 512

--sglang-ep-size and --sglang-max-running-requests are launcher options; leave them out and SGLang keeps its own defaults.

5.4 Optimizer

CPU Adam is enabled (--optimizer-cpu-offload --overlap-cpu-optimizer-d2h-h2d --use-precision-aware-optimizer).

5.5 Notable quirks

• The Megatron side uses --moe-token-dispatcher-type flex; DeepEP isn’t enabled here, unlike Qwen3-Next.
• The model config (scripts/models/qwen3.5-35B-A3B.py) reuses the Qwen3.5 spec: --attention-output-gate, --rotary-base 10000000, --rotary-percent 0.25, A_log kept in FP32 via the bridge. See Backends Beyond Megatron.

6. Pairs Well With

• Speculative Decoding
• Backends Beyond Megatron