CANN/cann-recipes-train：基于昇腾NPU的多轮工具调用代码强化学习

Code RL with Multi-Turn Tool Calling on Ascend NPUs【免费下载链接】cann-recipes-train本项目针对LLM与多模态模型训练业务中的典型模型、加速算法提供基于CANN平台的优化样例项目地址: https://gitcode.com/cann/cann-recipes-trainOverviewThis project is based on the Qwen3-1.7B model, employing the verl code sandbox service adapted for the Ascend platform, achieving efficient and stablelong-context multi-turn tool-call Code RLtraining. Our contributions include:Developed a scalable distributed code execution sandbox ScaleBox, supporting large-scale multi-node deployment, mainstream RL framework compatibility, and efficient unified evaluation across multiple models and benchmarks.Provided a unified deployment image combining verl and ScaleBox, supporting co-deployment of ScaleBox service and verl training tasks on a single node, with zero-cost migration to Huawei Cloud ModelArts.Validated Code RL training using the verl framework and ScaleBox sandbox on Ascend NPUs.Organized SFT data and SFT strategy for Coding Toolcall, and introduced multi-turn tool-call Coding Agent training in RL (the first open-source verl-based Coding Agent RL Recipe supporting multi-turn tool calling).Patches to integrate speculative decoding (EAGLE3 and Suffix) into the verl vLLM-Ascend rollout pipeline, with per-step metrics collection — draft token count, accepted token count, draft acceptance rate, mean acceptance length, and per-position acceptance rates.Validation of EAGLE3 speculative decoding within the multi-turn tool-call Code RL training loop on Ascend NPUs, achieving30% improvement in end-to-end throughputand25% reduction in training step timewithout loss of accuracy.ScaleBox 是一个可扩展的分布式代码执行沙盒其核心特性包括可扩展的分布式代码沙盒体系支持多机分布式沙盒部署与请求负载均衡支持单元测试并行与实例级并行面向 Code RL 的统一训练接口和评估套件提供高效的批量评估接口common_evaluate_batch相较于run_code通过单次请求处理多个测试用例显著提升训练效率内置对 LiveCodeBench、HumanEval、MBPP 等主流代码评测基准的支持实现一键式快速评估灵活的 Special Judge 判题机制支持自定义判题逻辑能够灵活适应具有多种正确答案的复杂编程题目Hardware RequirementsAtlas A2/A3 series, single node with 8 NPUs.Software RequirementsThe base recipe and SD extension share the same verl commit but differ in vLLM version. The SD extension uses vLLM0.13.0and vLLM-Ascendv0.13.0, which bring more stable speculative decoding support and an async implementation compared to0.11.0. Note that the software versions below reflect the tested environment — CANN8.3.RC1is expected to work for the SD extension as well.ComponentBase RecipeSD ExtensionEnvironmentDockerCondaverlcommitc651b7b(based on v0.7.0.dev)commitc651b7b(based on v0.7.0.dev)vllm0.11.00.13.0vllm-ascendv0.11.0rc1v0.13.0CANN8.3.RC18.5.0File Structure├── patches │ ├── verl # verl patch directory │ │ ├── 0001-verl-feature-improve_rl_usability.patch # General Code RL usability improvements (shared) │ │ ├── 0002-enable-tool-agent-loop.patch # Multi-turn tool-call support (shared) │ │ ├── 0003-toolcall-reward.patch # Tool-call reward (base recipe) │ │ └── 0004-enable-specrl-clean.patch # Suffix/EAGLE3 speculative decoding integration (SD extension) │ └── vllm │ └── 0001-enable-sprl.patch # vLLM-side EAGLE3 speculative decoding support (SD extension) ├── figures │ ├── evaluation_progress.png # Evaluation scores across training checkpoints (base) │ ├── training_progress.png # Training metrics progress (base) │ ├── sd_nosd_accuracy.png # Accuracy comparison: spec decode vs. no spec decode (SD) │ ├── throughput_speedup.png # Throughput speedup results (SD) │ └── acceptance_rate_overall.png # Draft acceptance rate across RL training steps (SD) ├── tool_config │ └── scalebox_tool_config.yaml # ScaleBox tool-call configuration (shared) ├── build_dataset.py # RL training dataset construction script (shared) ├── filter_sft_data.py # SFT tool-call dataset construction script (base) ├── scalebox.py # Custom reward function for ScaleBox integration (shared) ├── download_eagle.py # EAGLE3 draft model download script (SD extension) ├── run_code_rl_demo.sh # RL training script (base recipe) ├── run_multi_turn_livecodebench_eval.sh # Multi-turn LiveCodeBench evaluation script (base) ├── run_toolcall_sft_demo.sh # Multi-turn tool-call SFT training script (base) ├── spec_rl_run.sh # RL training script with speculative decoding (SD extension) ├── no_spec_rl_run.sh # RL training script without speculative decoding — baseline (SD extension) ├── process_all_the_logs_sprl.py # Log processing and metrics analysis script (SD extension) └── README.md # This documentPart 1: Base Recipe — Multi-Turn Tool-Call Code RLEnvironment SetupBuild Docker ImagesBuild the verl image supporting Code RL. Refer toverl.Dockerfileandverl_sandbox.Dockerfilefrom theagent_rl/qwen2_code_rlexample:docker build --networkhost -f verl.Dockerfile -t verl:main-c651b7b-py311-cann8.3.RC1 .Clone ScaleBox and build the combined verl ScaleBox image:git clone https://link.gitcode.com/i/cabdcdb331cef587028f0fd703a28949 docker build --networkhost -f verl_sandbox.Dockerfile -t verl_sandbox:main-c651b7b-py311-cann8.3.RC1 .Set Up verlClone verl and check out the specified commit:git clone https://github.com/volcengine/verl cd verl git checkout c651b7b4207e408875f132c4226969ef3495d408 cd ..Apply patches. The following modifications are included:Add support forcode_contestsdata source inprime reward managerReduce concurrent process count inprime reward managerfrom 64 to 32 to avoid sandbox resource contentionExtend task timeout inprime reward managerfrom 300s to 3000s to support code execution with larger batchesEnhanced logging during training for easier debuggingSupport for multi-turn tool-call Coding training logicAdded Toolcall reward to improve training stabilitygit apply patches/verl/0001-verl-feature-improve_rl_usability.patch git apply patches/verl/0002-enable-tool-agent-loop.patch git apply patches/verl/0003-toolcall-reward.patchDeploy ScaleBoxStart the combined verl_sandbox container:docker run -it --privileged --namestart_verl_sandbox --user root --network host \ --shm-size 500g \ --device/dev/davinci0 \ --device/dev/davinci1 \ --device/dev/davinci2 \ --device/dev/davinci3 \ --device/dev/davinci4 \ --device/dev/davinci5 \ --device/dev/davinci6 \ --device/dev/davinci7 \ --device/dev/davinci_manager \ --device/dev/hisi_hdc \ --device /dev/devmm_svm \ -v /usr/local/dcmi:/usr/local/dcmi \ -v /usr/bin/hccn_tool:/usr/bin/hccn_tool \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /etc/hccn.conf:/etc/hccn.conf \ -v /sys/fs/cgroup:/sys/fs/cgroup:ro \ verl_sandbox:main-c651b7b-py311-cann8.3.RC1 /bin/bashActivate the ScaleBox environment:source /home/ma-user/miniconda3/bin/activate sandbox-baseDeploy ScaleBox. The following command is for single-node Code RL training. For distributed deployment options, refer to the ScaleBox repository:export HOST0.0.0.0 # Server host address export PORT8080 # Service port export WORKERS32 # Number of Uvicorn parallel workers export MAX_MEM50000000 # Maximum memory per process cd ScaleBox make run-online deploy_${HOST}:${PORT}.log 21 Verify the service is running:curl http://localhost:8080/run_code \ -H Content-Type: application/json \ --data-raw {code: print(\Hello, world!\), language: python}Expected response:{status:Success,message:,compile_result:null,run_result:{status:Finished,execution_time:0.02984905242919922,return_code:0,stdout:Hello, world!\n,stderr:}Dataset PreparationSFT Tool-Call DataBased on Gen-Verse/Open-AgentRL-SFT-3K, this filters multi-turn Python tool-call reasoning data and converts it for RL training:python build_toolcall_sft_data.pyRL DataBased on PrimeIntellect/verifiable-coding-problems, this filters high-quality Python code samples as RL training data (verifiable-coding-problems-python-only):python build_rl_dataset.pySFT Fine-TuningDownload model weights:hf download Qwen/Qwen3-1.7B --local-dir Qwen/Qwen3-1.7BRun SFT usingrun_toolcall_sft_demo.sh, adjusting default model and data paths as needed:source /home/ma-user/miniconda3/bin/activate base mkdir -p log/sft_run_log bash run_toolcall_sft_demo.shSelect the sft_step_50 checkpoint and merge the trained model weights:python3 -m verl.model_merger merge \ --backend fsdp \ --local_dir checkpoint/multiturn-toolcall-sft-qwen-3-1b/global_step_50 \ --target_dir checkpoint/multiturn-toolcall-sft-qwen-3-1b/global_step_50/huggingfaceReinforcement Learning TrainingThe RL training script isrun_code_rl_demo.sh. Adjust the default model weights and data paths as needed:bash run_code_rl_demo.shTraining ResultsThe figures below show training metrics: model scores on training data (no repeated data), inference length and clip ratio, and tool-call interaction rounds.Model EvaluationThis experiment evaluates the models code generation capability on the LiveCodeBench dataset, following inference settings from DeepSeek-R1.Evaluation settings:release_version: v5start_date: 2024-08-01code_execution: ScaleBoxInference settings:n: 4temperature: 0.6top_p: 0.95max_tokens: 32768StepsLiveCodeBench (Pass1)2016.034016.746018.088018.6310019.1912020.1414021.3416024.4518026.2020025.9722026.3624028.39Part 2: Speculative Decoding ExtensionThis section describes how to enable EAGLE3 speculative decoding on top of the base recipe. It requires an updated vLLM version and a Conda-based environment instead of Docker.Our analysis shows the rollout phase accounts for up to78.3% of total RL step time(2816.6s out of 3596.5s per step on Qwen3-1.7B). Speculative decoding directly addresses this bottleneck by accelerating token generation during vLLM rollout, targeting a≥25% end-to-end training speedupwithout accuracy degradation.Note:0001-verl-feature-improve_rl_usability.patch,0002-enable-tool-agent-loop.patch,build_dataset.py, andscalebox.pyare shared with the base recipe unchanged. The remaining files in this section are new additions specific to the SD extension. The SFT fine-tuning and tool-call rewarding steps described in Part 1 are not required for the Speculative Decoding extension. The SD extension uses the public Qwen3-1.7B model weights directly from HuggingFace.Environment Setup1. Create Conda Environmentconda create -n verl-specrl python3.11 -y conda activate verl-specrl source /path/to/CANN_8.5.0/ascend-toolkit/set_env.sh source /path/to/CANN_8.5.0/nnal/atb/set_env.sh2. Install vLLMgit clone --depth 1 --branch v0.13.0 https://github.com/vllm-project/vllm.git cd vllm VLLM_TARGET_DEVICEempty pip install -v -e . cd ..3. Install vLLM-Ascendgit clone --depth 1 --branch v0.13.0 https://github.com/vllm-project/vllm-ascend.git cd vllm-ascend pip install decorator python -m pip install -U pip setuptools wheel python -m pip install -U cmake ninja pybind11 python -m pip install -U setuptools-scm8 pip install --no-cache-dir torch2.8.0 torch-npu2.8.0 pip install torchvision0.23.0 --no-deps pip install -e . --no-build-isolation --no-deps # vllm-ascend commit id: 6281c1207a7a499e9f23a42b3a1e7027469f2b10 cd ..4. Install verlgit clone https://github.com/volcengine/verl cd verl git checkout c651b7b4207e408875f132c4226969ef3495d408 pip install -r requirements-npu.txt pip install click8.2.1 pip install githttps://github.com/ShaohonChen/PyExt.gitpy311support pip install -e . cd ..5. Apply Patches# verl patches — run from inside the verl directory git apply ../patches/verl/0001-verl-feature-improve_rl_usability.patch git apply ../patches/verl/0002-enable-tool-agent-loop.patch git apply ../patches/verl/0004-enable-specrl-clean.patch # vLLM patch — run from inside the vllm directory cd /path/to/vllm git apply /path/to/cann-recipes-train/agent_rl/qwen3_code_toolcall/patches/vllm/0001-enable-eagle-sprl.patch cd ..6. Fix Dependenciespip install numba pip uninstall triton-ascend triton -y pip install transformers4.57.6 pip install setuptools80.10.2 pip install decorator pip install arctic-inference0.1.1Deploy ScaleBoxconda create -n scalebox python3.11 -y conda activate scalebox git clone https://github.com/icip-cas/ScaleBox.git cd ScaleBox pip config set global.index-url https://mirrors.aliyun.com/pypi/simple/ pip config set global.trusted-host mirrors.aliyun.com pip install -U pip setuptools wheel pip install -r requirements.txt pip install databases pip install aiosqliteexport HOST0.0.0.0 export PORT8080 export WORKERS32 export MAX_MEM50000000 cd ScaleBox make run-online deploy_${HOST}:${PORT}.log 21 Verify the service is running:curl http://localhost:8080/run_code \ -H Content-Type: application/json \ --data-raw {code: print(\Hello, world!\), language: python}Expected response:{status:Success,message:,compile_result:null,run_result:{status:Finished,execution_time:0.02984905242919922,return_code:0,stdout:Hello, world!\n,stderr:}}Dataset PreparationInherited from the base recipe — runpython build_dataset.pyas described in Part 1.Model PreparationDownload the target model and EAGLE3 draft model weights:python download_eagle.pyReinforcement Learning TrainingBefore running, set the required paths at the top of the respective script.Forno_spec_rl_run.sh:VariableDescriptionMODEL_PATHPath to Qwen3-1.7B target model weightsDATA_PATHPath to RL training datasetASCEND_HOME_TOOLKITPath to CANN toolkit (e.g./path/to/CANN_8.5.0/)Forspec_rl_run.sh:VariableDescriptionMODEL_PATHPath to Qwen3-1.7B target model weightsDRAFT_MODEL_PATHPath to EAGLE3 draft model weightsDATA_PATHPath to RL training datasetASCEND_HOME_TOOLKITPath to CANN toolkit (e.g./path/to/CANN_8.5.0/)Run baseline RL training without speculative decoding:bash no_spec_rl_run.shRun RL training with EAGLE3 speculative decoding:bash spec_rl_run.shProcess Training LogsOnce training is complete, collect all logs associated with an experiment into a single folder, then run:python process_all_the_logs_sprl.py path/to/logs/ -o path/to/output/combined_metrics.csvRun for both the SD and baseline runs to generate CSVs for comparison.Training ResultsSuffix and EAGLE3 speculative decoding achieve up to38% improvement in end-to-end throughputand25% reduction in training step timewith no loss of accuracy compared to the baseline.Since the EAGLE3 drafter is frozen during RL training, the draft acceptance rate gradually decreases as the actor policy drifts from the drafters training distribution:Future Work for Speculative DecodingNgram Speculative Decoding fixes: Fix a bug in the Ngram speculative decoding.Block Verification: Enable block verification in the rejection sampling module of speculative decoding.Online Drafter Training: Investigate co-training the EAGLE3 drafter alongside the actor during RL to counteract acceptance rate decay caused by policy drift.Elastic Speculation: Explore adaptively adjusting speculative decoding parameters (e.g. number of speculation tokens) during RL training.SD Recipe Evolution: As SD-specific features (block verification, online drafter training, elastic speculation, online MTP) mature, we will revisit whether a dedicated directory for the SD recipe is warranted.【免费下载链接】cann-recipes-train本项目针对LLM与多模态模型训练业务中的典型模型、加速算法提供基于CANN平台的优化样例项目地址: https://gitcode.com/cann/cann-recipes-train创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考