如何两台atlas-a2服务器物理机，基于vllm-ascend部署qwen3.5 397b-w8a8-mtp大模型

配置两台8*64的昇腾910b4服务器物理机未做任何虚拟化已经通过交换机进行互联模型和镜像下载模型https://modelscope.cn/models/Eco-Tech/Qwen3.5-35B-A3B-w8a8-mtp镜像不是openeuler系统就下不带openeuler的m.daocloud.io/quay.io/ascend/vllm-ascend:v0.18.0rc1-openeuler驱动https://www.hiascend.com/hardware/firmware-drivers/community?product4model32cann8.5.1driverAscendHDK25.5.2run文件就行上传以上所有文件到服务器的硬盘上服务器验货先分别查看两台服务器的npu状态#!/bin/bash # Check the remote switch ports for i in {0..7}; do hccn_tool -i $i -lldp -g | grep Ifname; done # Get the link status of the Ethernet ports (UP or DOWN) for i in {0..7}; do hccn_tool -i $i -link -g ; done # Check the network health status for i in {0..7}; do hccn_tool -i $i -net_health -g ; done # View the network detected IP configuration for i in {0..7}; do hccn_tool -i $i -netdetect -g ; done # View gateway configuration for i in {0..7}; do hccn_tool -i $i -gateway -g ; done # show ip addr for i in {0..7}; do hccn_tool -i $i -ip -g | grep ipaddr; done # View NPU network configuration cat /etc/hccn.conf注意如果这里面有执行错误就没法继续部署必须先找厂家解决验证两边的通信分别从两台主机的 for i in {0..7}; do hccn_tool -i $i -ip -g | grep ipaddr; done执行结果中找的找出npu的ip地址使用如下命令进行ping验证for i in {0..7};do hccn_tool -i 0 -ping -g address {其中任一npu的ip};done如果有ping不通说明服务器pcie或交换机有问题必须先解决驱动安装因为是使用docker镜像安装所以cann安装不是必须的只需要安装驱动从华为官网下载驱动如果没有商用权限就下载社区的都能用vllm-ascend:v0.18.0rc1使用cann8.5.1版本这个版本要求驱动得是25.5.2分别下载Ascend-hdk-910b-npu-firmware_7.8.0.7.220.runAscend-hdk-910b-npu-driver_25.5.2_linux-aarch64.run接下来看情况先使用npu-smi info看看驱动版本如果npu-smi执行成功了说明已经装好驱动了如果出现未找到命令这种说明没装驱动没装驱动的安装chmod x ./*.run ./Ascend-hdk-910b-npu-firmware_7.8.0.7.220.run --install ./Ascend-hdk-910b-npu-driver_25.5.2_linux-aarch64.run --install如果装了驱动但是版本低升级chmod x ./*.run ./Ascend-hdk-910b-npu-firmware_7.8.0.7.220.run --upgrade ./Ascend-hdk-910b-npu-driver_25.5.2_linux-aarch64.run --upgrade别管安装还是升级了都要做一次重启直接reboot命令就行部署前准备在两台服务器分别执行下面的脚本对两台服务器先统一状态如果中间有重启记着重新统一注意这一步如果缺少会导致多机部署的时候可能出现rank错误或卡死如果有类似问题试试执行统一状态for i in {0..7}; do hccn_tool -i $i -tls -s enable 0 ; done启动vllm-ascend的docker镜像使用脚本再两台服务器分别启动镜像注意其中的/data/qwen35_397b_w8a8_mtp换成你的实际模型权重的存放路径#!/bin/sh NAMEmodel-vllm PORT10020 DEVICES0,1,2,3,4,5,6,7 IMAGEm.daocloud.io/quay.io/ascend/vllm-ascend:v0.18.0rc1-openeuler # 加载镜像 docker run -itd -u 0 --ipchost --privileged \ -e VLLM_USE_MODELSCOPETrue -e PYTORCH_NPU_ALLOC_CONFmax_split_size_mb:256 \ -e ASCEND_RT_VISIBLE_DEVICES$DEVICES \ --name $NAME \ --nethost \ --shm-size100g \ --device /dev/davinci_manager \ --device /dev/devmm_svm \ --device /dev/hisi_hdc \ -v /usr/local/dcmi:/usr/local/dcmi \ -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \ -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \ -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /home/:/home/ \ -v /opt/data/:/opt/data/ \ -p $PORT:11025 \ -v /data/.cache:/root/.cache \ -v /data/qwen35_397b_w8a8_mtp:/model_weights -it $IMAGE bash启动模型分别进入两台的docker进去之后会卡几秒等下就行docker exec -it model-vllm /bin/bash启动master先聚焦到master的docker里执行master启动脚本start-master-service.sh#!/bin/bash export HCCL_IF_IP当前物理机ip #以下网卡名称为当前物理机ip相关联的那个网卡即可 export GLOO_SOCKET_IFNAME网卡名称 export TP_SOCKET_IFNAME网卡名称 export HCCL_SOCKET_IFNAME网卡名称 export HCCL_BUFFSIZE1024 export ASCEND_RT_VISIBLE_DEVICES0,1,2,3,4,5,6,7 export HCCL_OP_EXPANSION_MODEAIV export PYTORCH_NPU_ALLOC_CONFexpandable_segments:True export OMP_PROC_BINDfalse export OMP_NUM_THREADS100 export VLLM_USE_V11 export VLLM_ASCEND_ENABLE_FLASHCOMM10 export HCCL_INTRA_PCIE_ENABLE1 export HCCL_INTRA_ROCE_ENABLE0 export TASK_QUEUE_ENABLE1 export VLLM_API_KEYSK1234567890987654321 vllm serve /model_weights \ --served-model-name qwen35 \ --host 0.0.0.0 \ --port 11025\ --tensor-parallel-size 8 \ --data-parallel-size 2 \ --data-parallel-size-local 1 \ --data-parallel-start-rank 0 \ --data-parallel-address 当前物理机ip \ --data-parallel-rpc-port 13071 \ --max-num-seqs 64 \ --max-model-len 262144 \ --max-num-batched-tokens 16384 \ --gpu-memory-utilization 0.92 \ --enable-chunked-prefill \ --async-scheduling \ --api-key $VLLM_API_KEY \ --enable-expert-parallel \ --trust-remote-code \ --compilation-config {cudagraph_mode: FULL_DECODE_ONLY,cudagraph_capture_sizes:[1,2,4,8,16,32,64,80,96,128]} \ --mm_processor_cache_typeshm \ --quantization ascend \ --allowed-local-media-path / \ --no-enable-prefix-caching \ --speculative_config {method: qwen3_5_mtp, num_speculative_tokens: 3, enforce_eager: true} \ --additional-config {enable_cpu_binding:true,multistream_overlap_shared_expert: true} \ --default-chat-template-kwargs {enable_thinking: false} \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder再做个daemon脚本#!/bin/bash nohup ./start-master-service.sh ./service.log 21 启动worker再聚焦到worker的docker里执行worker启动脚本start-worker-service.shexport HCCL_IF_IP当前物理机ip #以下网卡名称为当前物理机ip相关联的那个网卡即可 export GLOO_SOCKET_IFNAME网卡名称 export TP_SOCKET_IFNAME网卡名称 export HCCL_SOCKET_IFNAME网卡名称 export HCCL_BUFFSIZE1024 export ASCEND_RT_VISIBLE_DEVICES0,1,2,3,4,5,6,7 export HCCL_OP_EXPANSION_MODEAIV export PYTORCH_NPU_ALLOC_CONFexpandable_segments:True export OMP_PROC_BINDfalse export OMP_NUM_THREADS100 export VLLM_USE_V11 export VLLM_ASCEND_ENABLE_FLASHCOMM10 export HCCL_INTRA_PCIE_ENABLE1 export HCCL_INTRA_ROCE_ENABLE0 export TASK_QUEUE_ENABLE1 export VLLM_API_KEYSK1234567890987654321 vllm serve /model_weights \ --served-model-name qwen35 \ --host 0.0.0.0 \ --port 11205 \ --headless \ --tensor-parallel-size 8 \ --data-parallel-size 2 \ --data-parallel-size-local 1 \ --data-parallel-start-rank 1 \ --data-parallel-address master物理机的ip地址 \ --data-parallel-rpc-port 13071 \ --max-num-seqs 64 \ --max-model-len 262144 \ --max-num-batched-tokens 16384 \ --gpu-memory-utilization 0.92 \ --enable-chunked-prefill \ --async-scheduling \ --api-key $VLLM_API_KEY \ --enable-expert-parallel \ --trust-remote-code \ --compilation-config {cudagraph_mode: FULL_DECODE_ONLY,cudagraph_capture_sizes:[1,2,4,8,16,32,64,80,96,128]} \ --mm_processor_cache_typeshm \ --quantization ascend \ --allowed-local-media-path / \ --no-enable-prefix-caching \ --speculative_config {method: qwen3_5_mtp, num_speculative_tokens: 3, enforce_eager: true} \ --additional-config {enable_cpu_binding:true,multistream_overlap_shared_expert: true} --default-chat-template-kwargs {enable_thinking: false} \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder再做个worker的daemon脚本#!/bin/bash nohup ./start-worker-service.sh ./service.log 21 分别启动master和worker的脚本先master再worker成功了master端会有(APIServer pid373) INFO: Started server process [373](APIServer pid373) INFO: Waiting for application startup.(APIServer pid373) INFO: Application startup complete.验证#!/bin/bash curl -N -XPOST http://{master节点ip}:11025/v1/chat/completions \ -H Content-type: application/json \ -H Authorization: Bearer SK1234567890987654321 \ -d { model: qwen35, messages: [{role: user, content: Please introduce the qwen3.5?}], stream: true, temperature: 0.7, top_p: 0.8, max_tokens: 1500 }首次访问会比较慢