基于高通跃龙IQ-9100的边端协同智能客服系统(2): 边缘端ASR/TTS模型部署实战

前文回顾在第一篇文章中我们介绍了边端协同架构的优势、高通跃龙IQ-9100平台的硬件特性以及系统整体架构设计。接下来我们将进入实战环节在IQ-9100平台上完成ASR和TTS模型的部署。1. 边缘端模型部署实战1.1 环境准备在高通跃龙IQ-9100开发板上进行模型部署前需要完成以下环境配置安装Qualcomm Linux BSP使用高通提供的Ubuntu 22.04 BSP镜像刷写开发板确保Hexagon DSP驱动正常加载安装AI运行时安装Qualcomm AI Engine Direct SDKQNN包含Hexagon NN Runtime安装Sherpa-ONNXSherpa-ONNX是新一代Kaldi团队开发的跨平台推理框架原生支持Whisper和VITS模型配置音频驱动确保ALSA或PulseAudio音频驱动正常工作配置好麦克风和扬声器# 基础环境安装命令sudoaptupdatesudoaptinstall-y\build-essential cmakegitwget\libasound2-dev portaudio19-dev\python3-dev python3-pip# 安装Qualcomm AI Engine Direct SDK (QNN)# 需要从Qualcomm开发者平台下载对应版本exportQNN_SDK_ROOT/opt/qcom/aistack/qnn/2.22.0exportPATH$QNN_SDK_ROOT/bin:$PATHexportLD_LIBRARY_PATH$QNN_SDK_ROOT/lib/aarch64-ubuntu-gcc11.4:$LD_LIBRARY_PATH# 安装Sherpa-ONNX支持QNN后端加速pip3installsherpa-onnx numpy sounddevice# 验证Hexagon NPU是否可用python3-cimport sherpa_onnx; print(Sherpa-ONNX version:, sherpa_onnx.__version__)1.2 ASR模型部署 — Whisper on IQ-91001.2.1 模型转换与量化为了在IQ-9100的Hexagon NPU上高效运行Whisper模型我们需要将PyTorch模型转换为ONNX格式并进行INT8量化以充分利用NPU的算力。# 步骤1下载预训练的Whisper ONNX模型Sherpa-ONNX已提供转换好的版本wgethttps://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-whisper-small.tar.bz2tarxvf sherpa-onnx-whisper-small.tar.bz2# 步骤2使用QNN工具链将ONNX模型转换为QNN格式可选进一步加速# 转换encoderqnn-onnx-converter\--input_networksherpa-onnx-whisper-small/encoder.onnx\--output_pathwhisper_encoder_qnn.cpp\--input_listinput_list_encoder.txt# 转换decoderqnn-onnx-converter\--input_networksherpa-onnx-whisper-small/decoder.onnx\--output_pathwhisper_decoder_qnn.cpp\--input_listinput_list_decoder.txt# 步骤3编译为Hexagon可执行库qnn-model-lib-generator\-cwhisper_encoder_qnn.cpp\-bwhisper_encoder_qnn.bin\-taarch64-ubuntu-gcc11.41.2.2 ASR推理代码使用Sherpa-ONNX框架进行实时语音识别importsherpa_onnximportsounddeviceassdimportnumpyasnpfromqueueimportQueueimportthreadingclassASREngine:def__init__(self,model_path./sherpa-onnx-whisper-small):初始化ASR引擎配置whisper模型参数self.recognizersherpa_onnx.OfflineRecognizer.from_whisper(encoderf{model_path}/encoder.onnx,decoderf{model_path}/decoder.onnx,tokensf{model_path}/tokens.txt,languagezh,num_threads4,# CPU线程数providercpu,# 可切换为qnn使用NPU加速decoding_methodgreedy_search,)self.sample_rate16000self.audio_queueQueue()self._runningFalsedefstart_stream(self,callback):启动实时语音识别流self._runningTrueself._callbackcallbackdefaudio_callback(indata,frames,time_info,status):ifself._running:self.audio_queue.put(indata[:,0].copy())self.streamsd.InputStream(samplerateself.sample_rate,channels1,dtypefloat32,blocksize4000,# 250ms per chunkcallbackaudio_callback,)self.stream.start()threading.Thread(targetself._process_loop,daemonTrue).start()def_process_loop(self):处理音频缓冲区执行VAD ASRbuffernp.array([],dtypenp.float32)silence_count0whileself._running:chunkself.audio_queue.get()buffernp.concatenate([buffer,chunk])energynp.sqrt(np.mean(chunk**2))ifenergy0.01:# 静音阈值silence_count1else:silence_count0# 检测到语音结束连续500ms静音ifsilence_count2andlen(buffer)self.sample_rate*0.5:textself._recognize(buffer)iftext.strip():self._callback(text)buffernp.array([],dtypenp.float32)silence_count0def_recognize(self,audio):执行单次语音识别streamself.recognizer.create_stream()stream.accept_waveform(self.sample_rate,audio)self.recognizer.decode_stream(stream)returnstream.result.textdefstop(self):self._runningFalseself.stream.stop()1.3 TTS模型部署 — VITS on IQ-91001.3.1 模型准备VITSVariational Inference with adversarial learning for end-to-end Text-to-Speech是一种端到端的语音合成模型以其高自然度和快速推理著称。我们使用Sherpa-ONNX提供的预训练中文VITS模型。# 下载中文VITS模型wgethttps://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/vits-zh-hf-theresa.tar.bz2tarxvf vits-zh-hf-theresa.tar.bz2# 模型文件结构# vits-zh-hf-theresa/# model.onnx # VITS模型 (~75MB)# lexicon.txt # 词典文件# tokens.txt # 音素token表# date.fst / number.fst # 文本正则化FST1.3.2 TTS推理代码使用Sherpa-ONNX框架进行中文语音合成importsherpa_onnximportsounddeviceassdimportnumpyasnpimportreclassTTSEngine:def__init__(self,model_path./vits-zh-hf-theresa):初始化TTS引擎tts_configsherpa_onnx.OfflineTtsConfig(modelsherpa_onnx.OfflineTtsModelConfig(vitssherpa_onnx.OfflineTtsVitsModelConfig(modelf{model_path}/model.onnx,lexiconf{model_path}/lexicon.txt,tokensf{model_path}/tokens.txt,data_dir,dict_dir,),),max_num_sentences1,)self.ttssherpa_onnx.OfflineTts(tts_config)self.sample_rateself.tts.sample_rate self.speaker_id0self.speed1.0defsynthesize(self,text):文本转语音返回numpy音频数组audioself.tts.generate(text,sidself.speaker_id,speedself.speed,)returnnp.array(audio.samples,dtypenp.float32)defsynthesize_and_play(self,text):合成并直接播放sentencesself._split_sentences(text)forsentenceinsentences:ifsentence.strip():audioself.synthesize(sentence)sd.play(audio,self.sample_rate)sd.wait()defsynthesize_streaming(self,text,chunk_callback):流式合成分句合成并通过回调输出降低首字延迟sentencesself._split_sentences(text)forsentenceinsentences:ifsentence.strip():audioself.synthesize(sentence)chunk_callback(audio,self.sample_rate)staticmethoddef_split_sentences(text):将长文本按标点分句便于流式合成returnre.split(r[。!.?],text)至此我们已经在边缘端成功部署了ASR和TTS模型。下一篇预告在系列文章的第三篇中我们将完成DeepSeek云端大模型的API接入并将ASR、TTS和LLM三个模块集成为一个完整的客服系统同时分享性能优化与部署清单。