智能客服——模型智商测试

测试方法看AI模型能力排行榜链接适合评估在线模型。使用在线模型需要考虑1.API是否开放2.token费用3. 国内备案4. 数据安全。自己进行能力测试适合评估需要私有化部署的模型。工具司南测评OpenCompass。魔搭社区modelscope链接。# OpenCompass 支持的数据集 humaneval, triviaqa, commonsenseqa, tydiqa, strategyqa, cmmlu, lambada, piqa, ceval, math, LCSTS, Xsum, winogrande, openbookqa, AGIEval, gsm8k, nq, race, siqa, mbpp, mmlu, hellaswag, ARC, BBH, xstory_cloze, summedits, GAOKAO-BENCH, OCNLI, cmnli # gen 数据集生成式数学、代码、翻译评测 # ppl 数据集判别式选择、判断、填空评测# 租借 AutoDL 服务器# 创建实例# 无卡开机# VSCode 远程连接# 学术加速开启代理source/etc/network_turbo# 准备 opencompass 环境# 创建虚拟环境conda create--nameopencompasspython3.10-ysource~/.bashrc conda activate opencompass# 安装 opencompass支持绝大多数数据集及模型pipinstall-Uopencompass# 新开一个终端conda activate opencompasssource/etc/network_turbocd/root/autodl-tmpgitclone https://github.com/open-compass/opencompass opencompass# 安装模型pipinstallmodelscopeexportDATASET_SOURCEModelScope# 下载 DeepSeek-R1-Distill-Qwen-1.5Bcd/root/autodl-tmp/mkdirDeepSeek-R1-Distill-Qwen-1.5B modelscope download--modeldeepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B--local_dir./DeepSeek-R1-Distill-Qwen-1.5B# 新开一个终端conda activate opencompass# 学术加速开启代理source/etc/network_turbocd/root/autodl-tmp/# 下载 DeepSeek-R1-Distill-Qwen-7BmkdirDeepSeek-R1-Distill-Qwen-7B modelscope download--modeldeepseek-ai/DeepSeek-R1-Distill-Qwen-7B--local_dir./DeepSeek-R1-Distill-Qwen-7B# 测试#两个模型的路径分别如下/root/autodl-tmp/DeepSeek-R1-Distill-Qwen-7B /root/autodl-tmp/DeepSeek-R1-Distill-Qwen-1.5Bcd/root/autodl-tmp/opencompass# 在 demo_gsm8k_chat_gen 上测试 DeepSeek-R1-Distill-Qwen-7Bpython run.py\--datasetsdemo_gsm8k_chat_gen\--hf-type chat\--hf-path /root/autodl-tmp/DeepSeek-R1-Distill-Qwen-7B\--tokenizer-path /root/autodl-tmp/DeepSeek-R1-Distill-Qwen-7B\--tokenizer-kwargspadding_sidelefttruncation_sidelefttrust_remote_codeTrue\--model-kwargsdevice_mapautotrust_remote_codeTrue\--max-seq-len8192\--max-out-len2048\--batch-size1# 在 demo_gsm8k_chat_gen 上测试 DeepSeek-R1-Distill-Qwen-7Bpython run.py\--datasetsdemo_gsm8k_chat_gen\--hf-type chat\--hf-path /root/autodl-tmp/DeepSeek-R1-Distill-Qwen-1.5B\--tokenizer-path /root/autodl-tmp/DeepSeek-R1-Distill-Qwen-1.5B\--tokenizer-kwargspadding_sidelefttruncation_sidelefttrust_remote_codeTrue\--model-kwargsdevice_mapautotrust_remote_codeTrue\--max-seq-len8192\--max-out-len2048\--batch-size1【注以上是为了方便所以就用DeepSeek-R1-Distill-Qwen-1.5B和DeepSeek-R1-Distill-Qwen-7B在demo_gsm8k_chat_gen数据集上做一个抛砖引玉。】