[linux] Qwen2Tokenizer报错 transformers版本问题

上午没问题，下午pull了新代码，就有了报错。。

发现是transformers版本问题。但。。其实我都默认安的是最新版本。。

也许这就是人生吧。。 

报错：

File "/Pai-Megatron-Patch/megatron_patch/tokenizer/__init__.py", line 213, in build_tokenizertokenizer = _Qwen2Tokenizer(args.load, args.extra_vocab_size)File "/Pai-Megatron-Patch/megatron_patch/tokenizer/__init__.py", line 174, in __init__tokenizer = _Qwen2Tokenizer(args.load, args.extra_vocab_size)File "/Pai-Megatron-Patch/megatron_patch/tokenizer/__init__.py", line 174, in __init__tokenizer = _Qwen2Tokenizer(args.load, args.extra_vocab_size)File "/Pai-Megatron-Patch/megatron_patch/tokenizer/__init__.py", line 174, in __init__tokenizer = _Qwen2Tokenizer(args.load, args.extra_vocab_size)File "/Pai-Megatron-Patch/megatron_patch/tokenizer/__init__.py", line 174, in __init__tokenizer = _Qwen2Tokenizer(args.load, args.extra_vocab_size)File "/Pai-Megatron-Patch/megatron_patch/tokenizer/__init__.py", line 174, in __init__tokenizer = _Qwen2Tokenizer(args.load, args.extra_vocab_size)File "/Pai-Megatron-Patch/megatron_patch/tokenizer/__init__.py", line 174, in __init__tokenizer = _Qwen2Tokenizer(args.load, args.extra_vocab_size)File "/Pai-Megatron-Patch/megatron_patch/tokenizer/__init__.py", line 174, in __init__tokenizer = _Qwen2Tokenizer(args.load, args.extra_vocab_size)File "/Pai-Megatron-Patch/megatron_patch/tokenizer/__init__.py", line 174, in __init__self.tokenizer = AutoTokenizer.from_pretrained(File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py", line 784, in from_pretrainedself.tokenizer = AutoTokenizer.from_pretrained(File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py", line 784, in from_pretrainedself.tokenizer = AutoTokenizer.from_pretrained(File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py", line 784, in from_pretrainedself.tokenizer = AutoTokenizer.from_pretrained(File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py", line 784, in from_pretrainedself.tokenizer = AutoTokenizer.from_pretrained(File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py", line 784, in from_pretrainedself.tokenizer = AutoTokenizer.from_pretrained(File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py", line 784, in from_pretrainedself.tokenizer = AutoTokenizer.from_pretrained(File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py", line 784, in from_pretrainedself.tokenizer = AutoTokenizer.from_pretrained(File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py", line 784, in from_pretrainedraise ValueError(raise ValueError(
ValueError    ValueErrorraise ValueError(: : 
Tokenizer class Qwen2Tokenizer does not exist or is not currently imported.Tokenizer class Qwen2Tokenizer does not exist or is not currently imported.raise ValueError(raise ValueError(    ValueError
raise ValueError(

解决方案：

原本的：

pip install -U transformers -i https://pypi.doubanio.com/simple/

改成：

pip install  transformers==4.40  -i https://mirrors.aliyun.com/pypi/simple/

Successfully installed huggingface-hub-0.23.3 safetensors-0.4.3 tokenizers-0.19.1 transformers-4.41.2