pyinstaller打包pytorch和transformers程序

记录使用pyinstaller打包含有pytorch和transformers库的程序时遇到的问题和解决方法。

环境和版本信息

操作系统：Windows 11

Python：3.10.12

pyinstaller：5.13.0

torch：2.2.2

transformers：4.40.1

打包过程和问题

打包命令：pyinstaller -w -F mainwindow.py

问题1：transformers找不到相关python包的元数据metadata

打包完成后在mainwindow.py所在目录下会生成一个dist文件夹，里面是打包生成的exe文件，这里是mainwindow.exe，直接双击执行该程序，会出现如下错误：

Traceback (most recent call last):File "transformers\utils\versions.py", line 102, in require_versionFile "importlib\metadata\__init__.py", line 996, in versionFile "importlib\metadata\__init__.py", line 969, in distributionFile "importlib\metadata\__init__.py", line 548, in from_name
importlib.metadata.PackageNotFoundError: No package metadata was found for tqdmDuring handling of the above exception, another exception occurred:Traceback (most recent call last):File "mainwindow.py", line 10, in <module>File "PyInstaller\loader\pyimod02_importers.py", line 385, in exec_moduleFile "task_thread.py", line 3, in <module>File "PyInstaller\loader\pyimod02_importers.py", line 385, in exec_moduleFile "image_deduplicate.py", line 15, in <module>File "PyInstaller\loader\pyimod02_importers.py", line 385, in exec_moduleFile "transformers\__init__.py", line 26, in <module>File "PyInstaller\loader\pyimod02_importers.py", line 385, in exec_moduleFile "transformers\dependency_versions_check.py", line 57, in <module>File "transformers\utils\versions.py", line 117, in require_version_coreFile "transformers\utils\versions.py", line 104, in require_version
importlib.metadata.PackageNotFoundError: No package metadata was found for The 'tqdm>=4.27' distribution was not found and is required by this application. 
Try: `pip install transformers -U` or `pip install -e '.[dev]'` if you're working with git main

从错误信息可以看出错误原因是找不到包tqdm的元数据，应该是打包时没有把这些数据一起打包过去，看了下pyinstaller的使用说明，有个参数可以解决这个问题：

--copy-metadata PACKAGENAMECopy metadata for the specified package. This option can be used multiple times.

在打包命令中加了该参数并指定tqdm包后，又出现了其他没有找到元数据的包，重复多次后，才将所有这些包的元数据都添加进去，这些包包括：tqdm、regex、requests、packaging、filelock、numpy、huggingface-hub、safetensors、pyyaml，最后的打包命令如下：

pyinstaller -w -F --copy-metadata tqdm --copy-metadata regex --copy-metadata requests --copy-metadata packaging --copy-metadata filelock --copy-metadata numpy --copy-metadata huggingface-hub --copy-metadata safetensors --copy-metadata pyyaml mainwindow.py

但是像这样在打包命令中一个一个加参数比较麻烦，其实在这个过程中可以发现，执行打包命令时，会先生成一个spec文件，这个文件是打包时pyinstaller根据传递给它的参数生成的一个python文件，里面说明了把一个py文件打包成exe程序需要执行的操作。在打包命令中添加--copy-metadata会相应地在生成spec文件中添加如下代码：

from PyInstaller.utils.hooks import copy_metadatadatas = []
datas += copy_metadata('tqdm')
datas += copy_metadata('regex')
datas += copy_metadata('requests')
datas += copy_metadata('packaging')
datas += copy_metadata('filelock')
datas += copy_metadata('numpy')
datas += copy_metadata('huggingface-hub')
datas += copy_metadata('safetensors')
datas += copy_metadata('pyyaml')

所以可以先使用pyinstaller -w -F mainwindow.py先生成spec文件，再对这个文件进行修改。使用spec文件打包只需要执行命令pyinstaller mainwindow.spec即可，不需要添加其他参数，因为参数对应的操作已经编码在spec文件中了。

问题2：没有transformers/__init__.py文件

上述问题解决后，再执行打包后的exe程序，又会出现以下错误：

Traceback (most recent call last):File "transformers\utils\import_utils.py", line 1510, in _get_moduleFile "importlib\__init__.py", line 126, in import_moduleFile "<frozen importlib._bootstrap>", line 1050, in _gcd_importFile "<frozen importlib._bootstrap>", line 1027, in _find_and_loadFile "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlockedFile "<frozen importlib._bootstrap>", line 688, in _load_unlockedFile "PyInstaller\loader\pyimod02_importers.py", line 385, in exec_moduleFile "transformers\models\auto\processing_auto.py", line 28, in <module>File "<frozen importlib._bootstrap>", line 1027, in _find_and_loadFile "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlockedFile "<frozen importlib._bootstrap>", line 688, in _load_unlockedFile "PyInstaller\loader\pyimod02_importers.py", line 385, in exec_moduleFile "transformers\processing_utils.py", line 46, in <module>File "transformers\utils\import_utils.py", line 1539, in direct_transformers_importFile "<frozen importlib._bootstrap_external>", line 879, in exec_moduleFile "<frozen importlib._bootstrap_external>", line 1016, in get_codeFile "<frozen importlib._bootstrap_external>", line 1073, in get_data
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\yuany\\AppData\\Local\\Temp\\_MEI264762\\transformers\\__init__.py'The above exception was the direct cause of the following exception:Traceback (most recent call last):File "mainwindow.py", line 10, in <module>File "<frozen importlib._bootstrap>", line 1027, in _find_and_loadFile "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlockedFile "<frozen importlib._bootstrap>", line 688, in _load_unlockedFile "PyInstaller\loader\pyimod02_importers.py", line 385, in exec_moduleFile "task_thread.py", line 3, in <module>File "<frozen importlib._bootstrap>", line 1027, in _find_and_loadFile "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlockedFile "<frozen importlib._bootstrap>", line 688, in _load_unlockedFile "PyInstaller\loader\pyimod02_importers.py", line 385, in exec_moduleFile "image_deduplicate.py", line 15, in <module>File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlistFile "transformers\utils\import_utils.py", line 1501, in __getattr__File "transformers\utils\import_utils.py", line 1500, in __getattr__File "transformers\utils\import_utils.py", line 1512, in _get_module
RuntimeError: Failed to import transformers.models.auto.processing_auto because of the following error (look up to see its traceback):
[Errno 2] No such file or directory: 'C:\\Users\\yuany\\AppData\\Local\\Temp\\_MEI264762\\transformers\\__init__.py'

这个错误说明没有把transformers相关的文件打包进去，可以通过在spec文件中添加如下代码解决该问题：

from PyInstaller.utils.hooks import copy_metadata, collect_data_files datas = []
datas += copy_metadata('tqdm')
datas += copy_metadata('regex')
datas += copy_metadata('requests')
datas += copy_metadata('packaging')
datas += copy_metadata('filelock')
datas += copy_metadata('numpy')
datas += copy_metadata('huggingface-hub')
datas += copy_metadata('safetensors')
datas += copy_metadata('pyyaml')
datas += collect_data_files('transformers', include_py_files=True, includes=['**/*.py'])

现在可以通过执行命令pyinstaller mainwindow.spec再次打包成exe程序。

问题3：找不到PyTorch和Tokenizers

再次启动生成的exe程序，又会出现如下错误：

None of PyTorch,TensorFlow >= 2.0,or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.ImportError:
CLIPTokenizerFast requires the Tokenizers library but it was not found in your environment. You can install it with:
pip install tokenizers

说明找不到pytorch和tokenizers库，以通过在spec文件中添加如下代码解决该问题：

from PyInstaller.utils.hooks import copy_metadata, collect_data_filesdatas = []
datas += copy_metadata('tqdm')
datas += copy_metadata('regex')
datas += copy_metadata('requests')
datas += copy_metadata('packaging')
datas += copy_metadata('filelock')
datas += copy_metadata('numpy')
datas += copy_metadata('huggingface-hub')
datas += copy_metadata('safetensors')
datas += copy_metadata('pyyaml')
datas += copy_metadata('tokenizers')
datas += copy_metadata('torch') 
datas += collect_data_files('transformers', include_py_files=True, includes=['**/*.py'])
datas += collect_data_files('tokenizers', include_py_files=True, includes=['**/*.py'])
datas += collect_data_files('torch', include_py_files=True, includes=['**/*.py'])

总结

最终的spec文件如下，记得需要把datas传入到Analysis和EXE中：

# -*- mode: python ; coding: utf-8 -*-
from PyInstaller.utils.hooks import copy_metadata, collect_data_filesdatas = []
datas += copy_metadata('tqdm')
datas += copy_metadata('regex')
datas += copy_metadata('requests')
datas += copy_metadata('packaging')
datas += copy_metadata('filelock')
datas += copy_metadata('numpy')
datas += copy_metadata('huggingface-hub')
datas += copy_metadata('safetensors')
datas += copy_metadata('pyyaml')
datas += copy_metadata('tokenizers')
datas += copy_metadata('torch')
datas += collect_data_files('transformers', include_py_files=True, includes=['**/*.py'])
datas += collect_data_files('tokenizers', include_py_files=True, includes=['**/*.py'])
datas += collect_data_files('torch', include_py_files=True, includes=['**/*.py'])block_cipher = Nonea = Analysis(['mainwindow.py'],pathex=[],binaries=[],datas=datas,hiddenimports=[],hookspath=[],hooksconfig={},runtime_hooks=[],excludes=[],win_no_prefer_redirects=False,win_private_assemblies=False,cipher=block_cipher,noarchive=False,
)
pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)exe = EXE(pyz,a.scripts,a.binaries,a.zipfiles,a.datas,[],name='mainwindow',debug=False,bootloader_ignore_signals=False,strip=False,upx=True,upx_exclude=[],runtime_tmpdir=None,console=False,disable_windowed_traceback=False,argv_emulation=False,target_arch=None,codesign_identity=None,entitlements_file=None,
)