1. 创建虚拟环境,先安装cudatoolkit和cudnn
cudatoolkit版本不对容易报so库找不到的错。
具体安装参考:在conda虚拟环境中安装cuda和cudnn,解决pip install or python setup.py install 将安装包安装到全局环境中的问题(1.2部分)。 - 知乎
2. 安装对应的tensorflow版本与依赖
(bert4keras-tf22) conda list
# packages in environment at /home/anaconda3/envs/bert4keras-tf22:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main defaults
_openmp_mutex 5.1 1_gnu defaults
_tflow_select 2.1.0 gpu defaults
absl-py 0.15.0 pyhd3eb1b0_0 defaults
aiohttp 3.8.1 py38h7f8727e_1 defaults
aiosignal 1.2.0 pyhd3eb1b0_0 defaults
astor 0.8.1 py38h06a4308_0 defaults
asttokens 2.0.5 pyhd3eb1b0_0 defaults
astunparse 1.6.3 py_0 defaults
async-timeout 4.0.1 pyhd3eb1b0_0 defaults
attrs 21.4.0 pyhd3eb1b0_0 defaults
backcall 0.2.0 pyhd3eb1b0_0 defaults
blas 1.0 mkl defaults
blinker 1.4 py38h06a4308_0 defaults
brotlipy 0.7.0 py38h27cfd23_1003 defaults
c-ares 1.18.1 h7f8727e_0 defaults
ca-certificates 2022.10.11 h06a4308_0 defaults
cachetools 4.2.2 pyhd3eb1b0_0 defaults
certifi 2022.9.24 py38h06a4308_0 defaults
cffi 1.15.1 py38h74dc2b5_0 defaults
charset-normalizer 2.0.4 pyhd3eb1b0_0 defaults
click 8.0.4 py38h06a4308_0 defaults
cryptography 37.0.1 py38h9ce1e76_0 defaults
cudatoolkit 10.1.243 h6bb024c_0 defaults
cudnn 7.6.5 cuda10.1_0 defaults
cupti 10.1.168 0 defaults
dataclasses 0.8 pyh6d0b6a4_7 defaults
debugpy 1.5.1 py38h295c915_0 defaults
decorator 5.1.1 pyhd3eb1b0_0 defaults
entrypoints 0.4 py38h06a4308_0 defaults
executing 0.8.3 pyhd3eb1b0_0 defaults
frozenlist 1.2.0 py38h7f8727e_0 defaults
gast 0.3.3 py_0 defaults
google-auth 2.6.0 pyhd3eb1b0_0 defaults
google-auth-oauthlib 0.4.4 pyhd3eb1b0_0 defaults
google-pasta 0.2.0 pyhd3eb1b0_0 defaults
grpcio 1.42.0 py38hce63b2e_0 defaults
h5py 2.10.0 py38hd6299e0_1 defaults
hdf5 1.10.6 hb1b8bf9_0 defaults
idna 3.3 pyhd3eb1b0_0 defaults
importlib-metadata 4.11.3 py38h06a4308_0 defaults
intel-openmp 2021.4.0 h06a4308_3561 defaults
ipykernel 6.15.2 py38h06a4308_0 defaults
ipython 8.4.0 py38h06a4308_0 defaults
jedi 0.18.1 py38h06a4308_1 defaults
jupyter_client 7.3.5 py38h06a4308_0 defaults
jupyter_core 4.11.1 py38h06a4308_0 defaults
keras 2.3.1 0 defaults
keras-applications 1.0.8 py_1 defaults
keras-base 2.3.1 py38_0 defaults
keras-preprocessing 1.1.2 pyhd3eb1b0_0 defaults
ld_impl_linux-64 2.38 h1181459_1 defaults
libffi 3.3 he6710b0_2 defaults
libgcc-ng 11.2.0 h1234567_1 defaults
libgfortran-ng 7.5.0 ha8ba4b0_17 defaults
libgfortran4 7.5.0 ha8ba4b0_17 defaults
libgomp 11.2.0 h1234567_1 defaults
libprotobuf 3.20.1 h4ff587b_0 defaults
libsodium 1.0.18 h7b6447c_0 defaults
libstdcxx-ng 11.2.0 h1234567_1 defaults
markdown 3.3.4 py38h06a4308_0 defaults
matplotlib-inline 0.1.6 py38h06a4308_0 defaults
mkl 2021.4.0 h06a4308_640 defaults
mkl-service 2.4.0 py38h7f8727e_0 defaults
mkl_fft 1.3.1 py38hd3c417c_0 defaults
mkl_random 1.2.2 py38h51133e4_0 defaults
multidict 5.2.0 py38h7f8727e_2 defaults
ncurses 6.3 h5eee18b_3 defaults
nest-asyncio 1.5.5 py38h06a4308_0 defaults
numpy 1.21.5 py38h6c91a56_3 defaults
numpy-base 1.21.5 py38ha15fc14_3 defaults
oauthlib 3.2.0 pyhd3eb1b0_1 defaults
openssl 1.1.1s h7f8727e_0 defaults
opt_einsum 3.3.0 pyhd3eb1b0_1 defaults
packaging 21.3 pyhd3eb1b0_0 defaults
parso 0.8.3 pyhd3eb1b0_0 defaults
pexpect 4.8.0 pyhd3eb1b0_3 defaults
pickleshare 0.7.5 pyhd3eb1b0_1003 defaults
pip 22.1.2 py38h06a4308_0 defaults
prompt-toolkit 3.0.20 pyhd3eb1b0_0 defaults
protobuf 3.20.1 py38h295c915_0 defaults
psutil 5.9.0 py38h5eee18b_0 defaults
ptyprocess 0.7.0 pyhd3eb1b0_2 defaults
pure_eval 0.2.2 pyhd3eb1b0_0 defaults
pyasn1 0.4.8 pyhd3eb1b0_0 defaults
pyasn1-modules 0.2.8 py_0 defaults
pycparser 2.21 pyhd3eb1b0_0 defaults
pygments 2.11.2 pyhd3eb1b0_0 defaults
pyjwt 2.4.0 py38h06a4308_0 defaults
pyopenssl 22.0.0 pyhd3eb1b0_0 defaults
pyparsing 3.0.9 py38h06a4308_0 defaults
pysocks 1.7.1 py38h06a4308_0 defaults
python 3.8.13 h12debd9_0 defaults
python-dateutil 2.8.2 pyhd3eb1b0_0 defaults
pyyaml 6.0 py38h7f8727e_1 defaults
pyzmq 23.2.0 py38h6a678d5_0 defaults
3. tensorflow转onnx环境与依赖
(bert4keras-tf25) conda list
# packages in environment at /home/anaconda3/envs/bert4keras-tf25:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main defaults
_openmp_mutex 5.1 1_gnu defaults
absl-py 0.15.0 pypi_0 pypi
asttokens 2.0.5 pyhd3eb1b0_0 defaults
astunparse 1.6.3 pypi_0 pypi
backcall 0.2.0 pyhd3eb1b0_0 defaults
bert4keras 0.11.3 pypi_0 pypi
ca-certificates 2022.10.11 h06a4308_0 defaults
cachetools 5.2.0 pypi_0 pypi
certifi 2022.9.24 py38h06a4308_0 defaults
charset-normalizer 2.1.0 pypi_0 pypi
coloredlogs 15.0.1 pypi_0 pypi
debugpy 1.5.1 py38h295c915_0 defaults
decorator 5.1.1 pyhd3eb1b0_0 defaults
entrypoints 0.4 py38h06a4308_0 defaults
executing 0.8.3 pyhd3eb1b0_0 defaults
flatbuffers 1.12 pypi_0 pypi
gast 0.4.0 pypi_0 pypi
google-auth 2.10.0 pypi_0 pypi
google-auth-oauthlib 0.4.6 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
grpcio 1.34.1 pypi_0 pypi
h5py 3.1.0 pypi_0 pypi
humanfriendly 10.0 pypi_0 pypi
idna 3.3 pypi_0 pypi
importlib-metadata 4.12.0 pypi_0 pypi
ipykernel 6.15.2 py38h06a4308_0 defaults
ipython 8.4.0 py38h06a4308_0 defaults
jedi 0.18.1 py38h06a4308_1 defaults
jupyter_client 7.3.5 py38h06a4308_0 defaults
jupyter_core 4.11.1 py38h06a4308_0 defaults
keras 2.3.1 pypi_0 pypi
keras-applications 1.0.8 pypi_0 pypi
keras-nightly 2.5.0.dev2021032900 pypi_0 pypi
keras-preprocessing 1.1.2 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1 defaults
libffi 3.3 he6710b0_2 defaults
libgcc-ng 11.2.0 h1234567_1 defaults
libgomp 11.2.0 h1234567_1 defaults
libsodium 1.0.18 h7b6447c_0 defaults
libstdcxx-ng 11.2.0 h1234567_1 defaults
markdown 3.4.1 pypi_0 pypi
markupsafe 2.1.1 pypi_0 pypi
matplotlib-inline 0.1.6 py38h06a4308_0 defaults
mpmath 1.2.1 pypi_0 pypi
ncurses 6.3 h5eee18b_3 defaults
nest-asyncio 1.5.5 py38h06a4308_0 defaults
numpy 1.23.2 pypi_0 pypi
oauthlib 3.2.0 pypi_0 pypi
onnx 1.12.0 pypi_0 pypi
onnxruntime 1.12.1 pypi_0 pypi
openssl 1.1.1s h7f8727e_0 defaults
opt-einsum 3.3.0 pypi_0 pypi
packaging 21.3 pyhd3eb1b0_0 defaults
parso 0.8.3 pyhd3eb1b0_0 defaults
pexpect 4.8.0 pyhd3eb1b0_3 defaults
pickleshare 0.7.5 pyhd3eb1b0_1003 defaults
pip 22.1.2 py38h06a4308_0 defaults
prompt-toolkit 3.0.20 pyhd3eb1b0_0 defaults
protobuf 3.19.4 pypi_0 pypi
psutil 5.9.0 py38h5eee18b_0 defaults
ptyprocess 0.7.0 pyhd3eb1b0_2 defaults
pure_eval 0.2.2 pyhd3eb1b0_0 defaults
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pygments 2.11.2 pyhd3eb1b0_0 defaults
pyparsing 3.0.9 py38h06a4308_0 defaults
python 3.8.13 h12debd9_0 defaults
python-dateutil 2.8.2 pyhd3eb1b0_0 defaults
pyyaml 6.0 pypi_0 pypi
pyzmq 23.2.0 py38h6a678d5_0 defaults
readline 8.1.2 h7f8727e_1 defaults
requests 2.28.1 pypi_0 pypi
requests-oauthlib 1.3.1 pypi_0 pypi
rsa 4.9 pypi_0 pypi
scipy 1.9.0 pypi_0 pypi
setuptools 61.2.0 py38h06a4308_0 defaults
six 1.15.0 pypi_0 pypi
sqlite 3.39.2 h5082296_0 defaults
stack_data 0.2.0 pyhd3eb1b0_0 defaults
sympy 1.10.1 pypi_0 pypi
tensorboard 2.10.0 pypi_0 pypi
tensorboard-data-server 0.6.1 pypi_0 pypi
tensorboard-plugin-wit 1.8.1 pypi_0 pypi
tensorflow 2.5.0 pypi_0 pypi
tensorflow-estimator 2.5.0 pypi_0 pypi
termcolor 1.1.0 pypi_0 pypi
tf2onnx 1.12.1 pypi_0 pypi
tk 8.6.12 h1ccaba5_0 defaults
tornado 6.2 py38h5eee18b_0 defaults
traitlets 5.1.1 pyhd3eb1b0_0 defaults
typing-extensions 3.7.4.3 pypi_0 pypi
urllib3 1.26.11 pypi_0 pypi
wcwidth 0.2.5 pyhd3eb1b0_0 defaults
werkzeug 2.2.2 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0 defaults
wrapt 1.12.1 pypi_0 pypi
xz 5.2.5 h7f8727e_1 defaults
zeromq 4.3.4 h2531618_0 defaults
zipp 3.8.1 pypi_0 pypi
zlib 1.2.12 h7f8727e_2 defaults
特别要注意:keras-nightly 2.5.0.dev2021021100 的安装,其版本号有点特别
4. 踩坑指南
(1) so库找不到
2023-11-25 01:10:45.410281: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /path/to/cuda/lib64:/path/to/cudnn/lib64:
安装:cudatoolkit 10.1.243
(2) tf转onnx错误
2023-12-07 04:17:32,217 - ERROR - Tensorflow op [StatefulPartitionedCall/model_1/Transformer-0-MultiHeadSelfAttention/BroadcastArgs: BroadcastArgs] is not supported
安装:bert4keras 0.11.3