Bert4keras相关环境安装
2023-12-13 07:53:02
1. 创建虚拟环境，先安装cudatoolkit和cudnn
cudatoolkit版本不对容易报so库找不到的错。
具体安装参考：在conda虚拟环境中安装cuda和cudnn，解决pip install or python setup.py install 将安装包安装到全局环境中的问题（1.2部分）。 - 知乎
2. 安装对应的tensorflow版本与依赖
(bert4keras-tf22) conda list
# packages in environment at /home/anaconda3/envs/bert4keras-tf22:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main defaults
_openmp_mutex 5.1 1_gnu defaults
_tflow_select 2.1.0 gpu defaults
absl-py 0.15.0 pyhd3eb1b0_0 defaults
aiohttp 3.8.1 py38h7f8727e_1 defaults
aiosignal 1.2.0 pyhd3eb1b0_0 defaults
astor 0.8.1 py38h06a4308_0 defaults
asttokens 2.0.5 pyhd3eb1b0_0 defaults
astunparse 1.6.3 py_0 defaults
async-timeout 4.0.1 pyhd3eb1b0_0 defaults
attrs 21.4.0 pyhd3eb1b0_0 defaults
backcall 0.2.0 pyhd3eb1b0_0 defaults
blas 1.0 mkl defaults
blinker 1.4 py38h06a4308_0 defaults
brotlipy 0.7.0 py38h27cfd23_1003 defaults
c-ares 1.18.1 h7f8727e_0 defaults
ca-certificates 2022.10.11 h06a4308_0 defaults
cachetools 4.2.2 pyhd3eb1b0_0 defaults
certifi 2022.9.24 py38h06a4308_0 defaults
cffi 1.15.1 py38h74dc2b5_0 defaults
charset-normalizer 2.0.4 pyhd3eb1b0_0 defaults
click 8.0.4 py38h06a4308_0 defaults
cryptography 37.0.1 py38h9ce1e76_0 defaults
cudatoolkit 10.1.243 h6bb024c_0 defaults
cudnn 7.6.5 cuda10.1_0 defaults
cupti 10.1.168 0 defaults
dataclasses 0.8 pyh6d0b6a4_7 defaults
debugpy 1.5.1 py38h295c915_0 defaults
decorator 5.1.1 pyhd3eb1b0_0 defaults
entrypoints 0.4 py38h06a4308_0 defaults
executing 0.8.3 pyhd3eb1b0_0 defaults
frozenlist 1.2.0 py38h7f8727e_0 defaults
gast 0.3.3 py_0 defaults
google-auth 2.6.0 pyhd3eb1b0_0 defaults
google-auth-oauthlib 0.4.4 pyhd3eb1b0_0 defaults
google-pasta 0.2.0 pyhd3eb1b0_0 defaults
grpcio 1.42.0 py38hce63b2e_0 defaults
h5py 2.10.0 py38hd6299e0_1 defaults
hdf5 1.10.6 hb1b8bf9_0 defaults
idna 3.3 pyhd3eb1b0_0 defaults
importlib-metadata 4.11.3 py38h06a4308_0 defaults
intel-openmp 2021.4.0 h06a4308_3561 defaults
ipykernel 6.15.2 py38h06a4308_0 defaults
ipython 8.4.0 py38h06a4308_0 defaults
jedi 0.18.1 py38h06a4308_1 defaults
jupyter_client 7.3.5 py38h06a4308_0 defaults
jupyter_core 4.11.1 py38h06a4308_0 defaults
keras 2.3.1 0 defaults
keras-applications 1.0.8 py_1 defaults
keras-base 2.3.1 py38_0 defaults
keras-preprocessing 1.1.2 pyhd3eb1b0_0 defaults
ld_impl_linux-64 2.38 h1181459_1 defaults
libffi 3.3 he6710b0_2 defaults
libgcc-ng 11.2.0 h1234567_1 defaults
libgfortran-ng 7.5.0 ha8ba4b0_17 defaults
libgfortran4 7.5.0 ha8ba4b0_17 defaults
libgomp 11.2.0 h1234567_1 defaults
libprotobuf 3.20.1 h4ff587b_0 defaults
libsodium 1.0.18 h7b6447c_0 defaults
libstdcxx-ng 11.2.0 h1234567_1 defaults
markdown 3.3.4 py38h06a4308_0 defaults
matplotlib-inline 0.1.6 py38h06a4308_0 defaults
mkl 2021.4.0 h06a4308_640 defaults
mkl-service 2.4.0 py38h7f8727e_0 defaults
mkl_fft 1.3.1 py38hd3c417c_0 defaults
mkl_random 1.2.2 py38h51133e4_0 defaults
multidict 5.2.0 py38h7f8727e_2 defaults
ncurses 6.3 h5eee18b_3 defaults
nest-asyncio 1.5.5 py38h06a4308_0 defaults
numpy 1.21.5 py38h6c91a56_3 defaults
numpy-base 1.21.5 py38ha15fc14_3 defaults
oauthlib 3.2.0 pyhd3eb1b0_1 defaults
openssl 1.1.1s h7f8727e_0 defaults
opt_einsum 3.3.0 pyhd3eb1b0_1 defaults
packaging 21.3 pyhd3eb1b0_0 defaults
parso 0.8.3 pyhd3eb1b0_0 defaults
pexpect 4.8.0 pyhd3eb1b0_3 defaults
pickleshare 0.7.5 pyhd3eb1b0_1003 defaults
pip 22.1.2 py38h06a4308_0 defaults
prompt-toolkit 3.0.20 pyhd3eb1b0_0 defaults
protobuf 3.20.1 py38h295c915_0 defaults
psutil 5.9.0 py38h5eee18b_0 defaults
ptyprocess 0.7.0 pyhd3eb1b0_2 defaults
pure_eval 0.2.2 pyhd3eb1b0_0 defaults
pyasn1 0.4.8 pyhd3eb1b0_0 defaults
pyasn1-modules 0.2.8 py_0 defaults
pycparser 2.21 pyhd3eb1b0_0 defaults
pygments 2.11.2 pyhd3eb1b0_0 defaults
pyjwt 2.4.0 py38h06a4308_0 defaults
pyopenssl 22.0.0 pyhd3eb1b0_0 defaults
pyparsing 3.0.9 py38h06a4308_0 defaults
pysocks 1.7.1 py38h06a4308_0 defaults
python 3.8.13 h12debd9_0 defaults
python-dateutil 2.8.2 pyhd3eb1b0_0 defaults
pyyaml 6.0 py38h7f8727e_1 defaults
pyzmq 23.2.0 py38h6a678d5_0 defaults
readline 8.1.2 h7f8727e_1 defaults
requests 2.28.1 py38h06a4308_0 defaults
requests-oauthlib 1.3.0 py_0 defaults
rsa 4.7.2 pyhd3eb1b0_1 defaults
scipy 1.7.3 py38hc147768_0 defaults
setuptools 61.2.0 py38h06a4308_0 defaults
six 1.16.0 pyhd3eb1b0_1 defaults
sqlite 3.39.2 h5082296_0 defaults
stack_data 0.2.0 pyhd3eb1b0_0 defaults
tensorboard 2.8.0 py38h06a4308_0 defaults
tensorboard-data-server 0.6.0 py38hca6d32c_0 defaults
tensorboard-plugin-wit 1.8.1 py38h06a4308_0 defaults
tensorflow 2.2.0 gpu_py38hb782248_0 defaults
tensorflow-base 2.2.0 gpu_py38h83e3d50_0 defaults
tensorflow-estimator 2.6.0 pyh7b7c402_0 defaults
tensorflow-gpu 2.2.0 h0d30ee6_0 defaults
termcolor 1.1.0 py38h06a4308_1 defaults
tk 8.6.12 h1ccaba5_0 defaults
tornado 6.2 py38h5eee18b_0 defaults
traitlets 5.1.1 pyhd3eb1b0_0 defaults
typing-extensions 4.3.0 py38h06a4308_0 defaults
typing_extensions 4.3.0 py38h06a4308_0 defaults
urllib3 1.26.11 py38h06a4308_0 defaults
wcwidth 0.2.5 pyhd3eb1b0_0 defaults
werkzeug 2.0.3 pyhd3eb1b0_0 defaults
wheel 0.37.1 pyhd3eb1b0_0 defaults
wrapt 1.14.1 py38h5eee18b_0 defaults
xz 5.2.5 h7f8727e_1 defaults
yaml 0.2.5 h7b6447c_0 defaults
yarl 1.6.3 py38h27cfd23_0 defaults
zeromq 4.3.4 h2531618_0 defaults
zipp 3.8.0 py38h06a4308_0 defaults
zlib 1.2.12 h7f8727e_2 defaults3. tensorflow转onnx环境与依赖
(bert4keras-tf25) conda list
# packages in environment at /home/anaconda3/envs/bert4keras-tf25:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main defaults
_openmp_mutex 5.1 1_gnu defaults
absl-py 0.15.0 pypi_0 pypi
asttokens 2.0.5 pyhd3eb1b0_0 defaults
astunparse 1.6.3 pypi_0 pypi
backcall 0.2.0 pyhd3eb1b0_0 defaults
bert4keras 0.11.3 pypi_0 pypi
ca-certificates 2022.10.11 h06a4308_0 defaults
cachetools 5.2.0 pypi_0 pypi
certifi 2022.9.24 py38h06a4308_0 defaults
charset-normalizer 2.1.0 pypi_0 pypi
coloredlogs 15.0.1 pypi_0 pypi
debugpy 1.5.1 py38h295c915_0 defaults
decorator 5.1.1 pyhd3eb1b0_0 defaults
entrypoints 0.4 py38h06a4308_0 defaults
executing 0.8.3 pyhd3eb1b0_0 defaults
flatbuffers 1.12 pypi_0 pypi
gast 0.4.0 pypi_0 pypi
google-auth 2.10.0 pypi_0 pypi
google-auth-oauthlib 0.4.6 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
grpcio 1.34.1 pypi_0 pypi
h5py 3.1.0 pypi_0 pypi
humanfriendly 10.0 pypi_0 pypi
idna 3.3 pypi_0 pypi
importlib-metadata 4.12.0 pypi_0 pypi
ipykernel 6.15.2 py38h06a4308_0 defaults
ipython 8.4.0 py38h06a4308_0 defaults
jedi 0.18.1 py38h06a4308_1 defaults
jupyter_client 7.3.5 py38h06a4308_0 defaults
jupyter_core 4.11.1 py38h06a4308_0 defaults
keras 2.3.1 pypi_0 pypi
keras-applications 1.0.8 pypi_0 pypi
keras-nightly 2.5.0.dev2021032900 pypi_0 pypi
keras-preprocessing 1.1.2 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1 defaults
libffi 3.3 he6710b0_2 defaults
libgcc-ng 11.2.0 h1234567_1 defaults
libgomp 11.2.0 h1234567_1 defaults
libsodium 1.0.18 h7b6447c_0 defaults
libstdcxx-ng 11.2.0 h1234567_1 defaults
markdown 3.4.1 pypi_0 pypi
markupsafe 2.1.1 pypi_0 pypi
matplotlib-inline 0.1.6 py38h06a4308_0 defaults
mpmath 1.2.1 pypi_0 pypi
ncurses 6.3 h5eee18b_3 defaults
nest-asyncio 1.5.5 py38h06a4308_0 defaults
numpy 1.23.2 pypi_0 pypi
oauthlib 3.2.0 pypi_0 pypi
onnx 1.12.0 pypi_0 pypi
onnxruntime 1.12.1 pypi_0 pypi
openssl 1.1.1s h7f8727e_0 defaults
opt-einsum 3.3.0 pypi_0 pypi
packaging 21.3 pyhd3eb1b0_0 defaults
parso 0.8.3 pyhd3eb1b0_0 defaults
pexpect 4.8.0 pyhd3eb1b0_3 defaults
pickleshare 0.7.5 pyhd3eb1b0_1003 defaults
pip 22.1.2 py38h06a4308_0 defaults
prompt-toolkit 3.0.20 pyhd3eb1b0_0 defaults
protobuf 3.19.4 pypi_0 pypi
psutil 5.9.0 py38h5eee18b_0 defaults
ptyprocess 0.7.0 pyhd3eb1b0_2 defaults
pure_eval 0.2.2 pyhd3eb1b0_0 defaults
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pygments 2.11.2 pyhd3eb1b0_0 defaults
pyparsing 3.0.9 py38h06a4308_0 defaults
python 3.8.13 h12debd9_0 defaults
python-dateutil 2.8.2 pyhd3eb1b0_0 defaults
pyyaml 6.0 pypi_0 pypi
pyzmq 23.2.0 py38h6a678d5_0 defaults
readline 8.1.2 h7f8727e_1 defaults
requests 2.28.1 pypi_0 pypi
requests-oauthlib 1.3.1 pypi_0 pypi
rsa 4.9 pypi_0 pypi
scipy 1.9.0 pypi_0 pypi
setuptools 61.2.0 py38h06a4308_0 defaults
six 1.15.0 pypi_0 pypi
sqlite 3.39.2 h5082296_0 defaults
stack_data 0.2.0 pyhd3eb1b0_0 defaults
sympy 1.10.1 pypi_0 pypi
tensorboard 2.10.0 pypi_0 pypi
tensorboard-data-server 0.6.1 pypi_0 pypi
tensorboard-plugin-wit 1.8.1 pypi_0 pypi
tensorflow 2.5.0 pypi_0 pypi
tensorflow-estimator 2.5.0 pypi_0 pypi
termcolor 1.1.0 pypi_0 pypi
tf2onnx 1.12.1 pypi_0 pypi
tk 8.6.12 h1ccaba5_0 defaults
tornado 6.2 py38h5eee18b_0 defaults
traitlets 5.1.1 pyhd3eb1b0_0 defaults
typing-extensions 3.7.4.3 pypi_0 pypi
urllib3 1.26.11 pypi_0 pypi
wcwidth 0.2.5 pyhd3eb1b0_0 defaults
werkzeug 2.2.2 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0 defaults
wrapt 1.12.1 pypi_0 pypi
xz 5.2.5 h7f8727e_1 defaults
zeromq 4.3.4 h2531618_0 defaults
zipp 3.8.1 pypi_0 pypi
zlib 1.2.12 h7f8727e_2 defaults特别要注意：keras-nightly ? ? ? ? ? ? 2.5.0.dev2021021100 的安装，其版本号有点特别
4. 踩坑指南
(1) so库找不到
2023-11-25 01:10:45.410281: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /path/to/cuda/lib64:/path/to/cudnn/lib64:安装：cudatoolkit ? ? ? ? ? ? ? 10.1.243?
(2) tf转onnx错误
2023-12-07 04:17:32,217 - ERROR - Tensorflow op [StatefulPartitionedCall/model_1/Transformer-0-MultiHeadSelfAttention/BroadcastArgs: BroadcastArgs] is not supported安装：bert4keras ? ? ? ? ? ? ? ?0.11.3
(3) 线程值错误
/site-packages/keras/backend/tensorflow_backend.py", line 73, in symbolic_fn_wrapper
????if _SYMBOLIC_SCOPE.value:
AttributeError: '_thread._local' object has no attribute 'value'
将代码段注释掉：
参考：2021-05-21_if _symbolic_scope.value: attributeerror: '_thread-CSDN博客
文章来源:https://blog.csdn.net/try2find/article/details/134841066
本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。 如若内容造成侵权/违法违规/事实不符，请联系我的编程经验分享网邮箱：veading@qq.com进行投诉反馈，一经查实，立即删除！
本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。 如若内容造成侵权/违法违规/事实不符，请联系我的编程经验分享网邮箱：veading@qq.com进行投诉反馈，一经查实，立即删除！