Bert4keras相关环境安装

2023-12-13 07:53:02

1. 创建虚拟环境,先安装cudatoolkit和cudnn

cudatoolkit版本不对容易报so库找不到的错。

具体安装参考:在conda虚拟环境中安装cuda和cudnn,解决pip install or python setup.py install 将安装包安装到全局环境中的问题(1.2部分)。 - 知乎

2. 安装对应的tensorflow版本与依赖

(bert4keras-tf22) conda list
# packages in environment at /home/anaconda3/envs/bert4keras-tf22:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    defaults
_openmp_mutex             5.1                       1_gnu    defaults
_tflow_select             2.1.0                       gpu    defaults
absl-py                   0.15.0             pyhd3eb1b0_0    defaults
aiohttp                   3.8.1            py38h7f8727e_1    defaults
aiosignal                 1.2.0              pyhd3eb1b0_0    defaults
astor                     0.8.1            py38h06a4308_0    defaults
asttokens                 2.0.5              pyhd3eb1b0_0    defaults
astunparse                1.6.3                      py_0    defaults
async-timeout             4.0.1              pyhd3eb1b0_0    defaults
attrs                     21.4.0             pyhd3eb1b0_0    defaults
backcall                  0.2.0              pyhd3eb1b0_0    defaults
blas                      1.0                         mkl    defaults
blinker                   1.4              py38h06a4308_0    defaults
brotlipy                  0.7.0           py38h27cfd23_1003    defaults
c-ares                    1.18.1               h7f8727e_0    defaults
ca-certificates           2022.10.11           h06a4308_0    defaults
cachetools                4.2.2              pyhd3eb1b0_0    defaults
certifi                   2022.9.24        py38h06a4308_0    defaults
cffi                      1.15.1           py38h74dc2b5_0    defaults
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
click                     8.0.4            py38h06a4308_0    defaults
cryptography              37.0.1           py38h9ce1e76_0    defaults
cudatoolkit               10.1.243             h6bb024c_0    defaults
cudnn                     7.6.5                cuda10.1_0    defaults
cupti                     10.1.168                      0    defaults
dataclasses               0.8                pyh6d0b6a4_7    defaults
debugpy                   1.5.1            py38h295c915_0    defaults
decorator                 5.1.1              pyhd3eb1b0_0    defaults
entrypoints               0.4              py38h06a4308_0    defaults
executing                 0.8.3              pyhd3eb1b0_0    defaults
frozenlist                1.2.0            py38h7f8727e_0    defaults
gast                      0.3.3                      py_0    defaults
google-auth               2.6.0              pyhd3eb1b0_0    defaults
google-auth-oauthlib      0.4.4              pyhd3eb1b0_0    defaults
google-pasta              0.2.0              pyhd3eb1b0_0    defaults
grpcio                    1.42.0           py38hce63b2e_0    defaults
h5py                      2.10.0           py38hd6299e0_1    defaults
hdf5                      1.10.6               hb1b8bf9_0    defaults
idna                      3.3                pyhd3eb1b0_0    defaults
importlib-metadata        4.11.3           py38h06a4308_0    defaults
intel-openmp              2021.4.0          h06a4308_3561    defaults
ipykernel                 6.15.2           py38h06a4308_0    defaults
ipython                   8.4.0            py38h06a4308_0    defaults
jedi                      0.18.1           py38h06a4308_1    defaults
jupyter_client            7.3.5            py38h06a4308_0    defaults
jupyter_core              4.11.1           py38h06a4308_0    defaults
keras                     2.3.1                         0    defaults
keras-applications        1.0.8                      py_1    defaults
keras-base                2.3.1                    py38_0    defaults
keras-preprocessing       1.1.2              pyhd3eb1b0_0    defaults
ld_impl_linux-64          2.38                 h1181459_1    defaults
libffi                    3.3                  he6710b0_2    defaults
libgcc-ng                 11.2.0               h1234567_1    defaults
libgfortran-ng            7.5.0               ha8ba4b0_17    defaults
libgfortran4              7.5.0               ha8ba4b0_17    defaults
libgomp                   11.2.0               h1234567_1    defaults
libprotobuf               3.20.1               h4ff587b_0    defaults
libsodium                 1.0.18               h7b6447c_0    defaults
libstdcxx-ng              11.2.0               h1234567_1    defaults
markdown                  3.3.4            py38h06a4308_0    defaults
matplotlib-inline         0.1.6            py38h06a4308_0    defaults
mkl                       2021.4.0           h06a4308_640    defaults
mkl-service               2.4.0            py38h7f8727e_0    defaults
mkl_fft                   1.3.1            py38hd3c417c_0    defaults
mkl_random                1.2.2            py38h51133e4_0    defaults
multidict                 5.2.0            py38h7f8727e_2    defaults
ncurses                   6.3                  h5eee18b_3    defaults
nest-asyncio              1.5.5            py38h06a4308_0    defaults
numpy                     1.21.5           py38h6c91a56_3    defaults
numpy-base                1.21.5           py38ha15fc14_3    defaults
oauthlib                  3.2.0              pyhd3eb1b0_1    defaults
openssl                   1.1.1s               h7f8727e_0    defaults
opt_einsum                3.3.0              pyhd3eb1b0_1    defaults
packaging                 21.3               pyhd3eb1b0_0    defaults
parso                     0.8.3              pyhd3eb1b0_0    defaults
pexpect                   4.8.0              pyhd3eb1b0_3    defaults
pickleshare               0.7.5           pyhd3eb1b0_1003    defaults
pip                       22.1.2           py38h06a4308_0    defaults
prompt-toolkit            3.0.20             pyhd3eb1b0_0    defaults
protobuf                  3.20.1           py38h295c915_0    defaults
psutil                    5.9.0            py38h5eee18b_0    defaults
ptyprocess                0.7.0              pyhd3eb1b0_2    defaults
pure_eval                 0.2.2              pyhd3eb1b0_0    defaults
pyasn1                    0.4.8              pyhd3eb1b0_0    defaults
pyasn1-modules            0.2.8                      py_0    defaults
pycparser                 2.21               pyhd3eb1b0_0    defaults
pygments                  2.11.2             pyhd3eb1b0_0    defaults
pyjwt                     2.4.0            py38h06a4308_0    defaults
pyopenssl                 22.0.0             pyhd3eb1b0_0    defaults
pyparsing                 3.0.9            py38h06a4308_0    defaults
pysocks                   1.7.1            py38h06a4308_0    defaults
python                    3.8.13               h12debd9_0    defaults
python-dateutil           2.8.2              pyhd3eb1b0_0    defaults
pyyaml                    6.0              py38h7f8727e_1    defaults
pyzmq                     23.2.0           py38h6a678d5_0    defaults
readline                  8.1.2                h7f8727e_1    defaults
requests                  2.28.1           py38h06a4308_0    defaults
requests-oauthlib         1.3.0                      py_0    defaults
rsa                       4.7.2              pyhd3eb1b0_1    defaults
scipy                     1.7.3            py38hc147768_0    defaults
setuptools                61.2.0           py38h06a4308_0    defaults
six                       1.16.0             pyhd3eb1b0_1    defaults
sqlite                    3.39.2               h5082296_0    defaults
stack_data                0.2.0              pyhd3eb1b0_0    defaults
tensorboard               2.8.0            py38h06a4308_0    defaults
tensorboard-data-server   0.6.0            py38hca6d32c_0    defaults
tensorboard-plugin-wit    1.8.1            py38h06a4308_0    defaults
tensorflow                2.2.0           gpu_py38hb782248_0    defaults
tensorflow-base           2.2.0           gpu_py38h83e3d50_0    defaults
tensorflow-estimator      2.6.0              pyh7b7c402_0    defaults
tensorflow-gpu            2.2.0                h0d30ee6_0    defaults
termcolor                 1.1.0            py38h06a4308_1    defaults
tk                        8.6.12               h1ccaba5_0    defaults
tornado                   6.2              py38h5eee18b_0    defaults
traitlets                 5.1.1              pyhd3eb1b0_0    defaults
typing-extensions         4.3.0            py38h06a4308_0    defaults
typing_extensions         4.3.0            py38h06a4308_0    defaults
urllib3                   1.26.11          py38h06a4308_0    defaults
wcwidth                   0.2.5              pyhd3eb1b0_0    defaults
werkzeug                  2.0.3              pyhd3eb1b0_0    defaults
wheel                     0.37.1             pyhd3eb1b0_0    defaults
wrapt                     1.14.1           py38h5eee18b_0    defaults
xz                        5.2.5                h7f8727e_1    defaults
yaml                      0.2.5                h7b6447c_0    defaults
yarl                      1.6.3            py38h27cfd23_0    defaults
zeromq                    4.3.4                h2531618_0    defaults
zipp                      3.8.0            py38h06a4308_0    defaults
zlib                      1.2.12               h7f8727e_2    defaults

3. tensorflow转onnx环境与依赖

(bert4keras-tf25) conda list                    
# packages in environment at /home/anaconda3/envs/bert4keras-tf25:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    defaults
_openmp_mutex             5.1                       1_gnu    defaults
absl-py                   0.15.0                   pypi_0    pypi
asttokens                 2.0.5              pyhd3eb1b0_0    defaults
astunparse                1.6.3                    pypi_0    pypi
backcall                  0.2.0              pyhd3eb1b0_0    defaults
bert4keras                0.11.3                   pypi_0    pypi
ca-certificates           2022.10.11           h06a4308_0    defaults
cachetools                5.2.0                    pypi_0    pypi
certifi                   2022.9.24        py38h06a4308_0    defaults
charset-normalizer        2.1.0                    pypi_0    pypi
coloredlogs               15.0.1                   pypi_0    pypi
debugpy                   1.5.1            py38h295c915_0    defaults
decorator                 5.1.1              pyhd3eb1b0_0    defaults
entrypoints               0.4              py38h06a4308_0    defaults
executing                 0.8.3              pyhd3eb1b0_0    defaults
flatbuffers               1.12                     pypi_0    pypi
gast                      0.4.0                    pypi_0    pypi
google-auth               2.10.0                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.34.1                   pypi_0    pypi
h5py                      3.1.0                    pypi_0    pypi
humanfriendly             10.0                     pypi_0    pypi
idna                      3.3                      pypi_0    pypi
importlib-metadata        4.12.0                   pypi_0    pypi
ipykernel                 6.15.2           py38h06a4308_0    defaults
ipython                   8.4.0            py38h06a4308_0    defaults
jedi                      0.18.1           py38h06a4308_1    defaults
jupyter_client            7.3.5            py38h06a4308_0    defaults
jupyter_core              4.11.1           py38h06a4308_0    defaults
keras                     2.3.1                    pypi_0    pypi
keras-applications        1.0.8                    pypi_0    pypi
keras-nightly             2.5.0.dev2021032900          pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1    defaults
libffi                    3.3                  he6710b0_2    defaults
libgcc-ng                 11.2.0               h1234567_1    defaults
libgomp                   11.2.0               h1234567_1    defaults
libsodium                 1.0.18               h7b6447c_0    defaults
libstdcxx-ng              11.2.0               h1234567_1    defaults
markdown                  3.4.1                    pypi_0    pypi
markupsafe                2.1.1                    pypi_0    pypi
matplotlib-inline         0.1.6            py38h06a4308_0    defaults
mpmath                    1.2.1                    pypi_0    pypi
ncurses                   6.3                  h5eee18b_3    defaults
nest-asyncio              1.5.5            py38h06a4308_0    defaults
numpy                     1.23.2                   pypi_0    pypi
oauthlib                  3.2.0                    pypi_0    pypi
onnx                      1.12.0                   pypi_0    pypi
onnxruntime               1.12.1                   pypi_0    pypi
openssl                   1.1.1s               h7f8727e_0    defaults
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 21.3               pyhd3eb1b0_0    defaults
parso                     0.8.3              pyhd3eb1b0_0    defaults
pexpect                   4.8.0              pyhd3eb1b0_3    defaults
pickleshare               0.7.5           pyhd3eb1b0_1003    defaults
pip                       22.1.2           py38h06a4308_0    defaults
prompt-toolkit            3.0.20             pyhd3eb1b0_0    defaults
protobuf                  3.19.4                   pypi_0    pypi
psutil                    5.9.0            py38h5eee18b_0    defaults
ptyprocess                0.7.0              pyhd3eb1b0_2    defaults
pure_eval                 0.2.2              pyhd3eb1b0_0    defaults
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pygments                  2.11.2             pyhd3eb1b0_0    defaults
pyparsing                 3.0.9            py38h06a4308_0    defaults
python                    3.8.13               h12debd9_0    defaults
python-dateutil           2.8.2              pyhd3eb1b0_0    defaults
pyyaml                    6.0                      pypi_0    pypi
pyzmq                     23.2.0           py38h6a678d5_0    defaults
readline                  8.1.2                h7f8727e_1    defaults
requests                  2.28.1                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
scipy                     1.9.0                    pypi_0    pypi
setuptools                61.2.0           py38h06a4308_0    defaults
six                       1.15.0                   pypi_0    pypi
sqlite                    3.39.2               h5082296_0    defaults
stack_data                0.2.0              pyhd3eb1b0_0    defaults
sympy                     1.10.1                   pypi_0    pypi
tensorboard               2.10.0                   pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow                2.5.0                    pypi_0    pypi
tensorflow-estimator      2.5.0                    pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
tf2onnx                   1.12.1                   pypi_0    pypi
tk                        8.6.12               h1ccaba5_0    defaults
tornado                   6.2              py38h5eee18b_0    defaults
traitlets                 5.1.1              pyhd3eb1b0_0    defaults
typing-extensions         3.7.4.3                  pypi_0    pypi
urllib3                   1.26.11                  pypi_0    pypi
wcwidth                   0.2.5              pyhd3eb1b0_0    defaults
werkzeug                  2.2.2                    pypi_0    pypi
wheel                     0.37.1             pyhd3eb1b0_0    defaults
wrapt                     1.12.1                   pypi_0    pypi
xz                        5.2.5                h7f8727e_1    defaults
zeromq                    4.3.4                h2531618_0    defaults
zipp                      3.8.1                    pypi_0    pypi
zlib                      1.2.12               h7f8727e_2    defaults

特别要注意:keras-nightly ? ? ? ? ? ? 2.5.0.dev2021021100 的安装,其版本号有点特别

4. 踩坑指南

(1) so库找不到

2023-11-25 01:10:45.410281: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /path/to/cuda/lib64:/path/to/cudnn/lib64:

安装:cudatoolkit ? ? ? ? ? ? ? 10.1.243?

(2) tf转onnx错误

2023-12-07 04:17:32,217 - ERROR - Tensorflow op [StatefulPartitionedCall/model_1/Transformer-0-MultiHeadSelfAttention/BroadcastArgs: BroadcastArgs] is not supported

安装:bert4keras ? ? ? ? ? ? ? ?0.11.3

(3) 线程值错误

/site-packages/keras/backend/tensorflow_backend.py", line 73, in symbolic_fn_wrapper

????if _SYMBOLIC_SCOPE.value:

AttributeError: '_thread._local' object has no attribute 'value'

将代码段注释掉:

参考:2021-05-21_if _symbolic_scope.value: attributeerror: '_thread-CSDN博客

文章来源:https://blog.csdn.net/try2find/article/details/134841066
本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。