最近在做大模型的昇腾迁移，国产化框架踩坑不少，基本一天的工作量相当于之前做纯视觉算法时一周踩过的坑数了。
现在在modelarts上用八卡昇腾910跑llama，不同于之前自己配环境，昇腾生态创新中心都是用的镜像，虽说打包起来不用自己配的那么麻烦，但一键启动之后各种找不到包、调用函数出错的问题搞得我头大，真的每跑一次大模型都要找一个相匹配的镜像才可以吗？
Session stopped
- Press <Return> to exit tab
- Press R to restart session
- Press S to save terminal output to file
root@168.170.140.104's password:
Access denied
[email protected]'s password:
Access denied
root@168.170.140.104's password:
┌──────────────────────────────────────────────────────────────────────┐
│ • MobaXterm Personal Edition v23.2 • │
│ (SSH client, X server and network tools) │
│ │
│ ⮞ SSH session to root@168.170.140.104 │
│ • Direct SSH : ✓ │
│ • SSH compression : ✓ │
│ • SSH-browser : ✓ │
│ • X11-forwarding : ✓ (remote display is forwarded through SSH) │
│ │
│ ⮞ For more info, ctrl+click on help or visit our website. │
└──────────────────────────────────────────────────────────────────────┘
Welcome to Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-29-generic aarch64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Fri Jul 21 05:39:31 EDT 2023
Last login: Fri Jul 21 05:39:27 2023 from 10.10.10.42
-bash: /usr/local/Ascend/ascend-toolkit/set_env.sh: No such file or directory
root@ubuntu:~# docker images
REPOSITORY TAG IMAGE ID CREATED SI ZE
swr.cn-central-231.xckpjs.com/yangxi/chatglm_finetune v3 ce059a3eb951 5 weeks ago 18 .3GB
swr.cn-north-309.mtgascendic.cn/bj-aicc/chatglm_finetune v3 ce059a3eb951 5 weeks ago 18 .3GB
ascendhub.huawei.com/public-ascendhub/pytorch-modelzoo 23.0 .RC1-1.11.0 1d3158646e1f 2 months ago 11 GB
swr.cn-north-309.mtgascendic.cn/bj-aicc/pytorch_1_11_0 pyto rch_1.11.0-cann_6.0.1-python_3.8-euler_2.8-v2 a5b48c40f013 2 months ago 13 .8GB
swr.cn-northwest-229.yantachaosuanzhongxin.com/yantapublic/pytorch_1_11_0 pyto rch_1.11.0-cann_6.0.1-python_3.8-euler_2.8-v2 a5b48c40f013 2 months ago 13 .8GB
guoneng_paddle v1 264ea1b0d9c6 3 months ago 43 .8GB
root@ubuntu:~# docker pull swr.cn-central-221.ovaijisuan.com/mindformers/mindformers_dev_mindspore_1_10_1:mindformers_0.6.0dev_20230615_py39
mindformers_0.6.0dev_20230615_py39: Pulling from mindformers/mindformers_dev_mindspore_1_10_1
6df89f31df6b: Already exists
ed1971c1ae79: Already exists
3326c9a29aaf: Already exists
1dc8445cb1fe: Already exists
b88914ee22d4: Already exists
bcdb23ea4034: Already exists
7d3d47c0e6f5: Already exists
5d7b3572edf7: Already exists
c41695c50bba: Already exists
957b9a31e2a7: Already exists
1672b9d56d5b: Already exists
36e34e344951: Already exists
160f6265f56d: Already exists
c57c0bc5f46f: Already exists
2bda0d153e53: Already exists
9c51c5251e18: Already exists
ab932c597460: Already exists
35490305253a: Already exists
4796376a3c72: Already exists
b2ceab4e4b50: Already exists
5996ae12a711: Already exists
fe6448692e33: Pulling fs layer
ba8a93f762d8: Pulling fs layer
ba74ada194a5: Pulling fs layer
a52d93ccc3fe: Waiting
801949047265: Waiting
b057aae08ec4: Waiting
dc70880ef9ce: Waiting
26c307ef6dfd: Waiting
c94bf8f93adf: Waiting
775eda4359be: Waiting
7cc9392229c8: Pulling fs layer
06ec42708026: Waiting
b01566b68dfa: Waiting
5857e4b48466: Waiting
f99e1c6d3442: Waiting
7d92f226dfff: Waiting
03f18c2615dd: Waiting
86a4a68587b8: Waiting
477735d7e341: Waiting
e3a369c4494f: Waiting
ad8940b6f8df: Waiting
93f2a596f5df: Waiting
bb4d004c2c19: Waiting
0da308e252fc: Pulling fs layer
4a62494058f8: Pull complete
dc1cb36895af: Pull complete
ca2c9a6dd846: Pull complete
8d67ef0f3461: Pull complete
414f0f3a818a: Pull complete
9cc92ebceb93: Pull complete
6d049032fea1: Pull complete
f2a11686cbf5: Pull complete
8dfd91dcabf2: Pull complete
bdf89cff68ba: Pull complete
fb7e8b85b671: Pull complete
b5e26e2ef46f: Pull complete
5fa7652a5f59: Pull complete
f0f36ddff6c5: Pull complete
1beff86c5121: Pull complete
1575edbb7f89: Pull complete
5f55c80ebec4: Pull complete
dae80ce2a02f: Pull complete
59dc1d9207d2: Pull complete
ffeb0c7a0d16: Pull complete
be378e762b31: Pull complete
1b46742c3546: Pull complete
95070b7b32e4: Pull complete
b1e1e78f7857: Pull complete
e4e5c468dca8: Pull complete
4a52a09714e8: Pull complete
a84238f298eb: Pull complete
f19ed39cf0fd: Pull complete
e746be989e03: Pull complete
a3184e84e5a3: Pull complete
185c350cc7e5: Pull complete
627f0e117356: Pull complete
1adfbc226f8e: Pull complete
88e1a8908377: Pull complete
bf33dcb88c6b: Pull complete
06de45739437: Pull complete
cd9299928399: Pull complete
17bdda4bed23: Pull complete
f0847357169c: Pull complete
75d91495f443: Pull complete
2f6a64e953df: Pull complete
Digest: sha256:46d06d8d248304cac200a1610adab48bd43b6f96950d0d9998194cfbce2b2835
Status: Downloaded newer image for swr.cn-central-221.ovaijisuan.com/mindformers/mindformers_dev_mindspore_1_10_1:mindformers_0.6.0dev_20230615_py39
swr.cn-central-221.ovaijisuan.com/mindformers/mindformers_dev_mindspore_1_10_1:mindformers_0.6.0dev_20230615_py39
root@ubuntu:~# docker login -u cn-north-309@UQ0YCRWIIJ4Q7UR0A63R -p 07c02a1f97ef2e5f0506c72c9b22f7296de1e43e6e20ed0dc7c3f882faeec7fd swr.cn-north-309.mtgascendic.cn
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
root@ubuntu:~# mindformers_dev_mindspore_1_10_1:mindformers_0.6.0dev_20230615_py 39sudo docker tag swr.cn-central-221.ovaijisuan.com/mindformers/mindformers_dev_ mindspore_1_10_1:mindformers_0.6.0dev_20230615_py39 swr.cn-north-309.mtgascendic .cn/aicc_lz/mindformers_dev_mindspore_1_10_1:mindformers_0.6.0dev_20230615_py39^ C
root@ubuntu:~# sudo docker tag swr.cn-central-221.ovaijisuan.com/mindformers/min dformers_dev_mindspore_1_10_1:mindformers_0.6.0dev_20230615_py39 swr.cn-north-30 9.mtgascendic.cn/aicc_lz/mindformers_dev_mindspore_1_10_1:mindformers_0.6.0dev_2 0230615_py39
root@ubuntu:~# docker push swr.cn-north-309.mtgascendic.cn/aicc_lz/mindformers_d ev_mindspore_1_10_1:mindformers_0.6.0dev_20230615_py39
The push refers to repository [swr.cn-north-309.mtgascendic.cn/aicc_lz/mindforme rs_dev_mindspore_1_10_1]
9efef30c32aa: Preparing
01a9a16e2668: Preparing
c5712af8a17d: Preparing
db280e538c6f: Preparing
126fde8a37c0: Preparing
2cf4bfba4482: Preparing
384c9e22b815: Waiting
1fbd15305307: Waiting
751d398eb53e: Waiting
54ab28f4e567: Preparing
2cf4bfba4482: Waiting
575a89fa936e: Preparing
2687bb1ea55d: Waiting
3a3e6ae19d7d: Waiting
4b91214b876b: Waiting
98ca7054afdc: Waiting
08588caa9e8c: Waiting
56e3137741ae: Waiting
575a89fa936e: Waiting
fe6f3dc260b6: Waiting
3c3ecefc768b: Waiting
af0c33ac85dd: Waiting
f8981441cef5: Waiting
505abd2f61f4: Waiting
4d88882b4901: Waiting
f0702ff1eb55: Waiting
d784cec35c0a: Waiting
040122e8b80d: Preparing
fd29a9717d53: Waiting
9204d9b6a0e4: Waiting
92e7456a14c3: Waiting
ede48f319b26: Preparing
7ddbb7762e8e: Waiting
ede48f319b26: Waiting
76956a73296b: Waiting
a3ba54e994c7: Waiting
407b1d90f91c: Preparing
4a1597322419: Waiting
da1e986cc22b: Waiting
5a50d393b5b7: Waiting
31ae5b42a91b: Preparing
42d14b030e23: Waiting
d17a52443df0: Waiting
3fe4f9b7f48b: Waiting
0f342db0ed7f: Waiting
3a4bcfe0fe41: Waiting
15d160702436: Waiting
f505431eb4df: Waiting
6a7668bf6919: Waiting
871887744f4f: Waiting
cfc0ee9792ea: Waiting
6d6189f99686: Preparing
407b1d90f91c: Waiting
ae995dd2db25: Waiting
f629e8a3ffb4: Waiting
781482aef03d: Preparing
9c41a34b5ef4: Waiting
bbe349b90b37: Waiting
3af29fffef76: Waiting
ed07bbecf3c9: Waiting
d86197d6f42d: Preparing
148cab5ddebe: Waiting
3fe4f9b7f48b: Pushed
a2d820d41bd2: Pushed
ec32b9434d98: Pushed
7f51e3e75b81: Pushed
8a56b925a0b5: Pushed
3f801a69d844: Pushed
08e934f34681: Pushed
775ead37ec48: Pushed
f2bdfe13a9d7: Pushed
6cbf0c57a96f: Pushed
e1aa3d6213e9: Pushed
b6c802dc9d21: Pushed
1765cdabb4ca: Pushed
f8f69bd4e159: Pushed
1b411266d046: Pushed
81f633409851: Pushed
f0d75b487767: Pushed
f31e8bd1a778: Pushed
cb399a44cec6: Pushed
71c36a160255: Pushed
7f2d3f882fc8: Pushed
bd6c3167d7df: Pushed
ec65416376e3: Pushed
2c2c66a6bb1c: Pushed
mindformers_0.6.0dev_20230615_py39: digest: sha256:46d06d8d248304cac200a1610adab 48bd43b6f96950d0d9998194cfbce2b2835 size: 18352
root@ubuntu:~#