One-stop service from installing miniconda to configuring jupyter and GPU version pytorch without root privileges on Ubuntu

foreword

I have gone back and forth to the school server several times. The school does not have root user permissions for students. The teacher can only install miniconda and then manually configure jupyter, GPU version pytorch and other environments.

If you are a veteran, you can follow these commands to install. If you are a beginner and you encounter a bug, copy the details of the bug, focus on copying the error code and the summary of the error that follows, and then search online. Chinese Internet resources are already abundant, and you don’t have to Google for general bugs. You just have to learn the general steps of installation with the belief that you can solve them, and leave the source code to your experienced self in the future.

install miniconda

As the name of miniconda suggests, the mini version of conda is lightweight and takes a short time to install, which is why the teacher asked us to use it.

Another reason is that, in contrast, anaconda has a complete environment and a large package, which is not suitable for all students on the server to toss one.

The following describes how to install miniconda.

Get the installation package

wget https://mirrors.bfsu.edu.cn/anaconda/miniconda/Miniconda3-py38_4.9.2-Linux-x86_64.sh

Or manually download to the computer and rz to the server

The link is here:

https://mirrors.bfsu.edu.cn/anaconda/miniconda/

Or click to download directly:

https://mirrors.bfsu.edu.cn/anaconda/miniconda/Miniconda3-py38_4.9.2-Linux-x86_64.sh

then rz

rz

Not rz, scp can also be used.

For the whole process, refer to my article How to transfer files from a Windows computer to a Linux server without lrzsz and root permissions

Open the Windows command line.

(Open the command line, for details, please refer to my article How to open the Windows command line and Powershell window )

input the command:

scp D:\Download\Miniconda3-py38_4.9.2-Linux-x86_64.sh [email protected]:~

This is to transfer the Miniconda3-py38_4.9.2-Linux-x86_64.sh under the D:\Download folder to the target folder with an IP address of 101.102.103.104 and a user name of user007~ (that is, the user root directory).

Be optimistic about the location of your files, your public or private ip address, user name, and target folder, all of which must be corrected.

(If you want to transfer other large files that exceed the 4GB limit of lrzsz, you can also use scp)

Install with the installation package

input the command:

bash Miniconda3-py38_4.9.2-Linux-x86_64.sh

Then read the license license and enter the next line.

Then type yes on the keyboard to agree to the license.

Then wait for it to finish parsing the package to be downloaded, and ask you if you want to enter yes to agree when initializing, saving the process.

Wait for the installation to complete.

Configure environment variables

Enter the configuration file of environment variables

vim ~/.bashrc

At the end or wherever you feel comfortable, add the following line, make sure it is not commented out.

export PATH=/home/user007/miniconda3/bin:$PATH

Replace user007 with your user.

Activate the configured environment

source ~/.bashrc

Then you should switch to the default environment base of the conda environment. The command line will be converted from the above code below to the following:

user007@dlp10:~$ source ~/.bashrc 
(base) user007@dlp10:~$ 

examine

You can check it out:

(base) user007@dlp10:~$ conda -V
conda 4.9.2

Configure mirror source

The image source seems to be common to all virtual environments, so you don’t need to create a virtual environment first, you can do this step first.

Enter all of the following at the command line:

conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/menpo/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/msys2/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/

conda config --set show_channel_urls yes

The first 7 lines add the mirror source, and the latter item displays the channel address when searching.

remove default source

Enter the image source configuration file:

vim ~/.condarc

Find the line showing defaults and delete it.

channels:
    - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
    - https://...
    - https://...
    - defaults
show_channel_url: true

The file is similar to this, and then we delete the defaults line, for example, here is the fifth line.

Tips

In vim, when a line is selected, press dd to directly delete the entire line.

dd

examine

conda config --show-sources

Displayed as follows:

(base) user007@dlp10:~$ conda config --show-sources
==> /home/user007/.condarc <==
channels:
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/msys2/
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/bioconda/
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/menpo/
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
show_channel_urls: True

Create and use a new virtual environment

Create a new virtual environment for us to toss jupyter, pytorch, etc.

In fact, it is not impossible to install directly in the base environment, but the programmers need to master the different versions of the environment for different purposes. For example, if you run deep learning in pytorch, you may also need to use tensorflow to run, so you can create two different virtual environments for use together.

create environment

conda create --name d2l python=3.8

Create an environment called d2l, using python version 3.8.

activate or enter the environment

conda activate d2l

You know, d2l is changed to the environment name you took, and it enters your environment.

View all environments that exist

Either of the following two

conda info --envs
conda env list

The result is as follows:

(base) user007@dlp10:~$ conda activate d2l
(d2l) user007@dlp10:~$ conda info --envs
# conda environments:
#
base                     /home/user007/miniconda3
d2l                   *  /home/user007/miniconda3/envs/d2l

The one with * is your currently activated environment.

close environment

conda deactivate yourEnv

replace yourEnv with the name of your running environment.

delete environment

conda env remove -n yourEnv

Same as above, replace yourEnv with the name of the environment you want to delete.

Install and configure jupyter

Jupyter is an interactive notebook, which is convenient for us to call the server to execute python code on the web page. You only need to prepare a usable browser, enter the server address and connect the jupyter port, and you can use it conveniently.

install jupyter

input the command:

conda install jupyter

Then enter y and press Enter, agree to install the packages it finds.

Configure jupyter login address

First configure the password

enter:

ipython

Enter the following two statements in the subsequent running interface

from notebook.auth import passwd
passwd()

Then Enter passwd: will be displayed, you can enter the password you set, it will not be displayed on the screen, press Enter after confirmation

Then display Verity passwd:, this is to teach you to confirm the password, just enter the same password as last time.

Get a key, for example as follows:

Out[2]: 'sha1:5b5f*****11:371192**************539808392c6ff97'

The * is where I masked my key.

The specific ssh interface can refer to the following figure:

Method Two

enter:

jupyter notebook password

Then enter the password twice as above, and it will display:

[NotebookPasswordApp] Wrote hashed password to /home/user007/.jupyter/jupyter_notebook_config.json

Then enter the file to view:

vim /home/user007/.jupyter/jupyter_notebook_config.json

you can see:

{
  "NotebookApp": {
    "password": "sha1:0916*****f:a4614d9f**********615be8a8755865857fe0"
  }
}

The password behind it is the key you generated.

Save this key , it will be used later.

Generate jupyter notebook configuration file and set

enter:

jupyter notebook --generate-config

Then the next line will display the address of the generated configuration file:

Writing default config to: /home/user007/.jupyter/jupyter_notebook_config.py

Enter the configuration file and modify it:

vim ~/.jupyter/jupyter_notebook_config.py

These can then be added directly at the beginning.

c.IPKernelApp.pylab = "inline"
c.NotebookApp.ip = "*"
c.NotebookAPp.open_browser = False
c.NotebookApp.password = "sha1:b39*****79f:9b9ab99********b99a841a6403aa52647"
c.NotebookApp.port= 8888
c.NotebookApp.notebook_dir = "/home/user007/jupyter"
c.NotebookApp.allow_remote_access = True

Among them, the first line is to set all matplotlib images to be displayed through iline;

The second line is to allow any ip address to access the jupyter end;

The third line is to prohibit the browser from popping up while running ipython;

The fourth line is to set the startup password, just move the password you just remembered . Of course, if you use the second method above to generate a password, you don’t need to move it here;

The fifth line is to set the port, it seems that it can not be set;

The sixth line is to set the default folder of jupyter notebook. This folder must exist, otherwise an error will be reported;

The seventh line is to set to allow remote start, if it is not set, an error will be reported.

Of course, you can also search for the commented position in its file, and then uncomment and modify the subsequent settings. Also because the comment is the default, in order to save the default as a reference, add your settings below after finding it. (The last method is recommended, which is convenient for future viewing, and also helps to restore the default settings after viewing by yourself.)

vim tips

vim has three modes: (1) command mode; (2) edit mode (input mode); (3) last line mode (last line mode).

The last line mode is the mode when you enter : or / below.

Enter / followed by the character you want to find to realize character search. For example, if you want to find out where 'App.ip' is, enter:

/App.ip

The result is as follows:

There are others to mention. If you enter and find that you do not find what you are looking for, press Enter first , then

输入n为进入下一个查找位置，输入N为返回上方找到的字符位置。

Of course, the best way is to determine what kind of character you are looking for and locate it precisely.

Notice

Also need to explain the following. Many articles mention the need to set up an ssl certificate, but an error will be reported after use, so the following settings are not needed .

#命令行中
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout mykey.key -out mycert.pem
# 然后要你输入一些国家啊组织名什么的，随便输，都可以生成这个证书

# jupyter_notebook_config.py中的
# c.NotebookApp.certfile = u'/home/user007/.jupyter/mycert.pem'
# c.NotebookApp.keyfile = u'/home/user007/.jupyter/mykey.key'

I guess it might be because I don't recognize this kind of private certificate. Anyway, I don't even need to fix this kind of school server.

Login to jupyter

start juypter notebook

command line input

jupyter notebook

Observe the port it opens (because it is a server, someone else may be using it, so the port assigned to you may not be the 8888 you set, so you have to read it. In fact, you will use jupyter to type code later, and there are many errors that need to be reported in the This is from the background. )

For example, this time, it showed that port 8888 was already occupied, so I switched to port 8889. Next, remember not to lose 8888 again.

Enter jupyter from browser

Open the browser and enter your ip:port number in the address bar, such as

101.102.103.104：8889

Then press Enter, you can see that you have entered the login interface

small error

Of course, if you see Token authentication is enabled, it must be that your password is not set properly. (For example, you typed c.NotebookApp.password into ...App.passwd before)

懒得整成正确的再整到错误处重新截图了，就借用一下这个，几度热忱大佬的图吧

可以参考后台报错信息看。

可以确认，确实是打成passwd了。改好就行。

输入密码登录

如题，输入密码点Log in就成。

可以看到我们就进入了界面。

配置不同环境内核

方法一

好像，不同于网上大部分的，ipykernel都不用下，直接下这个：

conda install nb_contrib_kernels

方法二：配置ipykernel

首先下载ipykernel，好些所有你要配有的环境都要下：

conda install ipykernel

然后在你需要关联的环境中使用命令

ipython kernel install --user --name=d2l

比如，你在base下，然后你有tf2的虚拟环境，需要关联，那么你可以用：

(base)conda activate tf2
(tf2)conda install ipykernel
(tf2)ipython kernel install --user --name=tf2-kernel

这种方法来创建一个叫tf2-kernel的内核，和tf2虚拟环境关联。

（这里也说明了，你的ipkernel中的内核不一定非要和虚拟环境同名的）

不配置也不是不行……

当然，如果你是狼灭，比狠人还不止狠一点，可以一直手动切换环境启动jupyter，完全绕开这些内核配置……

(base)conda activate tf2
(tf2)conda install jupyter
(tf2)jupyter notebook

事实上，我以前比较单纯，一直都是这么做的，完全没意识到老师和论坛大佬们教我们这些人配虚拟环境，就是让我们base里启动jupyter，再在jupyter里面创建在不同环境下运行的文件，调好内核就行……

调内核等操作之后讲解……

Jupyter基本操作

创建Jupyter运行python的特有文件——ipykernel文件

点击右上角New，然后挑选你要选的内核。

比如挑选第二个。可以看到进入后是这样：

运行

d2l后，可在In后输入你要运行的代码，比如：

import math
x = math.sin(3.14 / 2)
print(x)

运行快捷键为shift+enter。

得到结果为：

0.9999996829318346

运行后，可在下方继续运行。继续运行的代码可以继承之前所有的变量、参数、类等。比如再返回x的反正弦函数：

y = math.asin(x) * 2
print(y)

得到结果为：

3.1399999999999753

可以观察到，第一个运行的，块中左上角为In [1]；第二个的，为In [2] 。这可以帮助你调试。

到这里，Jupyter就可以正常使用了。不过还有工作要做，如果你需要用torch跑深度学习的话。

安装GPU版pytorch

只装torch，没问题的。如果你是要在服务器上，顶着老版本CUDA，安装能够在GPU上跑的torch，难度可以说是有点死去活来。不过放心，看了我这篇文章，还是能整活的。

具体可以参考这篇文章：

torch报错AssertionError: Torch not compiled with CUDA enabled解决方法 torch适配CUDA降版本、选择gpu版本最终方案

就说简单流程：

首先更新一下conda

（nn滴，这是我写完后面试了之后试出来不对才加的，tmd上面刚说的还不死去活来，这次差点是没活过来，血的教训，一定吸取啊xdm）

输入：

conda update conda

可能不对，因为你不在base环境还是怎么的。然后你根据提示把它里面的代码输进去就行。（哈，我说什么之前，学会认真看出错的bug吧）

查看机子的CUDA版本

输入：

nvidia-smi

比如我这里CUDA是10.2

在官网查看适配版本

点击官网：https://pytorch.org/get-started/previous-versions/

这里查看到我10.2的CUDA适配1.12.1的pytorch，0.13.1的torchvision。

在镜像源直接找到对应的CUDA版本，即GPU版本pytorch下载

省了进镜像源查看的说明，自己找到对应的吧，直接用对应的包conda install下载。

记得自己找到哦

这里选用的是https://mirrors.bfsu.edu.cn/anaconda/cloud/pytorch这个源。

conda install https://mirrors.bfsu.edu.cn/anaconda/cloud/pytorch/linux-64/pytorch-1.12.1-py3.8_cuda10.2_cudnn7.6.5_0.tar.bz2

接着再用以下命令安装适配的相关的、前置的包：

conda install pytorch

然后torchvision就同理，就不赘述了。

检验

Jupyter中创建一个文件输入

import torch

print("是否可用：", torch.cuda.is_available())        # 查看GPU是否可用
print("GPU数量：", torch.cuda.device_count())        # 查看GPU数量
print("torch方法查看CUDA版本：", torch.version.cuda)  # torch方法查看CUDA版本
print("GPU索引号：", torch.cuda.current_device())    # 查看GPU索引号
print("GPU名称：", torch.cuda.get_device_name(1))    # 根据索引号得到GPU名称

运行改小块，得到结果如下：

是否可用： True
GPU数量： 2
torch方法查看CUDA版本： 10.2
GPU索引号： 0
GPU名称： Tesla T4

看到这样的结果，都能打印出来，就是安装成功了。

（独家）（死去活来记：没耐心中途关下载了，报错specified in the package manifest cannot be found）

有人没耐心等完下载，就把conda install的关了，我不说是谁

网上搜的，参考https://blog.csdn.net/qq_44747572/article/details/124960730

输入

conda clean --packages --tarballs

然后，

碰到这个BUG，你极有可能需要回到之前我说的第一步，就是更新conda那里。然后，你可能会丧心病狂地尝试conda再次安装、conda装完了回去cuda安装、cuda装一半发现没用，又试着pip安装、pip安装后发现装了cuda11.7版本的，继续抓狂……

不过，

总之最后我还是用这个方法安装成功了，又一次。对，第一次就是写最上面让你参考的那篇文章，死去活来。这一次，又一次差点气死，怎么说呢，可能作为程序员是难以想象下次又献祭几天寿命来整活来……

至于为什么不能直接conda install安装

可以试一试：

conda install pytorch

结果如下：

可以看到，它自动给我们下的是CPU版本的。

可能也是源没配置好。不过在源没配好的情况下，用这种方法解决，额，也能解决。

后记

写本文本来是因为重复整了次学校服务器的conda安装，有感就写了这篇长文。本来以为轻轻松松，没想到小坑不少，大坑掉得差点没得跑，希望后人看到这马到成功，马不停蹄赶紧好好整整自己和服务器的tm生活吧。

（ps：整这个的时候，还在学校写论文，挺苦逼的，希望后人能摆脱这样的痛苦，和服务器甜蜜双排）