Tool recommendation-Colab introduction and usage

Tool recommendation-Colab introduction and usage

The reason why I was able to get in touch with Colab was recommended by a good friend. First of all, I have to thank him for Amway's such a good thing, hahaha! Let me briefly talk about my current understanding of this Colab. I think it is equivalent to a cloud host, with Google Drive as the storage location, so first of all, there must be Google Drive, which can be built on Google Drive , and then you can create a new notebook file like the one in Jupyter under the folder, associate it with Colab, and run the code.

Why do you say it is good? For me, my Anaconda is installed on the C drive. When learning deep learning and other things, the storage space of the C drive is constantly decreasing, and some library files and installation packages must be downloaded on the C drive. Families that are not rich make things worse, hahahaha! With this Colab, I don't have to store all the files locally, and I feel like I can say goodbye to Anaconda. At the same time, this gadget also has something that sounds very powerful, that is, you can set the code operation to GPU mode (the hard disk will allocate a very powerful GPU), and the running data set will be quite fast! Before I trained the data set, it ran very slowly on this machine, and I had to wait every time. I just got started with this thing. I tried installing tensorflow, and it really runs very fast! I haven't had the chance to try some powerful functions yet. I will write this article after installation. If I encounter any problems in the future, I will update the content in this regard.

I've been busy all afternoon and one night, and finally got started. At the same time, I took some detours, and I want to record them to help more friends!

What is Colab?

1. Google Colab is a free cloud service and supports free GPUs
2. Colab is Google and the server is abroad

You only need to install colab on Google Drive to use it. If you need to upload data, you can upload it to Google Drive, mount it in colab, and then you can access it directly.

Graphics card: V100, P100, T4 and other mainstream graphics cards for training, with 16 G memory.

Disk storage: 15 G, which can be expanded using Google Drive.

Time limit: 12 hours each time.

But it has obvious problems and needs a ladder , it is a Google product after all.

Basic configuration of Colab

1. Search Google Drive (https://www.google.com/intl/zh-CN/drive/), click "Go to Google Drive"

2. Log in with your Google account (you can register yourself if you don’t have an account)

3. Install Colab

Right click as shown below -> More -> Associate more applications

4. Search for Colab and add it (install as shown in the figure below)

5. After the installation is successful (right click -> more) found a Colaboratory

6. Use Colab

Create a folder on Google Drive, this is my folder directory , creating hierarchical folders is for better management of your code files.

7. Preliminary test code

Click to enter the folder where you want to put the code, right click on the blank space of the page, click More->Google ColaBoratory, and then you will see the interface you are familiar with

Then you can set the notebook (set to GPU)

8. Mount Google Cloud

​ Click the folder on the left (view directory), wait for the connection status on the right to be connected, and finally click Mount Google Drive

After the loading is complete (continue to click on the folder on the left), you will find that there is an additional drive directory (drive/My Drive/Colab), where Colab is the folder we just created, and the Colab folder is our project file.

I don't quite know what it means to mount the cloud drive, and I don't know why it is done. I searched online and said that Google hard drive and colab are separate. We need to mount the hard drive to colab. Advantages: in colab You can directly read and write the directory of google drive, and the model can be directly saved on dirve, which is very convenient. Still don't understand. . . . But it should be beneficial. . .

9. Use Colab

View the current script directory (!ls) and click to run

enter project

import os
path = "/content/drive/My Drive/Colab/Notebooks/17.yoloV3"
os.chdir(path)

Test the data set in my project, as for how I upload the data set to Google Drive, please read on!

from dataset.vocdata_tfrecord import load_labels,write_to_tfrecord
import os

found that it can run successfully

In fact, I like to create a new file associated with Colab directly in the current directory of the project, that is, the file where the code is written, and then run the following code

# 加载盘
from google.colab import drive
drive.mount('/content/drive/')

# 指定当前的工作文件夹
import os
# 此处为google drive中的文件路径,drive为之前指定的工作根目录，要加上
os.chdir("/content/drive/My Drive/Colab/Notebooks/17.yoloV3") 

The command line to install some libraries, such as installing pytorch

Colab generally has its own pytorch environment. If not, install accordingly:

!pip install torch torchvision  # 在Colab中执行操作语句时，感叹号不能漏

Installing tensorflow is similar.

This is basically an introduction to Colab, and I can use it to run my code tomorrow without using the local Jupyter, hahaha!

Troubleshooting

Solve the problem that colab uploads data sets slowly

Don’t look like I’m doing well in the past. In fact, I wasted most of my time in the afternoon and evening. Uploading from the local to Google Drive is too slow. It is not recommended to upload directly. At the same time, it is recommended to open the upload to the folder first. In the form of a compressed package, I will talk about how to upload faster.

1. Upload the file to OneDrive first

It is not recommended to use the web version of OneDrive. It is very slow just to upload files (it is estimated that the file upload and synchronization come together), directly use the OneDrive that comes with the computer

Copy the compressed file to the document folder. My compressed file has 1 G. The speed of copying is very fast. The next step is synchronization. It is recommended to find a place with fast network speed. I found the school’s campus network to be true. Slow, I use traffic synchronization, and I feel sorry for the 1G traffic. . . .

The next small window of OneDrive will display the ones that allow you to synchronize and consume traffic. . . The speed is also okay!

Finally, use the MultCloud extension program to upload files from OneDrive to Google Drive, add it directly to the extension program of Google browser, and download it in the app store

After entering, the interface is like this, click Add Cloud

Add both Google Drive and OneDrive, then click Cloud Transfer, the file directory of OneDrive is on the left, and the file directory of Google Drive is on the right

Finally click Transfer Now, this speed is very fast, great success!

Then you can see your files in the directory of your Google Drive, but don’t be too happy, this is just a compressed package, and it has not been decompressed yet, and decompression made me almost autistic. . . .

Decompression: First mount the Google cloud disk, I just didn’t do this step, and then run the decompression command written by others, but I can’t find the local directory all the time, I’m really going to be autistic. . . Because their decompression code file paths all start with /content/drive/My Drive, I must not have this drive file directory if I did not mount it. . . . So it will always report an error!

import os
from google.colab import drive
drive.mount('/content/drive')

path = "/content/drive/My Drive"

os.chdir(path)
os.listdir(path)

Then run the following code, the file directory and output directory are all according to your own, and finally it shows successful operation

!pip install pyunpack
!pip install patool
from pyunpack import Archive
Archive('/content/drive/My Drive/Colab/Notebooks/yoloV3.rar').extractall('/content/drive/My Drive/Colab/Notebooks')

I also have this folder in my cloud drive, and the final compressed package was deleted when it was useless.

Google Drive could not connect to the code executor.

This is what I have encountered when I set the computer to GPU mode. I don’t know why, maybe I don’t want to use it. I usually change it to None silently. . . .

reference article

I read a lot of blog articles, and finally completed the introduction of Colab. I hope the articles I wrote can be helpful to you. If you don’t understand what I wrote, you can read these reference articles! If you see this, you might as well give it a like, hahahaha!

https://mp.weixin.qq.com/s/zYU3RY6dgH4pCWyW-WCRwQ teach free whoring gpu server

https://zhuanlan.zhihu.com/p/54389036 Google Colab usage experience of the hard-working student party
https://blog.csdn.net/qq_36890370/article/details/106160805 Colab usage tutorial
https://www.cnblogs. com/zgqcn/p/11186406.html Colab Practical Tutorial
https://www.freesion.com/article/42081149133/ Solve the problem of colab uploading data sets slowly

How to use Google Colab to decompress RAR ZIP and other compressed packages in GoogleDrive online https://www.uud.me/qiwenzalun/colab-unzip-unrar.html
How Colab decompresses rar and zip files https://blog.csdn.net/ weixin_42870380/article/details/114731820

Finally google drive could not connect to the code executor.