Deployment of deep learning lab environment Huawei cloud of ECS, CNN building model implementation scenic landmark recognition, experience AI know King brings convenience.
Log in elastic cloud servers ECS
1, view the server public IP.
2, the open Putty software, enter the public IP elasticity in the Host Name, and click Open be
connected.
3, ignore the warning, confirm the connection.
Anaconda installation software
1, this command is divided into Linux, you need to press Enter to execute the command after the entry is complete, do not perform during the execution of the command in
other operations, to avoid problems.
2, the software download Anaconda, enter the following command, the software in Putty can right-click paste:
wget https://repo.anaconda.com/archive/Anaconda3-5.3.1-Linux-x86_64.sh
. 3, the software installation Anaconda , enter the following command:
bash Anaconda3-5.3.1-Linux-x86_64.sh
Figure 2-15 Anaconda installation interface
4, the installation page warning the lack of bzip2, regardless temporarily, press enter, skip reading the agreement, and enter yes, proceed with the installation
software.
Figure 2-16 interface protocol agreed to install
5, press Enter to use the default installation location to install.
Figure 2-17 software installation position confirmation Anaconda
6, installed after being given bzip2, enter the following command, if there is no error, please ignore this step and jump directly to a
step:
yum install bzip2
installed Jupyter Notebook and open the Remote Access
1, quit from the Python environment, enter the following command:
Exit ()
2, the installation Jupyter Notebook, enter the following command:
Conda the install jupyter Notebook
. 3, the profile command generator Jupyter Notebook:
jupyter Notebook --generate-config
provided Jupyter Notebook remote login password, enter the command into the environment IPython:
IPython
are input following two command line:
from notebook.auth Import passwd
passwd ()
to enter a password, note that there will be prompt, and then save (full copy quotes, right after copying selected) Sha1, and then
create a new text document save a local computer, the next steps It will be used.
FIG 2-37 Sha1 generating an interface
6, enter the exit () exit IPyhton environment, and enter the following command vi Profile Editor:
vi ~ / .jupyter / jupyter_notebook_config.py
entering vi editing interface default command mode, press the i switching from the text input mode to the command mode (English input
mode), press esc return to text input mode from command mode, enter the command mode /, followed by the need to search for a
string (case sensitive), can press enter complete the search, n may be switched to the next matching object.
Locate the following four to modify and delete the # in front, it is recommended to enter the full search string in the file, Sha1 for the first
time set a password on the step generated Sha1.
= c.NotebookApp.ip '*'
c.NotebookApp.password
= U 'sha1: 2cff9606d39d: 7f9c0e43eeda25941bf274ed47d412a3022b6d4d'
replaced their sha1, remember to add u.
= False c.NotebookApp.open_browser
c.NotebookApp.port = 8888
Note: After each amendment shall return to command mode and then re-enter the search text input mode changes, all the changes are complete
return to command mode, enter a colon in the English state input :, enter the last line mode, then enter wq, and press enter to exit the security
deposit if the text input error, you can enter q in line mode! Exit without saving, followed by Action Illustration.
Upload local resources to server
1, click the Upload button on the page, select img-classified.zip file
Figure 2-49 Jupyter Notebook interface
Figure 2-50 Jupyter Notebook file upload interface
2, again click on the Upload to start uploading, using the same the method begins uploading train.csv, then return to the web
server page, click on the remote login, enter the account password to login to the server
Figure 2-51 Telnet server web interface
to achieve accurate identification of 34 attractions convolution-based neural network
Figure 2 - 52 remote web server control interface
3, an input box appears after clicking Input Commonds, we can here Paste command, and then press Enter Send
the entry box to send commands to the server to perform ECS
Figure 2-53 remote web server replication command input interface
4, enter the following command to copy decompression software to install, the installation procedure as before
yum -y install the unzip ZIP
5, after the file upload is complete enter ls, see if you can see just upload files
ls
achieve precise identification of 35 attractions convolution-based neural network
6, enter the following command to decompress compressed
unzip img-classified.zip
Preprocessing the raw data set
3.2.1 read original data set
1, first need to create a Notebook, then introducing the tool used in this experiment, each cell can operate independently
line, press Shift + Enter operation selected cells , import keyword responsible for import.
#import keyword is python import third-party packages, pandas can handle cs is a
package v, xls file format, as used here suggests pd instead of pandas, code more concise
Import PANDAS AS pd
#requests package allows python obtain data on a web page, this experiment use it to get web
page picture content
import requests
achieve scenic accurate recognition based on neural network convolution page 37
#os python responsible for communications and systems can be used to create a new folder
import os
2, read train.csv file data, and the data assigned to the variable content.
When #python create a variable, not required to declare the variable type, the inner PANDAS csv file to read
content stored in the content variable
content = pd.read_csv ( 'train.csv')
. 3, view the data set description.
View Data Set Description
content.describe ()
Figure 3-2 describe interface data set
4, data set contains see id, url, landmark_id three fields, an image corresponding id, the network address of the image,
category image id, count, unique, top, freq denote the number of data, the number of unique data, the
most current data number, number of times the most data appear appears. So there is no duplication of the image can be drawn, but
the network address of the image are missing values, contains a total of 14,952 kinds of scenic category, this data set just released inconsistent,
possibly because the data set is continually updated, while just one train.csv portion.
3.2.2 Screening of the original data set
because the original data set is large, so the need for screening, and then get the network address of the image after screening the corresponding
loaded onto elastic cloud server.
1, can know by the preceding analysis, id and url image is unique, and the last landmark_id is one pairs
and more, so you can filter the data set through this field, due to the index after the screening will not change, so as
if the row of the filtered data to traverse, will lead to faults run error, so it is necessary to reset the index.
Create a list of data sets are screened landmark_id
= ARR [ '0', '2', '. 3', '. 5', '. 7', '. 8', '. 11', '12 is', '13 is', '16']
#isin determines whether the previous element in the list that follows the inside, the filtered data set in the landm
ark_id in front of which arr
content_selected Content = [Content [ 'landmark_id']. ISIN (Ar
R & lt)]
After screening the data set or to maintain the original index, not continuous, we need to reset it
content_selected = content_selected.reset_index ()
Based on convolutional neural network recognition accuracy resort page 38.
2, see the filtered data set, can be found in a field index, since the index after a reset the old index
as a field to save, rather than directly discarded.
See front portion of the data set, the default front 5
content_selected.head ()
after drawing 3-3 View filter the data set
3, because we know that screening data sets landmark_id contains 10 species, but it is unclear how much data a total, so it is necessary to check
the number of data sets look after the screening, then as input subsequent procedures.
#content_selected element is stored in a table, shape [0] row in the table represents the
number, shape [1] is the number of columns in the table represents
NUM = content_selected.shape [0]
3.3 Experimental data set acquisition
code for program download implemented 3.3.1
1 now after screening data sets with the number of data sets has also been, this step is to write the code, in order to take the focus from the data
to the network address of the image, and then download the image, which is part of the complex, can gradually be viewed according to the comment, consider
the image of the country and the network address of the firewall failure reasons, can not access parts of the network connection address, so access to the network to
the code address portion of the abnormal catch to prevent because a network address inaccessible cause the program to stop running.
#def keyword is used to define the function of this part of the code will not be executed immediately, but in the calling function
will execute when the
def download_img (num):
This is part of a cycle, the number of samples to obtain the data set, and then generates a corresponding cycle times
Number sequentially downloaded
for I in Range (NUM):
#i represents a row index of the current sample, url set of image data corresponding to the link row index
cited, AT, then the current link to get the image corresponding to the sample according to a row and column indices
img_url = content_selected.at [i, 'url ']
convolutional neural network-based resort to achieve precise identification page 39
Convenience, we use the cycle numbers as a picture name, str digital type of order
Number is converted to a string type
img_name = str (i) + ' . Jpg'
We get to sample the corresponding landmark_id, and create a folder, which
Like a folder with the following pictures are of the same class, and the folder name is the label
folder = content_selected.at [i, 'landmark_id ']
Definition picture save path, creating the current label file sample img folder in
Folder, then the image write
path = 'img /' + folder + '/'
Determine whether there is a path, if the folder is not created, the path does not exist, write
The file will complain
isExists = os.path.exists (path)
IF not isExists:
This part of the module to create a folder using the os
os.makdirs (path)
#try and except abnormalities responsible for capturing and processing python inside, that is,
we often say that error, there may exist an invalid picture link, open questions, we have no
way, no need to address
try:
Get access in front of the image link
= requests.get img (img_url)
#with keyword after the completion of this part, just turn on automatically shut
resources, without having to manually manage, open a new file is in the path in front of the set and hit
open
with open (path img_name +, 'ab') AS f:
#write will get access to the content of the image link to a blank document written
in, after the completion of this part, with automatically refresh and save the content, the release of resources
f.write (img.content)
If an error occurs, it will jump to here, because we do not deal with, all only
Wrote pass, that is, over
the except:
Pass
achieve scenic accurate identification on page 40 Convolution-based neural network
2, after the screening data sets can be saved to your hard drive for easy viewing later, and then all preparations have been completed, call the
image download function to begin downloading images.
After screening the data set is written csv file
content_selected.to_csv(‘data.csv’)
Defined function calls download, start the download pictures
download_img (NUM)
3, because the image of a web address most of Google cloud, part of the link does not work, in order to ensure the normal operation of the experiment, the experimental part of the data set with manual packaging, namely 2.25 extracting img-classified.zip upload files, run after the code error can be directly without further experiments, without waiting.
4, this part of the source code for the download_img.ipynb, packaged with Lab Manual, it is recommended to manually enter the code, do not run directly source.
