Yanhuang's demand for Huawei Cloud
In the daily development of Yanhuang, we basically use cloud services for the needs on the server. At present, we mainly choose HUAWEI CLOUD. HUAWEI CLOUD's cloud hosts are relatively stable, and the configurations of cloud hosts provided are also quite diverse, which is very suitable for the needs of hardware configuration in different scenarios, and the response speed is also very fast in terms of technical support services.
Generally speaking, we will mainly purchase some yearly cloud hosts to meet the daily research and development and testing. But colleagues will also encounter some adhoc requirements, for example: for the problems encountered by users, we need to reproduce them in an experimental environment, and we will temporarily open several cloud hosts to simulate the customer's environment.
As Yanhuang Data has more and more employees, there are more and more demands for temporarily creating machines. We found that HUAWEI CLOUD's API is very complete in function, and HUAWEI CLOUD's API Explorer is a very convenient tool for debugging and testing. So we quickly developed Ichiban, a tool for creating cloud hosts in-house, through the API.
Introduction of Yanhuang Ichiban
Why is it called Ichiban? Because Yanhuang has a beer culture, the company’s refrigerator is filled with all kinds of craft beer in addition to beverages. We have a technology sharing meeting every Friday, and it is also a weekly Happy hour, with unlimited beer + food supply. So I think Ichiban+ grilled skewers are a must for our Yanhuang programmers in summer, welcome to Yanhuang to drink beer anytime~
First of all, Ichiban's login interface is as shown below. Because we use Microsoft Azure AD to manage our employees' accounts, we call Microsoft's API to log in, and record and manage user information.
After logging in, you can see the created cloud host information.
From this layer onwards, the APIs of Huawei Cloud are called to implement, and the functions include:
Create a cloud host
Cloud host startup, shutdown, restart, remote login and deletion
Extend cloud hosting lease
add hard drive, etc.
Ichiban's technology stack
We are using HUAWEI CLOUD's Python SDK. Here we have to praise HUAWEI CLOUD's API Explorer again. It is a tool that can provide rapid development functions and can automatically generate code based on the values you type.
You can also use the 'SDK code sample debugging' provided by HUAWEI CLOUD to debug the code, and it is the style of VS Code that we are all familiar with. Take a screenshot and experience it yourself. Are you familiar with friends who often use VS Code to develop?
Because we use Python as the development language, we use Flask + Gunicorn + Supervisor to host the web service.
Because Flask itself is written in Python, it has a high degree of compatibility with Python, and it is a lightweight web application framework.
Gunicorn is a WSGI HTTP server for Flask. It usually sits between a reverse proxy (such as Nginx) or load balancer (such as AWS ELB) and a web application (such as Django or Flask).
Here you only need to host the 5000 port of Flask, the configuration is as follows:
The service process of Supervisor is supervisord, which is mainly used to start the managed sub-process when starting itself, respond to the client's command, restart the crashed or exited sub-process, record the stdout and stderr output of the sub-process, generate and process the sub-process life cycle event. Related parameters can be configured in a configuration file, including the status of Supervisord itself. Create subprocesses in /etc/supervisord.d/xxxxx.ini, and manage the relevant attributes of each subprocess. The configuration file is generally located at /etc/supervisord.conf.
For example: subprocess configuration /etc/supervisord.d/ichiban.ini
The client of Supervisor is called supervisorctl, which provides a shell-like interface (that is, the command line) to use the functions provided by the supervisord server. Through supervisorctl, the user can connect to the supervisord server process, obtain the status of the child process controlled by the server process, start and stop the child process, and obtain the list of running processes.
The client communicates with the server through a Unix domain socket or a TCP socket, and the server has an identity certificate authentication mechanism, which can effectively improve security. When the client and the server are located on the same machine, the client and the server share the same configuration file /etc/supervisord.conf, and the configurations of the two are distinguished by different labels.
When restarting the process you can run:
Honghu collects data from HUAWEI CLOUD
Here comes the problem. . .
We have written the code of Ichiban, realized the function, and solved the problem of developers using cloud hosts. However, as the manager of Ichiban, I also encountered problems. How should I manage it? How do I know how much I spend each month, and the detailed usage?
Assuming that as the company has more and more colleagues, the use of ichiban will inevitably increase, and the cost will also increase. The boss asked if it was possible to make a budget for Ichiban every month or a quarter. At this time, without the support of data, it is difficult for us to come up with figures for the next budget.
At this time, the value of Honghu is highlighted. Honghu can help us to calculate the startup time of the cloud host, the number of cloud hosts, and estimate the cost very simply and clearly through the log files.
Ichiban's log is a log file in JSON format generated by calling Huawei Cloud's API, and then sent to Honghu's platform through fluentd. (You can also use Vector as a data transfer tool). In this example, the data type of log is JSON. In fact, you can choose different data types according to your actual situation. Honghu does not pick the type of data, it accepts any type of data and parses any type of data.
Honghu end
Added a cheap cloud host, followed the documentation (https://yanhuang.yuque.com/books/share/b1b27937-e331-40fd-bce1-bd521af4e662/vvv9si) and installed Honghu. The logo of the blue swan appeared.
Create HEI in Honghu (collect data through HTTP)
Issue Ichiban API Token
Ichiban end
Install Fluentd
Configure the monitoring data of fluentd
Modify the following configuration
It may be necessary to configure the root authority of the fluentd service, depending on the monitored file permissions
Received the data, yo ho. . .
Dashboard display of HUAWEI CLOUD usage status
Through a simple SQL query, we can count the usage of Ichiban in real time.
I also excerpted some configuration usage here
Data shows what do I see?
HUAWEI CLOUD usage. As the manager of HUAWEI CLOUD, I can clearly understand the demand for HUAWEI CLOUD Adhoc in a period of time, such as how many cloud hosts are used, and how much time is spent respectively.
user. Statistically, it is possible to find out who is a heavy user, understand the needs from users, and optimize.
total expenses. As shown in the figure above, we can clearly know the composition of our monthly HUAWEI CLOUD expenses. But to be honest, we saved a lot of unnecessary costs by temporarily creating machines as a strategy.
write at the end
HUAWEI CLOUD's API and related tools are doing very well, and the functions will be realized soon. But for managers, there are also challenges in cost management and operation and maintenance. But with Honghu, all this becomes much simpler. As long as there are enough logs and simple SQL statements, the headaches for managers can be easily solved.
In addition, through the analysis of real-time data, we can quickly find some problems. For example, we can add an alarm to the start-up time of the cloud host. Suppose: Zhang San’s cloud host has been on for a long time. Maybe someone forgot to close it. Then through the policy, send an alarm (Email or Webook) to notify Zhang San to reduce unnecessary costs.
we still have a lot to do
Looking forward, our follow-up development of Ichiban will continue to be done. We will connect Ichiban, Huawei Cloud and our CICD in series, create temporary git runners, test runners, etc. through Huawei Cloud API, and then monitor CI/CD through Honghu ,benefit:
1. Honghu can help with troubleshooting CI/CD issues.
2. Put the running status of CI/CD in the dashboard.
3. Reduce unnecessary costs. For example, we may run a certain test for 8 hours, and the case may be interrupted in a certain hour. Honghu can remind us that we can use manual or trigger scripts to check if there is no work. The runner of the task is deleted.
Once the data is connected to Honghu, the door to data playability is opened. As long as you pass simple SQL, you can find a lot of interesting things, even things you have never thought of. It may be an inspiration for a powerful function, or it may be is a detector that finds vulnerabilities. A few lines of SQL can make your data clear.
enjoy Honghu~
Large numbers to simple
reference
HUAWEI CLOUD API:
https://apiexplorer.developer.huaweicloud.com/apiexplorer/doc?product=ECS&api=NovaCreateServers
Supervisor's documentation:
https://zhuanlan.zhihu.com/p/424346764
Flame Phoenix data:
http://www.yanhuangdata.com/