foreword
Cloud Log Service (CLS) is a one-stop log service platform provided by Tencent Cloud. It provides multiple services from log collection, log storage to log retrieval, chart analysis, monitoring and alarming, and log delivery, etc. To solve business operation and maintenance, service monitoring, log audit and other scenarios.
In short, CLS provides cloud storage of logs, and provides query, analysis, monitoring, alarm and other functions. So today, with curiosity, let's explore how to use python to write native logs to CLS.
Environment configuration
The official document provides detailed usage steps. The document link is as follows: https://cloud.tencent.com/document/product/614/34340
1. Service activation
Click https://cloud.tencent.com/product/cls to enter the page, you can click to activate the service immediately ;
of course, you can also click on the event announcement to check the free usage quota:
it can be seen that the free quota traffic quota is 5GB/day, The event ends at the end of 2021.
After activating the service, enter the clk service page. It can be seen that various log access solutions are provided.
2. Create log sets and log topics
Logset (Logset) is the project management unit of Log Service, used to distinguish logs of different projects. Log topic (Topic) is the basic management unit of Log Service, used to store log files. A log set can contain multiple log topics.
Click https://console.cloud.tencent.com/cls/overview CLS configuration page.
Click Log Topics in the sidebar , and click Create Log Topic .
It may be that the document is too old. The official document says that you need to create a log set first, but you can't find this button. In fact, when you enter a log set when creating a topic for the first time, it will be created automatically.
The created log topic is as follows, what is needed here is the topic ID , and the log is written to this topic through the ID when writing.
3. Create a machine group
CLS uses machine groups to uniformly manage a set of log source machines. . At the same time, I also understand it as a white list, only these machines can access the CLS service.
Python logs written to CLS
On the overview page of CLS at the beginning, you can see that CLS provides a variety of quick access solutions, many of which require the installation of Loglistener to collect logs. Here, Python is used to write access solutions using API to write logs without installation.
Clicking on each access scheme will enter the corresponding development document. Here I click API write to enter the development document.
The official API write specification is provided:
POST /structuredlog?topic_id=xxxxxxxx-xxxx-xxxx-xxxx HTTP/1.1
Host: <Region>.cls.tencentyun.com
Authorization: <AuthorizationString>
Content-Type: application/x-protobuf
x-cls-compress-type:lz4
<LogGroupList 的 PB 格式打包内容>
It can be seen from the above that there are two parameters, one is topic_id , which is the log topic id; the other is LogGroupList , which is the content of the log transfer protocol. Here PB refers to using protobuf for serialization, so protobuf must be installed first.
Install protobuf
Protobuf is a binary serialization format, which is smaller in size and faster in transmission than json. The purpose of installing protobuf is mainly to compile proto files into python, c, Java callable interfaces.
# 如果gcc版本较低，需要升级gcc
wget https://main.qcloudimg.com/raw/d7810aaf8b3073fbbc9d4049c21532aa/protobuf-2.6.1.tar.gz
tar -zxvf protobuf-2.6.1.tar.gz -C /usr/local/ && cd /usr/local/protobuf-2.6.1
./configure
make && make install
# 可以在/etc/profile或者~/.bash_profile末尾设置永久有效
export PATH=$PATH:/usr/local/protobuf-2.6.1/bin
Use the following command to check whether the installation is successful.
protoc --version
build python callable PB
Create a cls.proto file to define the serialization structure:
package cls;
message Log
{
message Content
{
required string key = 1; // 每组字段的 key
required string value = 2; // 每组字段的 value
}
required int64 time = 1; // 时间戳，UNIX时间格式
repeated Content contents = 2; // 一条日志里的多个kv组合
}
message LogTag
{
required string key = 1;
required string value = 2;
}
message LogGroup
{
repeated Log logs = 1; // 多条日志合成的日志数组
optional string contextFlow = 2; // 目前暂无效用
optional string filename = 3; // 日志文件名
optional string source = 4; // 日志来源，一般使用机器IP
repeated LogTag logTags = 5;
}
message LogGroupList
{
repeated LogGroup logGroupList = 1; // 日志组列表
}
Just use the following command to convert the proto file to a python callable interface.
protoc cls.proto --python_out=./
After execution, cls_pb2.py will be generated in this directory .
python code development
The code development is mainly divided into three parts: Protobuf structure construction, Authorization encryption construction and request upload log. There are too many details in the first two parts, especially the Authorization construction involves sha1 encryption, hmac-sha1 signature and the construction of four encryption parameters. :
1. Protobuf structure construction
import cls_pb2 as cls
import time
import requests
from hashlib import sha1
import hmac
# 构建protoBuf日志内容
LogLogGroupList = cls.LogGroupList()
LogGroup = LogLogGroupList.logGroupList.add()
LogGroup.contextFlow = "1"
LogGroup.filename = "python.log"
LogGroup.source = "localhost"
LogTag = LogGroup.logTags.add()
LogTag.key = "key"
LogTag.value = "value"
Log = LogGroup.logs.add()
Log.time = int(round(time.time() * 1000000))
Content = Log.contents.add()
Content.key = "Hello"
Content.value = "World"
print(LogLogGroupList)
# 序列化
LogLogGroupList = LogLogGroupList.SerializeToString()
2. Authorization encryption structure
View the document: https://cloud.tencent.com/document/product/614/12445 , which contains detailed parameter generation steps and samples.
This part of code development definitely tests personal patience and carefulness, and is suitable for doing it by yourself in the dead of night.
# 公共参数部分
secretId = '替换成你的secretId'
secretKey = '替换成你的secretKey'
region = 'ap-nanjing'
host = f'{region}.cls.tencentyun.com'
start = int(time.time())
end = start + 1000
uri = 'structuredlog'
method = 'post'
params = 'topic'
# 构建HttpRequestInfo
HttpRequestInfo = f'{method}\n' + f'/{uri}\n' + '\n\n'
sha1_info = sha1()
sha1_info.update(HttpRequestInfo.encode('utf-8'))
print(sha1_info.hexdigest())
# 根据HttpRequestInfo构建StringToSign
StringToSign = 'sha1\n' + f'{start};{end}\n' + sha1_info.hexdigest() + '\n'
key = secretKey.encode('utf-8')
value = f'{start};{end}'.encode('utf-8')
SignKey = hmac.new(key, value, 'sha1')
print(SignKey.hexdigest())
# 根据StringToSign构建Signature
key = SignKey.hexdigest().encode('utf-8')
value = StringToSign.encode('utf-8')
Signature = hmac.new(key, value, 'sha1').hexdigest()
print(Signature)
# 构建Authorization
Authorization = f'q-sign-algorithm=sha1&q-ak={secretId}&q-sign-time={start};{end}&q-key-time={start};{end}&q-header-list=&q-url-param-list=&q-signature={Signature}'
This part is the most difficult for me. There are too many pitfalls in it. One parameter sets one parameter, which is easy to make people confused, so most of the time is spent on the part of test parameter generation.
As shown in the picture, it keeps reminding me that the signature calculation is wrong. The reasons are summarized as follows: 1. The parameter splicing is wrong; 2. The kv of hmac is reversed.
Originally, I also added the lz4 compression format in the request header, but when I ran it, I found that the lz4 compression was not implemented in the code, so the following error was reported.
Finally, the lz4 request header was discarded, and the compression function was not implemented. Those who are interested can do it.
3. Upload log
In fact, it is to construct a post request, serialize the log into protobuf format and upload it to the server.
# 发起请求
url = f'https://{host}/{uri}?topic_id=717eba7d-85bb-4cd5-9c68-dfaa9f672bc6'
headers = {
'Authorization': Authorization, 'Host': host, 'Content-Type': 'application/x-protobuf'}
response = requests.post(url, headers=headers, data=LogLogGroupList)
print(response.status_code, response.text)
print(LogLogGroupList)
run test
python3 running program:
The protobuf structure information, binary data, encrypted parameters and request status code of the log are printed.
Then enter the CLS page to view, the log has been written.
search analysis
CLS provides the function of retrieval and analysis, we can use sql to query and analyze the uploaded logs.
As shown in the figure, the log data of each time period is counted. We click Add to Dashboard to graphically display the above analysis results.
Dashboard chart display
Click on the dashboard on the left to enter and view the chart just added. Click Edit on the right side of the icon to change the chart style.
At this time, the chart has no data, and we select the time dimension in the upper right corner to query the data.
Then the line chart successfully displays the data.
Alarm configuration
At the same time, CLS also provides alarm configuration, which triggers alarms through preset notification templates and alarm conditions.
Moreover, the alarm information is recorded in detail and displayed.
epilogue
It took about five or six hours to follow the documentation from learning CLS and python development to collecting materials and organizing them into articles. I was working on it after work. It was already one o'clock in the morning when I wrote this. It's because the liver doesn't move.
I hope this article can help you understand CLS and look forward to the next encounter.
Post-95 young programmers, write about personal practice in daily work, from the perspective of beginners, write from 0 to 1, detailed and serious. The article will be published on the public account [ Getting Started to Give Up Road ], looking forward to your attention.
