1 background
Experience is one of the business keywords of Dewu. For front-end development, improving user experience is one of the most important tasks.
The front-end platform of Dewu currently has inspection systems, monitoring platforms and other means to ensure the stable operation of online pages, but there are still some problems in "monitoring blind spots", and inspections and monitoring are post-alarm methods. In order to ensure that the pages are online We can get a certain user experience guarantee in advance, combined with the company's strategic goals, we decided to develop an H5 page detection service, which is used to pre-detect the pages that are about to be launched, and expose the possible problems of the pages in advance to feed back to the corresponding development/operation , we call this service: "Experience Checkpoint".
This article starts from the development practice of the "experience bayonet" service, and introduces the structure and design of the Dewu inspection system, hoping to provide some learning and reference value for the development partners involved in the stability construction.
2 User Experience Quantification Standards
When we try to quantify issues affecting user experience, we need to think about the following two main questions:
What affects the user experience?
We have gained a deep understanding of the factors that affect user experience through rich data support and practical experience. Based on past online problem feedback collection and development experience, we roughly divide experience problems into two levels:
P0 level: These problems seriously affect the page loading speed or involve security risks, for example, the page contains large pictures/media resources, and the page contains personal privacy information;
P1 level: These problems may have a potential impact on user experience, for example, there are interface requests with a response time exceeding 300ms on the page.
How to detect and quantify the problem?
Once we have defined and graded our experience issues, we need to establish appropriate mechanisms to detect them. For the checkpoint service, we can take the following steps to quantify the problem, convert it into executable detection code, and generate a corresponding detection report through the checkpoint service for the caller to use:
-
Identify metrics and metrics: First, we need to identify metrics and metrics to quantify experience issues. For example, for the problem of interface request speed, the interface response time can be used as an indicator, and certain standards can be set at the same time, such as exceeding a specific time threshold is considered a problem.
-
Writing automation scripts: Based on indicators and standards, we can write automation scripts to simulate users performing related operations in a headless browser, such as loading pages, clicking buttons, sending requests, etc. These scripts will perform performance measurement and problem detection based on the set metrics.
-
Execute tests using a headless browser: We can run automated scripts in a headless browser to simulate user behavior and collect corresponding performance data.
-
Result analysis and report generation: Through the collected performance data, we can conduct result analysis and convert problems and related data into test reports. The report may include a detailed description of the problem, problem level, relevant performance indicators and data.
Provide to the caller: Finally, through the checkpoint service, we can provide the generated detection report to the caller. The caller can make corresponding optimizations and improvements based on the problems and data in the report to improve user experience.
Such a mechanism can help us automatically detect and quantify experience problems, and provide executable detection code and related reports. In this way, we can more effectively identify and solve problems, and provide accurate data for development teams to optimize.
The following are the detection cases that we have sorted out that need to be implemented:
After collecting the specific cases that affect the user experience, it is necessary to determine the specific development plan. Since the bayonet service and the Dewu front-end platform inspection system have many technical overlaps, we decided to use the existing inspection architecture to integrate The "experience checkpoint" is integrated into the existing inspection system, which can save a lot of development time.
3 Inspection system infrastructure
The program goal of the inspection system is summarized in one sentence: regularly obtain the list of page addresses to be detected from the data source, and then perform batch detection and generate reports.
In order to meet the personalized needs in different scenarios, the inspection system abstracts three base classes of inspectors, and each scene inherits the base classes to realize the customization requirements.
3.1 Patrol base class
1. DataProviderBase (data provider base class):
dataSlim(): Simplify redundant data;
fetchData(): Get remote data, process and return the url list of pages to be detected;
isSkipTime(): Used to set conditions and skip timing tasks under certain conditions;
schedule(): Set the running interval of the scheduled task;
2. PageInspectorBase (page inspector base class):
check(): Checker entry, used to open the specified detection page and initialize the monitoring of various resources;
injectRequestHeaders(): Inject cookies, tokens, etc. required by the page interface request;
urlCheck(): url address check;
onRequest(): Monitor page requests;
onResponse(): monitor page response;
onPageError(): Monitor page errors;
3. DataReporterBase (data report base class):
buildReporter(): Generate a detection report based on the collected error information;
feishuNotify(): Send the generated report to the specified notification group through Feishu;
getHTMLReporterUrl(): generate a static html file for the report according to the ejs template and upload it, and return the online report address;
We can visually compare these three base classes to three departments with different divisions of labor in a restaurant, which makes it easier to understand:
The front desk of the hotel is responsible for receiving the orders provided by the customers, the kitchen prepares and prepares dishes according to the orders, and the waiters provide the prepared meals to the customers.
DataProviderBase (data provider base class): Responsible for regular polling to receive the list of pages to be detected provided externally. This component is similar to the front desk of a restaurant, receiving orders from customers. It is responsible for obtaining the list of pages to be detected from the outside, and passing these pages to the detector for detection.
PageInspectorBase (page inspector base class): Detect each URL in the page list one by one, and detect potential problems in the page. Similar to the process of cutting ingredients, cooking and loading dishes according to the order, this component is responsible for detecting the URLs in the list of pages to be detected one by one, and detecting problems on each page. It can use a series of detection methods and rules to determine whether there are potential problems with the page.
DataReporterBase (data report base class): After the problems collected by the detection are further sorted out, the report is sent. Similar to the waiter providing ready-made meals to customers, this component is responsible for sorting out and summarizing detected problems and generating corresponding reports. Reports can include information such as a description of the problem, severity, and URLs of related pages. Reports can then be sent to relevant stakeholders, such as development or operations.
3.2 Patrol
Based on the above three base classes, different inspectors (inspectors) are developed according to different inspection scenarios. Each inspector includes three subclasses that inherit the above three base classes, and the subclasses that inherit the base class inspectors The inspector realizes its own individual requirements by overriding/extending the base class method. The following is a minimalist inspector example:
// data-provider.ts
export class DataProvider extends DataProviderBase {
// 实现特定的页面列表获取逻辑
async fetchData(args) {
return await axios.get('https://xxx.xxx').then(res => res.data.urlList)
}
// 每隔15分钟获取一次待检测列表
async schedule() {
return [{cron: '*/15 * * * *',args: {}}]
}
}
// page-inspector.ts
export class PageInspector extends PageInspectorBase {
async onPageOpen(page, reporter: PageReporter, data) {
const pageTitle = await page.evaluate('window.document.title')
console.log('这里可以获取到页面title', pageTitle)
}
}
// data-reporter.ts
export class DataReporter extends DataReporterBase {
async beforeFeishuNotify(data: InspectorReportBase) {
console.log('在飞书通知前做点什么', data)
return data
}
}
3.3 Inspection main program
In the inspection system, the detection task of each page is an independent asynchronous task, and the sorting and sending of each detection report is also an independent asynchronous task. In order to facilitate the management and maintenance of these asynchronous tasks and the storage and delivery of task messages, the inspection system uses Redis combined with Bull as the asynchronous task management tool of the inspection system.
Redis is an in-memory database that provides high-performance data storage and access capabilities.
Bull is a Redis-based task queue library, which provides the functions of task scheduling, execution and message passing.
With the inspector and asynchronous task management capabilities, the main work of the main program is as follows:
-
Define tasks: use Bull to create two task queues, page_queue is used to store "page detection tasks", reporter_queue is used to store "report generation tasks".
-
Production task: In the inspection system, the producer (main program) of the page detection task and report generation task is responsible for adding the task to the corresponding queue. When the inspector needs to perform page detection, the producer adds the page detection task to the page_queue; when it needs to generate a report, the producer adds the report generation task to the reporter_queue.
-
Consumption task: The task consumer (main program) in the inspection system is responsible for obtaining and executing tasks from the task queue. A detection task will have >=1 page detection task, and the page inspector PageInspector introduced above will execute the page Check, and then store the detection report in Redis. When all the pages of the detection task are detected, the reporter_queue task is created and consumed by the DataReporter of the inspector.
4 bayonet service
After introducing the inspection system, let's see how to integrate the bayonet service into the self-inspection system.
The main function of the checkpoint service can be summed up in one sentence: access to the existing architecture of the inspection system, expose a remote interface to the outside, provide the interface caller with the ability to actively detect the page, and then send the detection report back to the caller.
Compare the difference between the existing inspection system and the bayonet service:
From the introduction of the inspection system architecture above and the analysis of the above table, we can see that the development of the bayonet service is to customize and implement an inspection device based on the inspection device architecture of the inspection system.
4.1 Bayonet Service Running Sequence
Before starting to develop the checker for the bayonet service, let’s sort out the running sequence of the entire bayonet service:
Among them, the main development tasks of bayonet service: steps 2, 3, 4, and 7.
4.2 Create task interface
We mentioned above that patrol inspection is a post-detection method, so the main capability of the DataProviderBase (data provider base class) of the patrol inspection system is: "Periodic polling to receive the list of pages to be inspected provided externally".
For the bayonet service, the detection task is actively created by the detection party, so we don't need to pay too much attention to the implementation of DataProviderBase, but start an api service to be responsible for creating the detection task. The sample code is as follows:
app.post('/xxx.xxx', async (req, res) => {
const urls = req.body?.urls // 待检测url列表
const callBack = req.body?.callBack // 调用方接收报告的回调接口地址
const transData = req.body?.transData // 调用方需要在回调中拿到的透传数据
// 巡检系统检测任务创建函数
newApp.createJob(urls.map(url => ({ url,
// 在redis任务队列中传递的信息
pos: { callBack, transData },
})),
jobId => { // 返回任务id给调用方
res.json({ taskId: jobId })
}
)
})
4.3 Page Detection
PageInspectorBase (the base class of the page inspector) is the focus of the transformation of the bayonet service. In terms of subclass implementation of this base class, we need to do the specific detection cases mentioned above to be implemented. There are mainly two types of detection cases:
1. Request resource type detection case: override the onResponse method in the subclass, and execute different detection logic for different resource types;
2. Runtime detection case: override the onPageOpen method in the subclass, inject the js script through the Page object passed in from the base class, and perform page runtime detection;
//
页面检测类
class PageInspector extends PageInspectorBase {
// ...
// 针对不同资源类型检测方法配置Map
checkResponseMethodsMap = new Map([['image', this.checkImageResponse]])
// 请求资源型检测入口 针对请求资源进行检测
async onResponse(response: Response, reporter: PageReporter, data: IJobItem) {
const resourceType = response.request().resourceType()
const checkMethod = this.checkResponseMethodsMap.get(resourceType)
await checkMethod(response, reporter, data)
}
// 检测图片资源
async checkImageResponse(response: Response, reporter: PageReporter, data: IJobItem) {
// ...
if (imageCdnList.includes(url)) {reporter.add({ errorType: "图片类型错误.非cdn资源" })}
// ...
}
// 运行时检测入口 在页面打开时执行注入的js脚本进行运行时检测
async onPageOpen(page, reporter: PageReporter, data) {
// ...
const htmlText = await page.evaluate('window.document.documentElement.innerHTML')
const phoneRegex = /\b((?:\+?86)?1(?:3\d{3}|5[^4\D]\d{2}|8\d{3}|7(?:[35678]\d{2}|4(?:0\d|1[0-2]|9\d))|9[189]\d{2}|66\d{2})\d{6})\b/g;
let phoneMatch: RegExpExecArray
let collectMessage = []
while ((phoneMatch = phoneRegex.exec(html)) !== null) {
const phone = phoneMatch[1];collectMessage.push(`手机号码:${phone}`);
}
collectMessage.forEach(val => {reporter.add({ errorMessage: `敏感信息：${val}`})})
// ...
}
// ...
}
RegExp.prototype.exec()
JavaScript RegExp objects are stateful when the global or sticky flag is set (eg /foo/g or /foo/y). They record the position since the last successful match in the lastIndex property. Using this feature, exec() can be used to iterate over multiple matches in a single string (including captured matches), in contrast, String.prototype.match() will only return the matched result.
4.4 Report and callback
After the detection task is executed, a new "report generation task" will be created in the reporter_queue, and the main program calls the subclass that inherits DataReporterBase to perform the following operations:
-
Sort out the detected items one by one, classify the collected errors, and sort out the report source data;
-
According to the report source data combined with the ejs template to generate static html and upload it to get the address of the online detection report;
-
Callback instrumentation report to caller.
4.4.1 Generate a report template using ChatGPT
The UI design level of most of the research and development is obvious to all. Instead of smashing the style and interaction of the report template, it is better to leave it to AI. Input the pre-defined data structure and requirements, ChatGPT can quickly give a template, and then only need a little A little bit of tweaking will get you to a usable state.
Q: 我需要你写一个ejs模板，这个模板用来展示一个统计报告，为了样式和交互方便，模板会通过内置bootstrap的css样式和Jquery，你可以用到他们来实现html元素的组合和设置交互，具体的数据结构如下：
interface CheckResult {
name: string;
status: 0 | 1;
message: string;
severityLevel: 0 | 1;
}
interface DetectionResult {
url: string;
status: 0 | 1;
checkResults: CheckResult[];
}
interface InspectorReport {
taskId: string;
reporterUrl?: string;
detectionResults: DetectionResult[];
startTimeStr: string;
endTimeStr: string;
duration: number;
pageTotal: number;
taskStat: {
completed: number,
waiting: number,
active: number,
delayed: number,
paused: number,
failed: number,
}
}
export interface InspectorReportBase {
manifest: IManifest,
data: InspectorReport;
}
export interface IManifest {
name: string
desc?: string
notifyTo?: string[]
}
A: 好的，以下是基于提供的数据结构设计的ejs模板：
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<link href="https://xxx.xxx.css" rel="stylesheet">
<script src="https://xxx.xxx.js"></script>
<title>
<%=data.manifest.desc%>
</title>
</head>
<body>
<div class="container">
<h2 class="text-center">
<%= data.manifest.desc %> 统计报告
</h2>
<div class="row">
<div class="col-md-4">
<div class="card">
<div class="card-body">
<h5 class="card-title">任务信息</h5>
<p class="card-text">任务ID: <%= data.data.taskId %>
</p>
<p class="card-text">检测开始时间： <%= data.data.startTimeStr %>
</p>
<p class="card-text">检测结束时间： <%= data.data.endTimeStr %>
</p>
<p class="card-text">检测时长： <%= data.data.duration %> 秒</p>
<p class="card-text">检测页面数： <%= data.data.pageTotal %>
</p>
<!--以下省略 -->
</body>
</html>
AI generated online report template:
4.4.2 Decoupling from the caller
As a service provider, the bayonet service does not want to pay too much attention to who the caller is, so how to achieve decoupling from the caller?
——Looking back at the "Create Task Interface" link above, the input parameters of this interface design include callBack and transData fields:
-
callBack: When the detection task ends, the bayonet service will call this address to return the detection report;
-
transData: When the detection task ends, the data that needs to be transparently passed to the callBack;
After the page detection task is completed, in the callback test report link, the bayonet service will fetch these two values from the cache of the redis queue task, and use the POST request to send the report and transData to the callback.
卡口服务回调示例代码
axios.post(callBack, {
data: { msg: "本次检测检测报告如下：xxxxx", transData: `透传的数据如下：${transData}` }
})
In the follow-up planning, in order to make the bayonet service adapt to the different needs of more scenarios, referring to the concept of the back-end micro-service registration center, a simple abstract model of the registration center can be realized to further decouple the bayonet service and its caller The logic between them can expand more functions at the same time: custom detection items, custom report templates, etc.
5 summary
For bayonet service, learning and reading the source code of inspection is an important pre-work. By in-depth understanding of the implementation details and underlying architecture design of the inspection system, you can better understand how the inspection system works, so as to better customize and expand. These experiences have also helped improve your coding and design capabilities. It can be applied and practiced in subsequent technical projects. I hope that the development students who have read this article can gain something from this practical summary~
Citation/Reference Link
GitHub - OptimalBits/bull
RegExp.prototype.exec() - JavaScript | MDN
Text: Hang Fei
This article belongs to Dewu technology original, source: Dewu technology official website
It is strictly forbidden to reprint without the permission of Dewu Technology, otherwise legal responsibility will be investigated according to law!