Spent almost more than one month's time, any talent, Qi Shuai and I work together to complete the three reconstruction work before and after the end of the code Privacy Information Management System.
Among them, Qi Shuai independently of the front-end code reconstruction work, talent responsible for any server-side code in most of the "write" operation as well as with third-party platform docking code works, I was responsible for the service and most of the front-side code docking "read" operations and compliance with privacy rules docking platform preset code works.
Here to talk about my experience during the reconstruction process and views.
The reason reconstruction
The reason is reconstructed, because before the system even though it can continue to develop, but because of the code structure and business logic, follow-up maintenance will be more and more difficult, I understand that the main problem is:
- Code structure relatively free, server-side logic scattered, difficult to maintain, good readability
- There are hard-coded phenomenon, front-end, back-end maintenance were maintaining profiles of the same data, prone to problems
- Database design field is not uniform, resulting in increasing the number of processing logic
- The front end of the secondary work a lot of data processing
Reconstruction is to solve the problem, so for the above issues, the reconstruction idea is:
- Selection Egg framework in place Koa, the code uses a consistent set of conventions to develop, so try to clear logic
- Data related configuration files maintained by the back-end, large changes stored in the database, it is necessary to form an interface back to the front
- Field database design optimization, combined with the relevant configuration file, do not reduce the processing logic
- Most of the server to complete the data processing in order to form a more REST interface specification data back to the front end, the front end of the data can be displayed directly
Selection Framework
Before the reconstruction, I conducted a survey for the BFF, were compared against several Web development framework , a more feasible solution is to Egg and Koa (Express), and finally chose instead with Egg Koa as a framework for development, mainly based on the following reasons:
- Egg packaging is carried out on the basis of Koa, fully compatible with Koa middleware, and more friendly document
- Best practices ants gold dress package, there are a large number of plug-in support, complete and perfect
- Egg is more suitable for enterprise development, agreed to replace the configuration, you can achieve a unified coding standard, improve code maintainability
Koa is more flexible than the Egg, which leads to code that no matter how to write, are able to run, and many supporting the need to support themselves with, and Egg offers a complete solution that is more suitable for stringent requirements, long-term maintenance of corporate update projects, and better able to achieve progressive enhancement of research and development experience. When the service development to a certain extent, can abstract from the Egg, an intermediate piece belonging to a specific service package, a specific pattern, or even plug frame.
Use the Egg, feel the biggest benefit is twofold:
(1) a large number of convention
A large number of convention, not only limits the developer, and gave developers free.
In Egg agreement, MVC architecture is very clear, reasonable directory structure, you can choose on demand.
egg-project
├── package.json
├── app.js (可选)
├── agent.js (可选)
├── app
| ├── router.js
│ ├── controller
│ | └── home.js
│ ├── service (可选)
│ | └── user.js
│ ├── middleware (可选)
│ | └── response_time.js
│ ├── schedule (可选)
│ | └── my_task.js
│ ├── public (可选)
│ | └── reset.css
│ ├── view (可选)
│ | └── home.tpl
│ └── extend (可选)
│ ├── helper.js (可选)
│ ├── request.js (可选)
│ ├── response.js (可选)
│ ├── context.js (可选)
│ ├── application.js (可选)
│ └── agent.js (可选)
├── config
| ├── plugin.js
| ├── config.default.js
│ ├── config.prod.js
| ├── config.test.js (可选)
| ├── config.local.js (可选)
| └── config.unittest.js (可选)
└── test
├── middleware
| └── response_time.test.js
└── controller
└── home.test.js
According to its agreement to develop, developers no longer need to care about what features should be placed where, how what services should be called. When a team of developers to follow a unified convention, they do not care about these things, only care about how to achieve specific business logic, because other things are to help you get the Egg
(2) improve the function
Egg is a framework for enterprise-class research and development, it also provides many of the core enterprise application development features such as unit testing, local debugging, logging, exception handling, etc., encapsulates the cookie and session operation, we only need to follow it the document configuration, you can achieve most of the demand.
To breakpoint debugging , for example, other frameworks may also need to manually browse or IDE configuration, and the Egg is very simple to implement, it is to start the command by the npm run devchange npm run debug,
Then open the console output browser Devtoolsto debug in Kay's debugger, break point corresponding to the address of the back:
Very easy.
In addition to the functional framework provided directly, you can use many third-party plug-ins to help us achieve functions such as parameter checking and exception handling , use egg-validatecan greatly simplify our validation logic parameters in the Controller:
// 查询网络上传全部详细信息的校验规则
const indexRule = {
id: { type: 'string', trim: true, format: /^.{24}$/, message: '非法ID' }, // Mongo生成的ID长度为24位
packageName: { type: 'string', trim: true },
}
class LogController extends Controller {
// 获得某个 App 的网络上传全部详细信息
async index() {
const { ctx } = this
const { params } = ctx
// 参数校验
ctx.validate(indexRule, params)
// 参数
const id = params.id
const packageName = params.packageName
// 响应内容
const data = await ctx.service.network.findPrivacyNetwork({ id, packageName: packageName.trim() })
ctx.response.status = this.config.httpCodeHash.ok.code
ctx.response.body = { count: data.length, items: data }
}
}
Of course, Egg such as a feature-rich, large and comprehensive framework, use is also there are some problems, will be mentioned later.
Configuration process
Said earlier, before possible with a configuration file, such as encryption method, the front and rear ends may be maintained, if the new encryption method, it may need to change the front and rear ends of the code, but also easy to maintain due to the inconsistency, resulting bug .
So this reconfiguration, such profile have got the service side, under the config directory, our business will be required to put all the configuration files customin the directory, and then config.default.jsimport
In use, the server directly this.configcan get to the corresponding profile, and the front end if necessary the corresponding profile, provides a direct interface to the data provided
This will not only need to maintain only one configuration file, and if the data changes, you only need to change here once, do not search all the code changes, do not avoid the hard-coded elegant phenomenon.
Optimized database field
The part I did not actually say anything, because my experience level and familiarity with the business are not enough, and the business is constantly changing, increasing demand is likely to lead to the original design of the field, not in the late suit one's needs. And I want to design a more perfect and scalability of good data structure, not an easy thing.
In reality, when the original data structures and conflicting demands, the latter to solve a lot of trouble, resulting in many complex processing logic.
For example, system names are acquired packet is detected is .divided, for example com.android.browser, the key storage object can not have Mongo database ., the original is com.android.browserchanged to com_android_browserbe stored, so that it is changed over to change over need to be processed strings, if they already contain some special _package name also require special handling
This reconstruction, and any talent I agreed, all packages use the name com.android.browserstored in the database, since most cases are stored in the data Gzip compression, so it does not matter, but must, as keywhen, in his write-only when the replacement processing, and we added a preset rule table of data packageNamefields, used to store com.android.browser, achieved by arranging relationship with com_android_browserthe corresponding, part of the processing logic can be reduced, the processing logic and the remaining concentrated
There are some other details of the optimization, such as encryption information storage structures and Log network requests to uniform the like.
We were able to make changes to the database field, a big reason is because the historical data privacy management system is relatively less important, if we provide a buffer before the data can no longer be concerned about. If not, I want to achieve this optimization is not a good deal, because to consider a lot of compatibility issues. And although this reconstruction by optimizing may bring some benefits, but perhaps with the emergence of demand, optimized field facing new problems. So this is a test of the level of things.
Providing processed data to front end
Before the code, from the front-end interface to obtain data, or to perform a lot of processing before they can appear on the UI. The BFF concept is a front-end dispensing a small data processing back-end, front-end processing logic to reduce the data, so that the center of attention in front of the interactive logic and UI rendering on. This is equivalent to a reconstruction of events BFF, but most of the data directly from the database, rather than calling other back-end interfaces and services acquired it.
Adhering to this concept, the server code is reconstructed substantially according to the data format of the distal end UI interface, as required by the front end colleagues, it is supplied to the front end through the interface. After the front end data processing logic will get less.
Log display information such as the front end and the network requests the flag, before the front end of a mark after the reconstruction of the information according to the privacy information value, the item server according to the privacy flag, labeled is by appointment to form a front end and a tag, the labeled return information according to the front end of mthe properties to get the encryption method, directly to the corresponding CSS style, to show
/**
* 对隐私信息进行标记，加密/不加密通过m属性判断
* @param { string } target 要标记（替换）的关键词
* @param { string } string 被标记的字符串
* @param { boolean } isEncrypted 要标记的是否为加密数据
* @param { string } method 加密方法
* @return { string } markedString 返回标记后的新字符串
*/
markPrivacyInfo(target = '', string = '', isEncrypted = false, method = 'unKnow') {
if (!target) {
return string
}
// 已经匹配过的带有<em>的不再重复匹配
const reg = new RegExp(`(?<!<em>)(${target})`, 'g')
const encryptedMethod = isEncrypted ? ` m='${method}'` : ''
return string.replace(reg, `<em${encryptedMethod}>$1</em>`)
},
Further, as a typical item separated front and rear ends, the only interaction is through the interface between the front and rear ends. REST interface design specification as close as possible to distinguish the type and purpose of the request through a different HTTP methods:
// 预设规则
router.get('/v1/privacy/settings/', controller.settings.index)
router.put('/v1/privacy/settings/', controller.settings.update)
router.post('/v1/privacy/settings/', controller.settings.create)
Prescribed by the unified response format and the introduction of egg-errorsplug-ins, to achieve a unified format error response when the data formats and interfaces is successful failure. Only to achieve the structure in response to concise, and easy handling by the abnormality distal HTTP status code.
Many other existing back-end development of server-side interfaces do not strictly comply with the norms REST, or do not work in this direction, we try to do our norms and standards. (Here, too, the students learn to Qi Shuai, although just graduated, but capacity is very strong, very fair proposals to help the normative great interface.)
Are able to do to meet the requirements for front-end data formats, in addition to the reconstruction project, make the project better than better vision, another reason is that all my colleagues in the group, communicate more smoothly and peace, and all the tip came from, we know that to get the back end services like noodles cook, but unfortunately when the data is taken directly from the database back.
So this is the biggest benefit BFF addition to reducing the burden beyond the front-end data processing, it is to reduce the front and rear ends of the tear force. Some small data changes, no longer need the back-end, data ourselves through the middle layer of micro Node handle the services provided, returned to the front end's own responsibility, this process really Peace and Love.
Room for improvement
Although the reconstruction, but there is still much room for improvement and reflect
(1) About Egg
The aforementioned benefits of a bunch of Egg, indeed, for the development of enterprise-class applications, Egg indeed very easy to use. But also because of this, it becomes very thick, a lot of features, and not necessarily a lot of configuration we need them, there is no problem.
It's like we just ate a clutch cake, but it gives us a full set of cutlery French cuisine is served. Eat it? No problem, but there may be some waste. A better way or should be customized for Egg, abstract framework of their own development team and the business.
Another point is that it is possible for the support classes, when he took the talent to develop object-oriented, class-based development model, different types of data abstracted into different classes, linked to the ctxsubject, so that helps abstract behavior of different data objects, better readability.
But by context.jsthe time the mount, there is no way to get Egg instance, as a function of the form only has to be mounted in order to obtain Egg instance, and mount exported in the form of classes is not enough, there is no way to get a result on Egg examples mount configuration files.
It still makes people very uncomfortable. Egg invoke methods to support and encourage each other in the service, but when we want to have the same behavior and methods of a class of data abstraction as a Class, but less convenient to use.
Finally, although the config.default.jsconfiguration directly pegged to globalsolve this problem, but still feel uncomfortable. You may also need to look behind the research, so that we use gestures not quite right, or it itself is not welcome in this form.
module.exports = appInfo => {
// 省略
// 将config挂载到全局对象上
global.config = FinalConfig
return FinalConfig
}
(2) with respect to performance
Now When reading data, when a larger amount of data, due to the need to unpack the data, plus the data processing process, the interface in response to a long time, the performance is not very good.
When the performance of late become a bottleneck, and the like can consider introduction Redis improved query cache to speed, or the process (because the length of time the detection insensitive) when storing the detection result.
(3) on the development process
Documentation in Wiki, Bug feedback through micro letter Screenshot, Bug fix we own record in Wiki, bug fixes after the testing process is also missing.
The special nature of the project and may be the right team, but every time the code is on the line, or feel uneasy, because basically there is no test, recorded nowhere query.
(4) on business
For this system, before reconstruction code logic is one of the reasons hindering the couple accepted, another reason is more complex business logic. The former can be resolved by reconstruction, which I think is not a good solution.
For important projects, if there is a more detailed illustration of business process documentation + + rational and reasonable code comments + reasonable structure of the code, I think it can really claim maintenance on the project healthy development.
Reconstruction, for project leader or team leader should be able to stand on a higher point of view, introduce (or in the form of the document) of the entire business logic to project members, grasp the development of the project from the perspective of the overall situation, be project planning. After the team members have a clearer understanding of the entire business process and logic, in order to be better and faster process of project development.
As I project participants, this area is not very good. Because of the relatively complex business logic are not familiar with server-side development of less experienced, my main concern is how to achieve this point in the reconstruction, the function point, a little bit to increase understanding of the business, finally ending their scope of development, business logic outside the scope of or do not understand.
It should be more proactive to understand all of the business in order to better understand the code, after the take over will be easier.
to sum up
I personally feel that this refactoring server, based on the original system to ensure that the function on the realization of the reconstruction of the code, the code logic to simplify and improve the readability and maintainability of the code, and to achieve a new the demand, in general, or successful, can leave some experience.
After reconstruction, I think the project is very important for business development team, the quality of the code and maintainability of the code also has a relationship, or should pay attention to these things, basic things and attention to detail, but also in important projects initial planning and design should be more careful, quality, standardized code, the code of subsequent development should be more concerned about.
Do the usual accumulation, so that the code can be long-term maintenance, reconstruction had to postpone that day to come.
Reconstruction can improve the experience, but one of concern and the usual accumulation of detail, but also raise the level of direction.
These are my humble opinion, give you a big God laughed.