With feelings and dry goods, WeChat search [ San Taizi Ao Bing ] pays attention to this different programmer.
This article has been included on GitHub https://github.com/JavaFamily , with the complete test sites, materials and my series of articles for interviews with major companies.
background
With the development of business, the system architecture has changed from a monolithic architecture to a service-oriented architecture, a horizontal layered architecture, and then a microservice architecture.
In the service grid, the interaction between services and services is becoming more and more complex. How to design an interface elegantly and what aspects need to be considered? Especially for public services (such as BFF) that need to provide public domain name interfaces to the outside world, how to ensure security? I have compiled some common measures since my work and how to implement them:
Data validity check
Legality verification includes: regular verification and business verification;
general verification: including required field verification, length verification, type verification, format verification, etc.;
business verification: according to actual business, For example, the order amount cannot be less than 0, etc.;
Idempotent design
The so-called idempotence, to put it simply, means that the result of multiple calls to an interface is the same as that of one call. It is necessary to be idempotent only when the data changes. Some interfaces are naturally guaranteed to be idempotent.
For example, in the query interface, some changes to the data are constant, and there are no other records and operations, which can also be said to be idempotent. In other cases, it is necessary to prevent repetitive operations from happening for all data modification and status changes. The idempotence of the interface is realized indirectly to prevent the impact of repeated operations.
Another example is the more common addition and subtraction of GMV in our e-commerce companies. No matter how many times it comes, the result should only be added or subtracted once, otherwise it will cause the amount of money to be wrong or even cause capital loss.
Request level: The result of multiple executions is the same
business level: The same user does not place orders repeatedly, the goods are not oversold, and MQ does not repeat consumption
The essence of idempotence is the problem of distributed locks. Distributed locks can be implemented through redis or zookeeper.
In a distributed environment, lock the globally unique resource to serialize the request, which actually acts as a mutual exclusion lock to prevent duplication and solve idempotence
safety
1. Data encryption
We know that data is easy to be captured during transmission. If it is directly transmitted, such as http protocol transmission, then the data may be obtained by anyone during the transmission process.
Therefore, the data must be encrypted. A common practice is to md5 encrypt sensitive data such as ID numbers. The current mainstream approach is to use the https protocol to add a data security layer (SSL layer) between http and tcp, which is responsible for data encryption and decryption. How to configure and use https, please read my historical articles to study by yourself.
Symmetric encryption: The key is unchanged during the encryption process and the decryption process. Common algorithms are DES and AES; the advantage is that the encryption and decryption calculation speed is fast; the disadvantage is that the two parties must agree on the key before data transmission. If the key is leaked, the encrypted information will be insecure.
Asymmetric encryption: The keys appear in pairs. After one key is encrypted, another key is used to decrypt; the private key is placed in the server file, and the public key can be released to anyone; the advantage is that it is more secure than symmetric encryption. However, the speed of encryption and decryption is much slower than that of symmetric encryption. The RSA algorithm is widely used;
The implementation of https happens to be a combination of two encryption methods, integrating the advantages of both parties, and is relatively good in terms of security and performance. For the code implementation of symmetric encryption and asymmetric encryption, jdk provides related tools that can be used directly. This article will not introduce much.
2. Data signing
Introduce 3 types of data signing security strategies: summary [KEY] , signature [certificate] , signature + encryption [certificate]
| security strategy | description | Security Level |
|---|---|---|
| Abstract [Key] | Combine the data and Key (custom contract password) to summarize | The security level is low, and the security of the contract key is very low. In the case of contract key security, it can basically guarantee the non-tampering of data. |
| Signature | Use certificates and asymmetric signature algorithms to sign data | In the security level, the non-tampering and non-repudiation of data can be guaranteed, but the privacy of data cannot be guaranteed |
| Signature-encryption [certificate] | Use certificates and asymmetric algorithms to sign data, and use a one-time password and symmetric algorithms to encrypt data | The security level is high, which can guarantee the non-tampering and non-repudiation of the data, and the privacy of the data. |
- Confidentiality: Don't see without permission
- Integrity: no tampering
- Availability (Availability): prevent unavailability
- Non-Repudiation: Users cannot deny their actions
Abstract [KEY] process : combine the data to be submitted into a string in some way, and then generate an encrypted string through md5. This string is the signature of the data packet, such as:
str：参数1={参数1}&参数2={参数2}&……&参数n={参数n}$key={用户密钥};
MD5.encrypt(str);
Abstract [KEY] Principle : The Hash algorithm is irreversible, and the calculation result is unique. When the privacy of the key is guaranteed, the integrity can be guaranteed.
Abstract [KEY] Defect : The privacy of the key is difficult to guarantee, and it is transmitted in plain text
Signing [certificate] process : the client performs an md5/SHA calculation on the plaintext, encrypts the calculated value with the private key to obtain the ciphertext, the client sends the plaintext and ciphertext to the server, and the server uses the public key for the ciphertext Decrypt the value A, and the server does an md5/SHA calculation on the plaintext to obtain the value B. Compare the value A and the value B. The same verification is passed, which can guarantee non-tamperability and non-repudiation, but cannot guarantee the privacy of the data ( Clear text transmission)
Signature + encryption [certificate] process : the client generates a random string as the password, and then encrypts the password with the public key of B to generate ciphertext C, encrypts the plaintext of A with password to generate ciphertext B,
and makes the plaintext of A as MD5 The calculated value of /SHA is encrypted with the private key of A to obtain the signature D, and the ciphertext B, ciphertext C and signature D are sent to the server. The server decrypts the text C with the private key to obtain the password, and then decrypts the text B with the password. Obtain the plaintext of A, and the signature can be used to verify whether the sender is A and whether the data sent by A has been modified by a third party.
It can be assumed that there is a malicious party X, pretending to be A, and sending the ciphertext B (password generation). After the ciphertext C server receives the data, it can still decrypt the plaintext normally, but it cannot prove that the plaintext data was sent by A It was sent by malicious user B. The meaning of signature D is that A signs itself, and the server can verify it. Since X does not have A's private key, this signature cannot be impersonated and will be recognized by the server.
3. Timestamp mechanism
The data has been encrypted, and the hotel can’t see the real data if the data is captured; however, some criminals don’t care about the real data and make a malicious request directly after receiving the data. At this time, the simple method can consider the time stamp mechanism. When the current time is added to the request, the server will compare the time in the message with the current time of the system to see if it is within a fixed time range, such as 5 minutes. Maliciously forged data cannot change the time in the message. More than 5 minutes can be regarded as an illegal request.
The pseudo code is as follows:
long interval=5*60*1000；//超时时间
long clientTime=request.getparameter("clientTime");
long serverTime=System.currentTimeMillis();
if(serverTime-clientTime>interval){
return new Response("超过处理时长")
}
4. AppId mechanism
Most websites require a user name and password to log in. This is actually a security mechanism; the corresponding service can also use this mechanism. Not everyone can call it. Before calling the service, you must first apply for a unique appid and provide the relevant password. Key, the appid+key information needs to be provided when calling the interface, and the server will verify it.
The appid is randomly generated using letters, numbers, special symbols, etc. The generated unique appid depends on whether the system needs to be globally unique; regardless of whether it is globally unique, the following attributes are best:
Increasing trend: so that when saving the database, the performance of the index is better
Information security: randomly generated, not continuous, easy to be found regular
The common methods for generating the globally unique Id include the snowflake method, etc.
snowflake
The above schematic diagram describes the binary composition structure of a serial number.
The first digit is not used, it is always 0, which means a positive integer; the next 41 digits represent the timestamp, accurate to milliseconds. To save space, you can define this timestamp as the number of milliseconds elapsed since a certain point in time (Java default is 1970-01-01 00:00:00).
The next 10 bits are used to identify the working machine. If there is a cross-IDC situation, these 10 bits can be divided into two, one part is used to identify IDC, and the other is used to identify the server; the last 12 bits are the serial number, self-increasing .
The core idea of snowflake is the reasonable distribution of 64bit, but it is not necessary to strictly follow the division method shown in the figure above. If there are fewer machines, you can appropriately shorten the length of the machine id and leave it for the serial number.
5. Blacklist mechanism
If this appid has performed many illegal operations, or there is a special black system, after analysis, this appid will be directly blacklisted, and all requests will directly return an error code;
We can set a state for each appid, such as initial state, normal state, black state, closed state, etc.; or we can directly save the blacklist list through the distributed configuration center, and check whether it is in the list every time. can;
Current limiting mechanism
Commonly used current limiting algorithms include: token bucket current limiting , leaky bucket current limiting , counter current limiting ;
The principle of the token bucket current limiting token bucket algorithm is that the system puts tokens into the bucket at a certain rate, and discards the token when it is full; when the request comes, the token will be taken out of the bucket first. If the token can be obtained, You can continue to complete the request, otherwise wait or refuse service; the token bucket allows a certain degree of burst traffic, as long as there is a token, it can be processed, and it supports multiple tokens at once;
The principle of the leaky bucket flow limiting algorithm is to flow out requests at a fixed constant rate, and the incoming request rate is arbitrary. When the number of requests exceeds the capacity of the bucket, new requests wait or deny service; it can be seen that the leaky bucket algorithm can force limit data transfer speed;- Counter current limit
counter is a relatively simple and rude algorithm, mainly used to limit the total number of concurrency, such as database connection pool, thread pool, the number of spikes; counter current limit as long as the total number of requests within a certain period of time exceeds the set threshold Value is current limited;
Based on how the above algorithm is implemented, Guava provides the RateLimiter tool class based on the token bucket algorithm:
RateLimiter rateLimiter = RateLimiter.create(5);
The above code means that only five concurrent requests are allowed to be processed in one second. The above method can only be used for single-application request current limiting, not global current limiting; at this time, distributed current limiting is required, which can be implemented based on redis+lua;
to sum up
In fact, whether the interface is design or development, if it is not particularly urgent, everyone can think more, so that your system will be more stable, there will be fewer bugs in the process of going online and testing, and from the perspective of personal improvement, think more. Is a good thing.
Many times everyone is complaining: Oh, my company is small, and my school cannot grow in such an environment. Fool, a lot of times masters come in this way, but everyone's attitude is different for the same thing, and the result will be different over time.
Alright, everyone should be at work now. I stay up late on duty and I'm still on the spot (the article was written on weekends, I will write a summary now), I am Ao Bing, the more you know, the more you don’t know, we See you next time.
Talk
Ao Bing compiled his interview essay into an e-book with 1,630 pages!
Full of dry goods, the essence of every word. The content is as follows, as well as the interview questions and resume templates I summarized during the review, which are now given to everyone for free.
Link: https://pan.baidu.com/s/1ZQEKJBgtYle3v-1LimcSwg Password:wjk6
This is Ao Bing. The more you know, the more you don’t . Thank you for your talents: likes , favorites and comments . See you in the next issue!
The article is continuously updated. You can search for " San Tai Zi Ao Bing " on WeChat to read it for the first time, and reply to [ Information ] I have prepared the interview information and resume template of the first-line manufacturers. This article has been included on GitHub https://github.com/JavaFamily , There are complete test sites for interviews with major factories, and Star is welcome.