Table of contents
- 1 The amount of data uploaded and returned by the interface is too large
- 2 Return the corresponding slow query solution
- 3 Business logic complex thread pool optimization
- 4 The thread pool design is unreasonable
- 5 The lock design is unreasonable
- 6 machine problems
- 7 Cache and callback or reverse check
1 The amount of data uploaded and returned by the interface is too large
造成请求非常慢
The solution for the large amount of transmitted data uses gzip stream compression to reduce the upload speed. The interface performance bottleneck lies in the upload speed. According to the operator, the upload bandwidth is 10% of the download bandwidth, and the upload speed is much lower than the download speed. of. We usually feel that there are actually many reasons for the network card. For example, you said here that viewing niche websites will be slow. In fact, it is not a broadband problem, but these niche websites have limited service capabilities. When we watch videos, we don’t actually need to upload them. We download the video resources and play them locally.
后端工具类：
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;
public class GZIPUtils {
/**
* 字符串的压缩
*
* @param str
* 待压缩的字符串
* @return 返回压缩后的字符串
* @throws IOException
*/
public static String compress(String str) throws IOException {
if (null == str || str.length() <= 0) {
return null;
}
// 创建一个新的 byte 数组输出流
ByteArrayOutputStream out = new ByteArrayOutputStream();
// 使用默认缓冲区大小创建新的输出流
GZIPOutputStream gzip = new GZIPOutputStream(out);
// 将 b.length 个字节写入此输出流
gzip.write(str.getBytes("UTF-8"));
gzip.close();
// 使用指定的 charsetName，通过解码字节将缓冲区内容转换为字符串
return out.toString("ISO-8859-1");
}
后台解压代码
/**
* 字符串的解压
*
* @param b
* 对字符串解压
* @return 返回解压缩后的字符串
* @throws IOException
*/
public static String unCompress(byte[] b) {
try {
if (null == b || b.length <= 0) {
return null;
}
// 创建一个新的 byte 数组输出流
ByteArrayOutputStream out = new ByteArrayOutputStream();
// 创建一个 ByteArrayInputStream，使用 buf 作为其缓冲区数组
ByteArrayInputStream in;
in = new ByteArrayInputStream(b);
// 使用默认缓冲区大小创建新的输入流
GZIPInputStream gzip = new GZIPInputStream(in);
byte[] buffer = new byte[256];
int n = 0;
while ((n = gzip.read(buffer)) >= 0) {
// 将未压缩数据读入字节数组
// 将指定 byte 数组中从偏移量 off 开始的 len 个字节写入此 byte数组输出流
out.write(buffer, 0, n);
}
// 使用指定的 charsetName，通过解码字节将缓冲区内容转换为字符串
return out.toString("UTF-8");
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
}
String encodeStr = URLEncoder.encode(JSON.toJSONString(buildMenuTree(menus)), “UTF-8”);
encodeStr = Base64.encodeBase64String(encodeStr.getBytes(“UTF-8”));
String menuCompressStr = GZIPUtils.compress(encodeStr);
前端js
//data为后台返回的值
JSON.parse(unzip(data));
// 解压
function unzip(key) {
var charData = [];
var keyArray = key.split('');
for(var i = 0; i < keyArray.length; i++){
var item = keyArray[i];
charData.push(item.charCodeAt(0));
}
// var binData = new Uint8Array(charData);
// console.log('Uint8Array:' + binData);
// 解压
// var data = pako.inflate(binData);
var data = pako.inflate(charData);
// 将GunZip ByTAREAR转换回ASCII字符串
// let res= String.fromCharCode.apply(null, new Uint16Array(data));
let res= String.fromCharCode.apply(null, data);
return decodeURIComponent(Base64.decode(res));
}
// 压缩
function zip(str) {
//escape(str) --->压缩前编码，防止中午乱码
var binaryString = pako.gzip(escape(str), {
to: 'string' });
return binaryString;
}
When the Maximum call stack size exceeded occurs, it is because the amount of data is too large and needs to be used
function unzip(key) {
var charData = [];
var keyArray = key.split('');
for(var i = 0; i < keyArray.length; i++){
var item = keyArray[i];
charData.push(item.charCodeAt(0));
}
// var binData = new Uint8Array(charData);
// console.log('Uint8Array:' + binData);
// 解压
// var data = pako.inflate(binData);
var data = pako.inflate(charData);
// 将GunZip ByTAREAR转换回ASCII字符串
// let res= String.fromCharCode.apply(null, new Uint16Array(data));
//let res = String.fromCharCode.apply(null, data);
let res=''
for (x = 0; x < data.length / chunk; x++) {
res+= String.fromCharCode.apply(null, data.slice(x * chunk, (x+ 1) * chunk));
}
res+= String.fromCharCode.apply(null, data.slice(x * chunk));
return decodeURIComponent(Base64.decode(res));
}
Front-end decompression requires pako.min.js
2 Return the corresponding slow query solution
2.1 Deep pagination
The so-called deep paging problem involves the principle of mysql paging. Usually, the paging of mysql is written like this:
select name,code from student limit 100,20
The meaning is of course to check 20 pieces of data from 100 to 120 from the student table, mysql will check out the first 120 pieces of data, discard the first 100 pieces, and return 20 pieces.
It is of course no problem when the pagination is not deep. As the paging deepens, the sql may become like this:
select name,code from student limit 1000000,20
At this time, mysql will check out 1,000,020 pieces of data and discard 1,000,000 pieces of data. With such a large amount of data, the speed must not be fast.
How to solve it? In general, the best way is to add a condition:
select name,code from student where id>1000000 limit 20
In this way, mysql will take the primary key index, directly connect to 1000000, and then find out 20 pieces of data. However, this method requires the caller of the interface to cooperate with the modification, and the maximum id obtained from the last query is passed to the interface provider as a parameter, which will cause communication costs (caller: I will not change it!).
2.2 Indexing issues
This is the easiest problem to solve, we can do it by:
show create table xxxx（表名）
View the index of a table. There are too many specific indexed statements on the Internet, so I won’t repeat them here. But by the way, before adding an index, you need to consider whether it is necessary to add the index. If the field discrimination degree of the index is very low, it will not take effect even if the index is added.
In addition, the alter operation of adding indexes may cause table locking, so it must be executed during the off-peak period when executing SQL
This is the worst situation for slow queries to analyze. Although mysql provides explain to evaluate the query performance of a certain SQL, there are indexes used.
2.3 Too many joins or too many subqueries
I put together too many joins [too many subqueries]. Generally speaking, it is not recommended to use subquery, you can change the subquery to join to optimize. At the same time, the tables associated with join should not be too many. Generally speaking, 2-3 tables are appropriate.
It is relatively safe to associate several tables. It requires specific analysis of specific issues. If the amount of data in each table is small, hundreds or thousands, then the number of associated tables can be more appropriately, and vice versa.
In addition, it should be mentioned that in most cases, the join is performed in memory. If the amount of matching is relatively small, or the join_buffer is set relatively large, the speed will not be very slow.
However, when the amount of join data is relatively large, mysql will use the method of creating temporary tables on the hard disk to perform association matching of multiple tables. This is obviously extremely inefficient. Originally, the IO of the disk is not fast, and it needs to be associated.
Generally, when encountering such a situation, it is recommended to split from the code level. In the business layer, first query the data of a table, and then use the associated fields as conditions to query the associated table to form a map, and then assemble the data in the business layer.
Generally speaking, if the index is established correctly, it will be much faster than join. After all, splicing data in memory is much faster than network transmission and hard disk IO.
A very simple and practical solution to reduce the use of LEFT JOIN queries is to create an intermediate table and trade space for time.
In addition, when the amount of data is large to a certain extent, consider sub-database sub-table, you can see mycat middleware
2.4 in too many elements
This kind of problem is not easy to troubleshoot if you only look at the code. It is best to analyze it together with monitoring and database logs. If a query has in, the condition of in is added with a suitable index, and the SQL at this time is still relatively slow, so it can be highly suspected that there are too many elements in in.
Once the problem is found out, it is easier to solve it, but the elements are divided into groups and each group is checked once. If you want to be faster, you can introduce multi-threading.
Furthermore, if the amount of elements in in is too large to a certain extent and it still cannot be fast, it is best to have a limit:
select id from student where id in (1,2,3 ...... 1000) limit 200
Of course, it is best to make a restriction at the code level:
if (ids.size() > 200) {
throw new Exception("单次查询数据量不能超过200");
}
2.5 Too much data
This kind of problem can't be solved by simply tinkering with the code, and the entire data storage architecture needs to be changed. Either sub-table or sub-database + sub-table for the underlying mysql; or directly change the underlying database and convert mysql into a database specially designed for processing big data.
This kind of work is a system engineering, which requires rigorous research, scheme design, scheme review, performance evaluation, development, testing, and joint debugging. At the same time, it needs to design rigorous data migration schemes, rollback schemes, downgrade measures, and fault handling plans.
In addition to the work within the above team, there may also be cross-system communication work. After all, major changes have been made, and the way the downstream system calls the interface may need to be changed.
3 Business logic complex thread pool optimization
3.1 Loop call
In this case, the same piece of code is generally called in a loop, and the logic of each loop is consistent and unrelated.
For example, we want to initialize a list and pre-set 12 months of data to the front end:
List<Model> list = new ArrayList<>();
for(int i = 0 ; i < 12 ; i ++) {
Model model = calOneMonthData(i); // 计算某个月的数据，逻辑比较复杂，难以批量计算，效率也无法很高
list.add(model);
}
This kind of data calculation is obviously independent of each other, and we can use multi-threading to do it:
// 建立一个线程池，注意要放在外面，不要每次执行代码就建立一个，具体线程池的使用就不展开了
public static ExecutorService commonThreadPool = new ThreadPoolExecutor(5, 5, 300L,
TimeUnit.SECONDS, new LinkedBlockingQueue<>(10), commonThreadFactory, new ThreadPoolExecutor.DiscardPolicy());
// 开始多线程调用
List<Future<Model>> futures = new ArrayList<>();
for(int i = 0 ; i < 12 ; i ++) {
Future<Model> future = commonThreadPool.submit(() -> calOneMonthData(i););
futures.add(future);
}
// 获取结果
List<Model> list = new ArrayList<>();
try {
for (int i = 0 ; i < futures.size() ; i ++) {
list.add(futures.get(i).get());
}
} catch (Exception e) {
LOGGER.error("出现错误：", e);
}
3.2 Sequential calls
If it is not a loop call like the one above, but a sequential call again and again, and there is no dependence on the result between the calls, then it can also be done in a multi-threaded manner:
Look at the code:
A a = doA();
B b = doB();
C c = doC(a, b);
D d = doD(c);
E e = doE(c);
return doResult(d, e);
Then it can be solved with CompletableFuture:
CompletableFuture<A> futureA = CompletableFuture.supplyAsync(() -> doA());
CompletableFuture<B> futureB = CompletableFuture.supplyAsync(() -> doB());
CompletableFuture.allOf(futureA,futureB) // 等a b 两个任务都执行完成
C c = doC(futureA.join(), futureB.join());
CompletableFuture<D> futureD = CompletableFuture.supplyAsync(() -> doD(c));
CompletableFuture<E> futureE = CompletableFuture.supplyAsync(() -> doE(c));
CompletableFuture.allOf(futureD,futureE) // 等d e两个任务都执行完成
return doResult(futureD.join(),futureE.join());
In this way, the two logics of AB and DE can be executed in parallel, and the maximum execution time depends on which logic is slower.
4 The thread pool design is unreasonable
Detailed explanation and configuration of the seven parameters of the Java thread pool:https://blog.csdn.net/ZGL_cyy/article/details/118230264
5 The lock design is unreasonable
There are generally two types of unreasonable lock design: unreasonable use of lock type or too thick lock.
A typical scenario where the lock type is unreasonably used is the read-write lock. That is to say, reading can be shared, but you cannot write to shared variables when reading; and when writing, neither reading nor writing can be performed.
When read-write locks can be added, if we add mutex locks, the efficiency will be greatly reduced in scenarios where there are far more reads than writes.
A lock that is too thick is another common situation where the lock design is unreasonable. If we wrap the lock too much, the locking time will be too long, for example:
public synchronized void doSome() {
File f = calData();
uploadToS3(f);
sendSuccessMessage();
}
This piece of logic processes three parts in total, calculating, uploading results, and sending messages. Obviously, uploading results and sending messages can be completely unlocked, because this has nothing to do with shared variables.
So it can be completely changed to:
public void doSome() {
File f = null;
synchronized(this) {
f = calData();
}
uploadToS3(f);
sendSuccessMessage();
}
6 machine problems
fullGC, the machine restarts, and the threads are full
There are many reasons for this problem. The author encountered many reasons such as full GC caused by too large scheduled tasks, thread leaks in the code, high RSS memory usage, and machine restart and waiting.
It is necessary to combine various monitoring and specific analysis of specific scenarios, and then carry out work such as splitting large transactions, re-planning thread pools, and so on.
7 Cache and callback or reverse check
In the end, you can use caching third-party middleware if you can’t optimize it, or if some storage interfaces are too slow to return all of them directly, you can use third-party callbacks or reverse checks