Slow queries appear on the live network. In the case of the order of 5 million, the query speed of a single table is more than 30 seconds. SQL needs to be optimized. The SQL is as follows:
I constructed 5 million pieces of data in the test environment to simulate this slow query.
Simply put, it is to query which users are available under certain conditions. Very simple sql, you can see that the query takes 37 seconds.
Let’s talk about the distribution of the app_account field. 5000 different random numbers were randomly generated, and then distributed to these 5 million pieces of data. On average, there will be 1000 duplicate values for each app_account, and a total of 5000 types.
Second, look at the execution plan
As you can see, I added an index on the group by field and used it.
Three, optimization
To be honest, I don't know how to optimize, how can I optimize this stuff! First of all, the following ideas are useless.
Idea one:
Order by null should be added at the end; to avoid useless sorting, but in fact, it has little effect on the time-consuming results and is still very slow.
Idea two:
The where condition is too complicated and there is no index, which causes the query to be slow, but I added a composite index to all the fields of the where condition, which is still useless
Idea three:
Since group by is slow, try changing to distinct? ? (This is the magical place mentioned in this blog)
Damn it? ? ? ! ! ! What is the situation, so fast in an instant? ? ! ! !
Although I know that there is a small performance gap between group by and distinct, I did not expect that the gap is so big! ! ! Great discovery! !
4. Do you think this is over
I really hope that this is the end, then the problem will be solved very simply, and by the way, a new piece of knowledge has been discovered by the self-righteous.
but!
After the bug was transferred to the test, it was still more than 30 seconds after the test! ? What's happening here! ! ? ? ?
Of course I don't believe it anymore. I tried to execute SQL on the test computer, and it took more than 30 seconds. . .
I went back to my computer, connected to the same database, and executed sql, 0.8 seconds! ?
Under what circumstances, the same database, the same SQL, how can the difference between the execution of two computers be so big!
Later executed directly on the server:
Drunk, it was still more than 30 seconds. . . .
That seems to be a problem with my computer.
Later, I used the computer experiments of several colleagues, and finally came to the conclusion:
It's because I use SQLyog!
Hey, now I found out that only using sqlyog to execute this "optimized" sql will take 0.8 seconds, and it will take more than 30 seconds to execute directly on navcat and the server.
That is the problem of sqlyog, and it is not clear whether sqlyog is optimized or not. The problem of slow query is still being solved (I think the problem may lie in the parameters of mysql itself).
Here is just a record of this pit, sqlyog executes sql speed, and the server executes sql speed, there is a huge difference in some SQL, which is not reliable.
5. Follow-up (not yet resolved)
Thank you for your suggestions, and I will reply to the progress of the next problem:
1. The so-called sqlyog query is fast and the command line query is slow. The reason has been found. It is because sqlyog will add limit 1000 by default after the query statement, so it is very fast. This problem is no longer tangled.
2. The methods I have tried (none of them worked):
①Add an index to the app_account field.
②Add order by null after the sql statement.
③Adjust the query order of the fields in the where condition, put the index in the front.
④ Add a combined index to all the fields of where conditions.
⑤Using a subquery, first check the content in the where condition, and then remove the duplicates.
The test environment and the live network environment data are still a bit different. I posted a picture of SQL execution on the live network (1 minute...):
Six, the final solution
Thanks to @言枫大佬 on the 42nd floor in the comments!
After your reminder, I did find that in the explain execution plan, the index does not seem to use the idx_end_time I created.
Then I tried it decisively on the live network and forced to specify the idx_end_time index. The result was only 0.19 seconds!
At this point, the problem is solved. In fact, my colleagues also wondered yesterday whether the index of this table was built too much, which led to the wrong use. Originally, idx_org_id and idx_mvno_id were used.
Now it’s ok to force idx_end_time to be specified!
Finally, compare the execution plan before and after the change
Before the change (the query takes about 1 minute):
After the change (the query only takes a few hundred milliseconds):
