SQL implementation process:
1, the implementation of SQL, sql parsing engine is started
2, data types and data types of the database table definition is inconsistent, the database engine is automatically converted
3, database table defines more than one index, sql engine will help you choose the best one
4, the database engine optimization will be holding a good sql command semantic data hard to find, and then find the data back (if at this time to return a result set is too large, will cause the database IO busy, sql efficiency will be greatly damaged, so we generally use the reason lies in this page)
SQL optimization
First, the index: In general, the size of the database effective in hundreds of thousands and millions of levels the fastest time, even if there is a table associated with less complex, can greatly improve the operating efficiency of sql. It does not require code changes, low cost, effective obviously.
1, the index generally applied to the keyword query, the query if there are multiple keywords, you can also add a composite index, write sql need to pay attention, sql index fields and fields need to be consistent, otherwise the index will be invalid.
2, do not use the function in a query = front, otherwise it will cause the index does not take effect.
3, index fields to distinguish between the relatively high degree of
4, the establishment of a composite index, sql can continue to improve operating efficiency, but do not blindly, to distinguish between the same degree, if discrimination is not high enough, do not add up, more fields possible to distinguish between high field on the front, Also, note that the length of the index, the index taking into account the balance to the length and discrimination index of
5, the index will greatly improve query efficiency, but also loss after query modification efficiency , taking into account the balance to be noted that, once inserted in use, the effect of multiple queries the table preferably also to be noted that the combination of the index index will inevitably increase the length of the index will increase the storage space, and a discrimination index length attention balance
6, mysql actually supports full-text indexing, not tested efficiency, the normal use of full-text indexing is to use lunce, and solr above it and now normalizing elastisearch.
II: sub-library sub-table partition
1, sub-libraries
Can follow the business sub-libraries, database concurrency shunt pressure, the more organized of the database table, at least it is more easy to find, we had a system to query libraries and libraries (more frequent additions and deletions to the table) separated, so if there are large query, does not affect the system library
2 points table
The index for the amount of data to deal with raking in millions, thousands of levels the amount of data used for good, barely able to make do, but if the amount of data of millions of levels, the index can not do anything, because the single index file may have been hundreds of megabytes or more, then, turn to our points table partitioning debut.
Sub-table method:
a, if the business process is there, then we usually design a historical archive table or tables used to store historical data, this will ensure more efficient real-time data
b, for a queen-table can be divided into multiple tables based on search criteria, such as time, we can be two weeks or 10 days of data into a table (see the specific amount of data, think 3000W is a limit, most good control to one million level), every 10 days, we will automatically create a database table, and then insert the data, so, according to the time of the query, you must first locate the table to get to that number, so that efficiency can be improved significantly, of course, so there are problems to solve, such as cross-table, multiple tables require union, but also across tables can not support index
c, the above method is the most primitive part table we achieved directly through the program and database solution, now available in some mature software such as mycat, also supports the points table, before we have a company engaged specialize distributed these products appear databases across the table, you can use the program without the union, but also the index take effect, but need to have some grasp of the product
d, in general, a large table in the database after all, only a small part, this requires only a small part of a large table points table on it, no need to divide the table small table also increase the maintenance difficulty of development
3, partition
Three: Database Engine
A ten million complex sql test data volume, efficiency myisam can probably 1-2 times faster than innodb, although efficiency improvement is not very obvious, but has been raised, then checked some of the information, said the reason mysiam fast because his data storage structure index storage structure and innodb not the same, mysiam index structure is stored in memory, of course, mysiam have weaknesses, that is, he is table-level locking, and innodb is row-level locking, so, mysiam apply once inserted, the table multiple queries, or separate read and write in the library reading tables, insert and delete operations for modifying the table more frequently, it is inappropriate
Four: Pretreatment
1, real-time data (day data) by abstraction of the business, which can be placed in the cache, to enhance the operational efficiency of the system.
2, historical data, large tables and there is historical data table associated by conventional sql difficult to optimize, but the data usually has a common, that is, the next day to inquire previous day's data analysis reports do, that is to say on the timeliness requirements is not high, this solution is pretreated. These practices are associated with complex tables sql written timed tasks performed at midnight, to the implementation of the results into a results table next direct query results table, so, efficiency can be very significantly improved
3, Table solr association result is stored or elastisearch in order to enhance efficiency, our current project is so treated.
Five: Like Query
As we all know, like "% str%" does not support indexing, "str%" number is to support the index, therefore, if the business allows the method before you can use the matching database to quickly locate the data in the result set then like matching, If the result set is not large, it can greatly enhance the operational efficiency of the business and the need for flexible work procedures, if the business before it is allowed to match, it can be used to solr or elastisearch fuzzy matching, but fuzzy matching there is a premise, the original data is a string of words, not with special symbols, such as #, &,%, etc., this will result in segmentation allowed, leading to incorrect query results
Six: separate read and write
In large database concurrency, the best approach is to scale out, increasing the machine, to improve the resistance to concurrent, but also both data backup function