In the previous sections we completed the implementation of the sql interpreter. By parsing the sql statement, we can know what the sql statement wants to do, and then we need to execute the intention of the sql statement, that is, extract the required data from the given table. To execute SQL statements, we need to understand the so-called "relational algebra". The so-called algebra is essentially the definition of operators and operating objects. In relational algebra, there are three types of operators, namely select, project and product, and the operating objects are database tables. .
The operation corresponding to select is to extract rows that meet the conditions from a given data table while keeping the fields of each row unchanged. For example, the statement "select * from customer where id=1234". After this statement is executed, the rows with the id field equal to 1234 in all records in the customer table will be extracted to form a new table.
The operation corresponding to project is to select several fields from a given data table to form a new table. The columns of the new table change, but the number of rows is the same as the original table. For example, the statement "select name, age from customer", this statement Extract two fields name and age from the original table to form a new table. The new table has fewer columns than the original table, but the number of rows is not standardized.
Product, which corresponds to the Cartesian product, operates on two tables. It extracts a row from the left table in turn and combines it with all the rows of the right table. Therefore, if the number of rows and columns of the left table is Lr, Lc, the right table The number of rows and columns of the table is Rr, Rc. In the new table of the operation result, the number of rows is Lr * Rr, and the number of columns is Lc+Rc.
Combined with the above relational algebra, after parsing a given SQL statement, in order to perform the corresponding operation, we need to construct a specific data structure called a query tree. The characteristic of the query tree is that its leaf nodes correspond to the database table and its parent node Corresponding to the relational algebra operation we mentioned above, let's look at a specific example: "select name, age from customer where salary>2000"
This statement corresponds to two relational algebra operations, namely select and project. It can correspond to two types of query trees. The first one is:
This query tree means that the project operation is first performed on the data table customer, that is, first from the table Select the two columns name and age and keep the number of rows unchanged. Then filter each row on this result and select the rows whose field salary is greater than 2000. It is not difficult to imagine that we can also have another kind of query tree, that is, do the select operation first, that is, first select all the rows in the table that satisfy salary>2000, and then on this basis, add the two names of name and age. The columns are extracted and the corresponding query trees are as follows:
You may feel that different query trees are essentially the same. In fact, different query trees have a great impact on the efficiency of data operations. The efficiency of operations corresponding to one query tree may be ten times better than that of another. times, or even a hundred times, so after we construct all possible query trees, we also need to calculate the execution efficiency of different query trees, and then select the best one. This step is called planning.
In the previous chapter, we implemented the Scan interface. This interface can be applied to several operators described previously, so in the previous chapter we implemented TableScan, SelectScan, ProjectScan, and ProductScan respectively. It should be noted that the latter three Scan objects are During initialization, an object that implements the Scan interface must be input, which can correspond to the query tree structure above. The bottom leaf node corresponds to TableScan. The Scan object input by SelectScan at the upper level during initialization is TableScan. The top node Corresponding to ProjectScan, the Scan object it inputs during initialization is SelectScan.
Let's combine these sections and implement it in code. First, we can gain perceptual knowledge. Later, we will further analyze the planning mentioned above. In the previous parsing process, we parsed the select statement, which finally constructed a QueryData object. This object contains three parts. The first is fields, which can be used to implement project operations. The second part is the table name, which can be used To construct the TableScan object, the final pred object can be combined with the TableScan object to construct the SelectScan object.
We implemented TableScan before when we studied record_manager, and also provided a test file implementation called table_scan_test.go. There is a function called TestTableScanInsertAndDelete, which constructs the data storage of a data table, and then uses the TableScan object to traverse the table. Operation, here we imitate the practice at that time to construct a student table first, and set this table to have only 3 fields, namely name, which is a string type, age, which is an int type, and finally id, which is also a numeric type, and then we Add a few rows of data to this table and add the following code in main.go:
func main() {
//构造 student 表
file_manager, _ := fm.NewFileManager("recordtest", 400)
log_manager, _ := lm.NewLogManager(file_manager, "logfile.log")
buffer_manager := bmg.NewBufferManager(file_manager, log_manager, 3)
tx := tx.NewTransation(file_manager, log_manager, buffer_manager)
sch := NewSchema()
//name 字段有 16 个字符长
sch.AddStringField("name", 16)
//age,id 字段为 int 类型
sch.AddIntField("age")
sch.AddIntField("id")
layout := NewLayoutWithSchema(sch)
//构造student 表
ts := query.NewTableScan(tx, "student", layout)
//插入 3 条数据
ts.BeforeFirst()
//第一条记录("jim", 16, 123)
ts.Insert()
ts.SetString("name", "jim")
ts.SetInt("age", 16)
ts.SetInt("id", 123)
//第二条记录 ("tom", 18, 567)
ts.Insert()
ts.SetString("name", "tom")
ts.SetInt("age", 18)
ts.SetInt("id", 567)
//第三条数据 hanmeimei, 19, 890
ts.Insert()
ts.SetString("name", "HanMeiMei")
ts.SetInt("age", 19)
ts.SetInt("id", 890)
//构造查询 student 表的 sql 语句,这条语句包含了 select, project 两种操作
sql := "select name, age from student where id=890"
sqlParser := parser.NewSQLParser(sql)
queryData := sqlParser.Query()
//根据 queryData 分别构造 TableScan, SelectScan, ProjectScan 并执行 sql 语句
//创建查询树最底部的数据表节点
tableScan := query.NewTableScan(tx, "student", layout)
//构造上面的 SelectScan 节点
selectScan := query.NewSelectionScan(tableScan, queryData.Pred())
//构造顶部 ProjectScan 节点
projectScan := query.NewProductionScan(selectScan, queryData.Fields())
//为遍历记录做初始化
projectScan.BeforeFirst()
for true {
//查找满足条件的记录
if projectScan.Next() == true {
fmt.Println("found record!")
for _, field := range queryData.Fields() {
fmt.Printf("field name: %s, its value is : %v:\n", projectScan.GetVal(field))
}
} else {
break
}
}
fmt.Println("complete sql execute")
tx.Close()
}
We need to do some analysis on the above code. When SelectScan and ProjectScan are initialized, it needs to pass in the Scan object. If you follow the first query tree, ProjectScan will be passed in to SelectScan as a parameter for construction. If you follow the second query tree, SelectScan It will be passed to ProjectScan as a parameter for construction, but in any case, TableScan corresponds to the leaf node in the query tree, so it will be passed as a parameter to SelectScan or ProjectScan for construction, so ProjectScan.Next will be called in the code SelectScan.Next, and finally calls TableScan.Next. TableScan.Next is responsible for fetching each record from the bottom file storage and returning it to SelectScan. The latter determines whether the fetched record can satisfy the constraints of Predicate. If so, it returns True, so ProjectScan.Next will get True, so the code finds the record that satisfies the where condition, and then ProjectScan takes out the given field in the record, which completes the execution of the SQL statement. For more details, please visit station b. Search "coding Disney" to view the debugging demonstration video and download the corresponding code:
Link: https://pan.baidu.com/s/1hIACldEXaABVkbZiJAevIg Extraction code: vb46
Let's look at how to calculate the efficiency of a query tree. In the operation of the database system, the most resource- and time-consuming operation is reading the hard disk. Compared with reading the memory, the speed of reading the hard disk is two to three orders of magnitude slower. That is, reading the hard disk is one order of magnitude slower than reading the memory. More than a hundred times. Therefore, when we judge the execution efficiency of a query tree, we must judge how many times it needs to access the hard disk to return given data or records. The fewer the number of accesses, the higher its efficiency.
We use s to represent the object that implements the Scan interface, so s can represent instances of TableScan, SelectScan, ProjectScan, ProductScan and other objects. Use B(s) to represent the number of blocks that a given instance object needs to access to return records that meet the conditions, R(s) to represent the number of records that need to be queried before a given instance object returns the required records, and V(s, F) to represent In the records returned by the Scan instance object s after traversing the database table, the F field contains the number of different values. V(s, F) is relatively abstract. Let's look at a specific example. Suppose the Student table contains three fields, namely name, age. , id, assuming we execute the statement "select * from Student where age <= 20", the three records returned after this statement is executed are as follows: {(
name: john),(age:18),(id=1)}, {(name:jim),(age:19),(id=2)},{(name:Lily),(age:20),(id=3)},{(name:mike),(age: 20), (id=4)}
That is, four records are returned after the statement is executed. If F corresponds to the field name, then V(s, F)=4, because the contents of the four records are different relative to the field name. Therefore, V(s,F)=4. If F corresponds to the field age, then V(s,F)=3, because the values of the last two corresponding fields of the four records are both 20, so the values of the field age in the four records are , the number of records with different values is 3.
B(s), R(s), V(s,F) play a very important role in the derivation process of calculating query book efficiency. What we also need to know is that when we create SelectScan, ProductScan, and ProjectScan, the initialization function will pass in an object that satisfies the Scan interface, such as projectScan in the signature code:= query.NewProductionScan(selectScan, queryData.Fields()) , so for the three formulas B(s), R(s), V(s,F), s corresponds to the variable projectScan. It should be noted that the constructor inputs a selectScan instance. At the same time, when we enter the code, we can find , when projectScan investigates the Next() interface, it actually switches to calling the Next interface of selectScan, so calculating B (projectScan) actually needs to rely on calculating B (selectScan).
Therefore, we use s1 to represent the parameters input to construct the s instance. Then to calculate B(s), we need to calculate B(s1). Next, let’s look at B(s), R(s), V(s,F) derivation.
In the first case, s is an instance of TableScan. Since we did not enter other Scan interface instances when constructing TableScan, the corresponding value of B(s) is the number of blocks that TableScan needs to access during the execution of Next, and V(s) is after the TableScan instance executes Next(). The number of records traversed, V(s,F), is the number of records with different values for field F among the records traversed after executing the Next() survey.
If s is an instance of selectScan, remember that the constructor of the instance also has a Predicate object. Assume that the corresponding form of this Predicate is A=c, where A represents a field in the record, c is a constant, and s1 is used to represent the construction of selectScan. The object is the Scan object passed in. We can see from the SelectScan code that the Next() interface will call the Next() interface of the input Scan object when executed, so when Next() of SelectScan returns, the incoming Scan How many blocks has the object visited, then the SelectScan object has visited the corresponding block, so we have B(s) = B(s1), where s corresponds to the SelectScan object, and s1 corresponds to the Scan object passed in by the constructor.
Let's look at the value of R(s). Let's look at the Scan implementation of SelectScan. It first calls the Next interface of the incoming Scan object to obtain a record, and then calls the IsSatisfied interface of Predicate to determine whether the obtained record meets the filtering conditions. If If it is satisfied, it will return immediately. If it is not satisfied, it will continue to call the Next interface of the Scan object to obtain the next record until the Next of Scan returns False.
For example, assume that there are 100 records in the Student table, and each record contains the age field. Assume that there are 4 values of age in these 100 records (the specific values are not relevant here, we only need to know the value There are as many value situations as there are), so V(s1, age) == 4. Of course we cannot know the number of records corresponding to each situation, so we assume that the number of records corresponding to each value is equal, So the number of records corresponding to each value is R(s1) / V(s1, age) = 25. If the value of the query constant c meets one of the four conditions, then the number of records returned by Next of SelectScan is R(s1) /V(s1,age), from this we conclude that R(s) satisfies the filter condition "A=c" where A corresponds to the field and c is a constant. In this case, the value of R(s) is R( s1)/V(s1, A)
This problem will become complicated if the filter condition is A=B, where A and B are both fields of the table. This situation requires some knowledge of probability theory. First, assume that the number of value situations of field B is greater than that of field A. We use F_B to represent the number of value categories of field B in the table, and use F_A to represent the number of value categories of field A in the table. For the convenience of analysis, we make further assumptions. Assume that the table has 100 records, among which there are 10 value categories for field B and 4 value categories for field A. We randomly take a record from the table and the value of field B is The probability of a certain category among the 10 categories is 1/10, and the probability that field A has the same value as field B is also 1/10, because if the value of field A wants to be equal to B, it must also be in the value category of B. within the range, and the probability that it happens to have the same value as the current field B is also 1/10. Therefore, the probability that we randomly select this record to satisfy A=B is 1/10 * 1/10 = 1/100. Since the table There are 10 possible value categories for field B in the table. Therefore, if a record is randomly selected, the probability that it satisfies A=B is 10 * 1 / 100 = 1 / 10. Therefore, when we traverse every record in the table, we can satisfy The expected number of records for A = B is 100 * (1 / 10) = 10 records.
In the above analysis, we assume that the value of field B is greater than A. If the value of A is greater than B, then the analysis process is the same, which is to interchange A and B in the above analysis process. So when the filter condition is A = B, where A and B are both fields in the table, then the number of records expected to be returned by R(s) is R(s1) / max{V(s1,A), V(s1,B) }, since this involves some knowledge points of probability theory, it is a little complicated to understand.
If the instance corresponding to s is ProjectScan, then we can see from the implementation of its Next interface that it only calls the Next interface of the input Scan object, so as many blocks as the latter accesses and how many records are returned, it also accesses how many blocks and records, so when s corresponds to the ProjectScan object, B(s)=B(s1), R(s)=R(s1), V(s,F)=V(s1,F).
The most complicated one is ProductScan. Let's look at its implementation code first. Its constructor passes in two Scan objects, which we use s1 and s2 to represent respectively. In its BforeFirst() function, it first calls the Next function of s1. In its Next function, if Next of s2 returns true, then the interface execution is completed. If it returns false, then call the beforeFirst() function of s2 to redirect the record pointer of s2 to the first item, and then execute s1.Next(), also It means that the record of s1 points to the next one, which means that every record from s1 must be combined with all records from s2.
When s1.Next() returns false, ProductScan's Next() returns false, so when ProjectScan's Next() function returns, no matter how many blocks s1 traverses, it also traverses the same blocks. At the same time, whenever Next of s2 returns false, the number of blocks returned by s2 at this time is R(s2). Then s2 calls the BeforeFirst() interface to redirect the record pointer to the first record, and then s1 calls the Next interface to point the record to the next record. , and then s2 traverses all the records again. The number of blocks it accesses in this process is R(s2), so during the call of ProductScan's Next() interface, it accesses R(s1) * B(s2) blocks, so the number of blocks it accesses is B(s1) + R(s1) * B(s2).
For R(s), we can see in its Next() interface that every time s2 traverses all records, s1 points to the next record through the Next() pointer, and then s2 traverses all its records again. , so R(s) = R(s1) * R(s2).
For V(s,F), if F corresponds to the field in the table where s1 is located, then V(s,F) = V(s1, F). If it corresponds to the field in the table where s2 is located, then V(s, F)= V(s2, F).
One important point here is that if we swap the positions of the two input Scan objects when constructing the ProductScan object instance, then the value of B(s) will be different. Therefore, when constructing the object instance, the incoming The different order of Scan object parameters has different effects on its execution efficiency.
We introduce RPB(s) = R(s) / B(s), this variable represents the average number of records corresponding to each block. So B(s) = B(s1) + R(s1) * B(s2) can be converted into B(s) = B(s1) + (RPB(s1) * B(s1) * B(s2)) , if we replace the order of s1, s2 when constructing ProductScan, then we have B(s) = B(s2) + (RPB(s2) * B(s2) * B(s1)), because B(s2 ) * B(s1) is much larger than B(s1) or B(s2), so when judging the growth of B(s), we can ignore B(s1) or B(s2) on the right side of the equation, and then Examining the right side of the plus sign on the right side of the equation, it is not difficult to see that if RPB(s1) < RPB(s2), then s1 is placed in front of s2 when constructing ProductScan, then the constructed instance is accessed when executing the Next() function The number of blocks will be correspondingly smaller, and vice versa.
Let's look at a specific example. Suppose there are two tables, Student and Department. The Student table has fields SId, SName, GradYear, and MajorId, which respectively represent the student's ID, name, graduation time, and major ID. The Department table contains fields DId and DName, which represent professional id and professional name respectively.
Among them, the B(s) corresponding to the Student table is 4500, that is, the storage of all its data occupies 4500 blocks, and its R(s) is 45000, that is, the table has 45000 records, V(s, SId)= 45000, that is, there are 45000 SId value situations, that is, the SId field value corresponding to each record is different, V (s, SName) = 44960, that is, the name field value of all students is different. There are 44960 entries, that is, there are (45000 - 44960)=40 students among the 45000 students. Their names have the same name as others, V(s, MajorId)=40, that is, these students belong to 40 different majors.
For the Department table, its corresponding B(s) is 2, that is, all its data occupies 2 blocks, R(s)=40, that is, it has a total of 40 records, V(s, DId) = V(s , DName) = 40, that is, the values of fields DId and DName in each record are different.
Suppose we want to find the names of all students from the Department of Mathematics. So first we need to extract the records whose DName field is equal to "Mathematics" in the Department table, then perform the Product operation on the obtained results with the Student table, and finally find the records with the field MajorId = DId on the operation results. On this basis, we will Execute the Project operation to extract the SName field. The corresponding sql statement is select Student.SName where Student.MajorId=DepartMent.DId and DepartMent.DName=“math”. This sql statement will construct the following query after parsing. Tree:
According to the previous code, we execute the code corresponding to the above search tree as follows:
file_manager, _ := fm.NewFileManager("studentdb", 1024)
log_manager, _ := lm.NewLogManager(file_manager, "logfile.log")
buffer_manager := bmg.NewBufferManager(file_manager, log_manager, 3)
tx := tx.NewTransation(file_manager, log_manager, buffer_manager)
matedata_manager := mgm.NewMetaDataManager(true, tx)
slayout := metadata_manager.GetLayout("student", tx)
dlayout := metadata_manager.GetLayout("department",tx)
s1 := query.NewTableScan(tx, "student", slayout)
s2 := query.NewTableScan(tx, "department", dlayout)
pred1 := queryData.Pred() //对应dname = 'math'
s3 := query.NewSelectScan(s2, pred1)
s4 := query.NewProductScan(s1, s3)
pred2 := queryData.Pred() //对应 majorid=did
s5 := query.NewSelectScan(s4, pred2)
fields := queryData.Fields() //获得"sname"
s6 := query.NewProjectScan(s5, fields)
What needs to be noted in the above code is s4, which corresponds to ProductScan. The first Scan object entered is s1, which corresponds to the TableScan object of the student table. Its Next interface will traverse all records, which is 45,000 records. The first Scan object is s3, which is the result of s2 filtered by pred1. Since the content of the dname field in each record in the Department table is different, s3 only contains 1 record.
According to the ProudctScan code implementation, when calling its Next interface, it will call the Next interface of the first Scan object once, here this object corresponds to s1, and then call the Next interface of the second Scan object to traverse all its records, here it Corresponding to the input s3, since s3 is a SelectScan instance, when it returns a given record, the number of blocks it accesses is equal to the Scan object passed in when constructing it. The Scan object constructed in s3 in the code is the TableScan corresponding to the Department table. Since The table has 2 blocks, so to return a given record, s3 needs the incoming TableScan object to access all blocks, so s3 needs to access up to 2 blocks after executing the Next interface, so s4 executes the Next interface once, The s1 object entered during construction will be called, which is the Next interface of TableScan corresponding to the Student table. Since the table has 45,000 records, Next of s1 can be executed 45,000 times. Each time Next is executed, Next of s2 is called once. , so the two blocks of the Department table will be traversed, so a Next execution of s4 must access at least 2 * 45000 blocks.
But if we change the code that constructs s4 to: s4 = query.NewProductScan(s3, s1), then s4 executes the Next interface once, and the Next() of s3 will be executed. Since s3 has only 1 record, its Next is The code is only executed once, and then the Next interface of s1 will be executed 45,000 times, so all blocks of the database table corresponding to s1 will be traversed once. Since the Student table has 4500 blocks, the Next interface of s4 only accesses 4500 blocks. , compared with the previous analysis, we only exchange the positions of the two input parameters, the blocks accessed will be greatly reduced, and the speed will be greatly improved.
In the next section we will look at how to implement the theory in this section into code. For more information, please search Coding Disney on station b.