1. Problem background and applicable scenarios
In the previous article, we introduced that the relational database uses HASH segmentation technology when performing associations between tables. Assuming that the scales (number of records) of the two association tables are N and M respectively, the computational complexity of the HASH segmentation technology (the number of comparisons of the associated fields) is approximately SUM(Ni*Mi), where Ni and Mi are the HASH values respectively For the number of records in the two tables of i, satisfying N=SUM(Ni) and M=SUM(Mi), this probability will be much smaller than the complexity of the full traversal N*M (it will be K times smaller with good luck) , K is the range of HASH value).
If the two tables are in order for the association key, then we can use the merge algorithm to deal with the association, the complexity at this time is N+M; when N and M are both large (generally much larger than K), This number will be much smaller than the SUM(Ni*Mi) just now. Therefore, the calculation speed of orderly merge association is much faster than HASH segment association.
In practical applications, because the same-dimension tables and primary and sub-tables are always related to the primary key or part of the primary key, we can sort the data of these related tables according to their primary key in advance, and then we can always use the merge algorithm to achieve the association, which can improve efficiency a lot of. SPL uses such an algorithm.
Let's use esProc SPL to compare with relational database Oracle to test the efficiency of orderly merge association.
2. Test environment
1. Small data full memory test
The test machine has two Intel2670 CPUs, with a main frequency of 2.6G, a total of 16 cores, a memory of 128G, and an SSD solid state drive.
Using the 50G data generated by the TPCH standard, the main table is orders and the sub-table is orderdetail (generated by reducing the data records of the lineitem table). The records in the two tables are arranged in ascending order of O_ORDERKEY and L_ORDERKEY respectively.
Both Oracle and SPL use single-threaded testing.
2. Big data external storage test
The virtual machine in the aforementioned test machine is used, with 16G memory and SSD solid state drive.
Using 200G data generated by the TPCH standard, the main table is orders and the sub table is lineitem. The records in the two tables are arranged in ascending order of O_ORDERKEY and L_ORDERKEY respectively.
Due to the relatively large amount of data, Oracle and SPL both use 8-thread parallel testing.
Three, small data full memory test
1. Oracle test
(1 ) No correlation test
The SQL statement tested is as follows:
select
l_year,
sum (volume) as income,
sum(l_quantity) as quantity
from
(
select
extract(year from l_shipdate) as l_year,
(l_extendedprice * (1 - l_discount) ) as volume,
l_quantity
from
orderdetail
)
group by
l_year
union all
select
2019 as l_year,
count (o_orderkey) as income,
count(o_totalprice) as quantity
from
orders;
(2 ) Related tests
The SQL statement tested is as follows:
select
l_year,
sum (volume) as income,
sum(l_quantity) as quantity
from
(
select
extract(year from l_shipdate) as l_year,
(l_extendedprice * (1 - l_discount) ) as volume,
l_quantity
from
orders,
orderdetail
where
o_orderkey = l_orderkey
and l_quantity>0
)
group by
l_year;
The query condition l_quantity>0 is always true, and there is no filtering data to ensure that this column of data will be read.
2. SPL test
(1 ) No correlation test
Write the SPL script as follows:
 |
A |
| 1 | >orders=file("/home/ctx/orders.ctx").create().memory() |
| 2 | >orderdetail=file("/home/ctx/orderdetail.ctx").create().memory() |
| 3 | =now() |
| 4 | =orderdetail.cursor(L_ORDERKEY,L_EXTENDEDPRICE,L_DISCOUNT,L_SHIPDATE,L_QUANTITY).groups(year(L_SHIPDATE):l_year; sum(L_EXTENDEDPRICE * (1 - L_DISCOUNT)):revenue,sum(L_QUANTITY):quantity) |
| 5 | =orders.groups(;count(O_ORDERKEY),count(O_TOTALPRICE)) |
| 6 | =interval@s(A3,now()) |
(2 ) Related tests
Write the SPL script as follows:
|
A |
| 1 | >orders=file("/home/ctx/orders.ctx").create().memory() |
| 2 | >orderdetail=file("/home/ctx/orderdetail.ctx").create().memory() |
| 3 | =now() |
| 4 | = orders.cursor (O_ORDERKEY, O_TOTALPRICE; O_TOTALPRICE> 0) |
| 5 | =orderdetail.cursor(L_ORDERKEY,L_EXTENDEDPRICE,L_DISCOUNT,L_SHIPDATE,L_QUANTITY) |
| 6 | =joinx(A5:detail,L_ORDERKEY;A4:orders,O_ORDERKEY) |
| 7 | =A6.groups(year(detail.L_SHIPDATE):l_year; sum(detail.L_EXTENDEDPRICE * (1 - detail.L_DISCOUNT)):revenue, sum(detail.L_QUANTITY):quantity) |
| 8 | =interval@s(A3,now()) |
The joinx in A6 is an orderly merge association function, requiring the associated fields to be arranged in ascending order.
3. Test results and analysis
The test result list is as follows (unit: second):
| category | Unrelated | Related | Slowdown multiple | Association time |
| Oracle | 16 | 67 | 4.2 | 51 |
| SPL | 14 | 32 | 2.3 | 18 |
After each test result is run multiple times and the data is fully cached, the fastest one is taken.
Analyze two sentences of SQL. In the unrelated test, read the O_ORDERKEY and O_TOTALPRICE columns of the orders table and count the number of records, and read the five columns of L_ORDERKEY, L_EXTENDEDPRICE, L_DISCOUNT, L_SHIPDATE, and L_QUANTITY from the orderdetail table, and sum the sales price and L_QUANTITY Sum up. In the correlation test, the read volume of the orders and orderdetail tables is the same, and the sales price after the correlation is summed and L_QUANTITY is summed. The amount of reading and calculation in the two cases is basically the same. The extra operation is the association between the two tables, so the difference in running time between the two is the time for the association operation. The same is true for SPL.
Under the same hardware equipment and data scale, SPL association takes 18 seconds, Oracle association takes 51 seconds, almost three times the former, and SPL is a program written in Java, while Oracle is implemented in C++, which fully verifies the orderliness Merging correlation can greatly improve the efficiency of correlation. The speed of SPL with correlation is 2.3 times slower than when there is no correlation, and Oracle is 4.2 times slower, indicating that the speed of ordered merge calculation is equivalent to that of ordinary calculation, while HASH correlation is much slower than ordinary calculation.
4. Big data external storage test
When the two tables to be JOIN are too large to fit in the memory, the relational database still uses the HASH segmentation technology. According to the HASH value of the associated field, the data is divided into several segments, each of which is small enough to be loaded into the memory and then implement the HASH segmentation algorithm of the memory. But this will cause the problem of external memory switching. The data needs to be written out in segments and then read in. The extra one is written and the other is read. The external memory read is inherently not fast, and the write is slower, so the performance will be much worse.
Orderly merge association does not have this problem. The data of the two tables only needs to be traversed once. Not only is the amount of CPU calculations reduced, but the amount of external memory IO also drops significantly. Moreover, the execution of the merge algorithm requires very little memory, as long as a few cache records are kept for each table in the memory, it will hardly affect the memory requirements of other concurrent tasks.
1. Oracle test
(1 ) No correlation test
The SQL statement tested is the same as the small data test, just change the orderdetail table to the lineitem table, and add "/*+ parallel(8) */" after the first select to use 8 threads in parallel.
(2 ) Related tests
The SQL statement tested is the same as the small data test, just change the orderdetail table to the lineitem table, and add "/*+ parallel(8) */" after the first select to use 8 threads in parallel.
2. SPL test
(1 ) No correlation test
Write the SPL script as follows:
|
A |
| 1 | =now() |
| 2 | =file("/home/ctx/lineitem.ctx").create().cursor@m(L_ORDERKEY,L_EXTENDEDPRICE,L_DISCOUNT,L_SHIPDATE,L_QUANTITY;;8) |
| 3 | =A2.groups(year(L_SHIPDATE):l_year; sum(L_EXTENDEDPRICE * (1 - L_DISCOUNT)):revenue,sum(L_QUANTITY):quantity) |
| 4 | =file("/home/ctx/orders.ctx").create().cursor@m(O_ORDERKEY,O_TOTALPRICE;;8) |
| 5 | =A4.total(count(O_ORDERKEY),count(O_TOTALPRICE)) |
| 6 | =interval@s(A1,now()) |
The amount of data is large. Both A2 and A4 use 8-channel parallel cursor readings.
(2 ) Related tests
Write the SPL script as follows:
|
A |
| 1 | =now() |
| 2 | =file("/home/ctx/orders.ctx").create().cursor@m(O_ORDERKEY,O_TOTALPRICE;O_TOTALPRICE>0;8) |
| 3 | =file("/home/ctx/lineitem.ctx").create().cursor(L_ORDERKEY,L_EXTENDEDPRICE,L_DISCOUNT,L_SHIPDATE,L_QUANTITY;;A2) |
| 4 | =joinx(A3:detail,L_ORDERKEY;A2:orders,O_ORDERKEY) |
| 5 | =A4.groups(year(detail.L_SHIPDATE):l_year; sum(detail.L_EXTENDEDPRICE * (1 - detail.L_DISCOUNT)):revenue,sum(detail.L_QUANTITY):quantity) |
| 6 | =interval@s(A1,now()) |
A2 is used as a parameter when the cursor is generated in A3, which means that it is read in segments synchronously with the primary key of the main table orders.
The joinx in A4 is an orderly merge association function, requiring the associated fields to be arranged in ascending order.
3. Test results and analysis
The test result list is as follows (unit: second):
| category | Unrelated | Related | Slowdown multiple | Association time |
| Oracle | 265 | 863 | 3.3 | 598 |
| SPL | 70 | 101 | 1.4 | 31 |
Calculation analysis and correlation time calculation are the same as small data test.
Under the same hardware equipment and data scale, SPL association takes 31 seconds and Oracle association takes 598 seconds, which is 19 times the former, further verifying that orderly merge association can greatly improve association efficiency. Compared with the small data test, the multiple is greatly increased, indicating that the larger the amount of data, the more obvious the performance improvement advantage of orderly merge correlation over HASH correlation.
The slower multiples of correlation than no correlation can also draw the same conclusion as in the small data test, but there is more time to read the data than in the small data test (the data is all cached in the memory when the small data is small), so no The time base of the correlation test has become larger, and the slowness factor seems to be reduced compared to the small data test.