https://wangbinbin0326.github.io/2017/03/31/oracle%E6%95%B0%E6%8D%AE%E8%BF%81%E7%A7%BB%E4%B9%8BExp%E5%92%8CExpdp%E5%AF%BC%E5%87%BA%E6%95%B0%E6%8D%AE%E7%9A%84%E6%80%A7%E8%83%BD%E5%AF%B9%E6%AF%94%E4%B8%8E%E4%BC%98%E5%8C%96/
Performance comparison and optimization of exported data using Exp and Expdp
foreword
Data backup is crucial to the safe operation of information systems. More and more of our users use RMan or third-party professional backup software, but many users still keep traditional Exp as part of their backup strategy, mainly because of this. This backup method is simple and easy to use, and it is also very convenient to restore to other machines. Therefore, although there are other backup methods, the Exp method will still be used at the same time, and there are even many users who only have this backup method (maybe because they feel that mastering other techniques is too complicated).
With the increase in the amount of user data, the problems existing in the Exp export method are becoming more and more prominent. The main problem is that it takes a long time, and some even exceed 3 hours. In addition, common background automatic operations: summary table calculation, automatic cost calculation, statistics The collection of information and other work make the schedule of a night more and more tense.
Is there a way to improve the performance of Exp execution? In the past, I have done some experiments, consulted a lot of information, and summed up some experience. Recently, I got nearly 300G data of a user, and conducted a series of export comparison experiments with large data volume again. After optimization, the performance of Exp export has been greatly improved, but compared with Expdp, the latter is still much faster. The following is an introduction to the test situation and related knowledge, so that we can help Users can refer to this when formulating backup strategies.
test environment
硬件：
processor: 8
model name : Intel(R) Xeon(R) CPU E7- 2870 @ 2.40GHz
cpu MHz : 2396.863
MemTotal: 16333448 kB
Disk：SCSISoftware : Linux+Oracle 11.2.0.0
Data : The original total is 300G. After shrinking the rollback table space, temporary table space, and unused space of some tablespace files, it occupies a total of 267G, which contains a large amount of LOB data related to electronic medical records.
experiment method
Time consuming to experiment with 4 different ways of exporting data in the same environment
| Way | time consuming | illustrate |
|---|---|---|
| Exp regular path, not optimized | 5 hours 15 minutes | 5 |
| Exp direct path export without parameter optimization | 2 hours 38 minutes | direct=y |
| Exp direct path, parameter optimization | 1 hour 40 minutes | direct=y recordlength=65535 buffer=104857600 |
| Export with Expdp, parameter optimization | 59 points | parallel=3 |
The results show that
the fastest is the Expdp method, and the Exp method is nearly 3 times faster than the case without any parameters after parameter optimization.
Principle description
Exp defaults to the traditional path. In this mode, Select is used to query data, then write to the buffer cache, write the data to the evaluate buffer, and finally transfer it to the Export client, and then write the dump file.
In the direct path mode, the data is directly read from the hard disk, and then written to the PGA. The format is the Export format, and no conversion is required. The data is directly transferred to the Export client and written to the dump file. This mode does not go through the evaluation buffer, there is one less process, and the export speed is obviously improved.
Exp does not have parallel parameters. To perform parallel export, you can write multiple commands to export at the same time. This method can be used in specific data migration situations to perform fast data migration by table or table space.
Expdp is an advanced data export method launched on Oracle 10G, which has a greater performance improvement than Exp. Expdp can be regarded as an upgraded version of Exp, which is equivalent to exp + direct mode + parallel.
Note : In general, Exp can be replaced, but it cannot be completely replaced, mainly because it needs to be executed on the database server, and Exp can be executed on any client. In addition, according to the test, when Expdp exports a large partition table (above 1T), the analysis time is more than 2 hours, and there are some bugs. Therefore, some users still use Exp for data backup.
Parameter optimization
Exp related parameters
Through the above analysis, we know that the use of "direct path" can improve the export speed. This mode focuses on 2 parameters: DIRECT and RECORDLENGTH parameters.
DIRECTThe parameter defines whether the export uses direct path mode (DIRECT=Y) or regular path mode (DIRECT=N). Conventional path export uses the SELECT statement to extract data from the table, and then writes it after evaluation, while direct path export directly reads the data from the disk to the PGA and writes it to the export file as it is, thus avoiding the data conversion process of the SQL command processing layer. , which greatly improves the export efficiency.
BUFFERThe parameter is used to set the size of the cache for reading records, in bytes, that is, the maximum number of records in the array. This parameter is only valid for regular path mode exports.
RECORDLENGTHThe parameter is a parameter used in conjunction with DIRECT=Y, which defines the size of the Export I/O buffer, which is similar to the BUFFER parameter used for conventional path export. It is recommended to set the RECORDLENGTH parameter to the maximum I/O buffer, which is 65535 (64kb).It should be emphasized that even if the direct path export mode is used, the tables involving LOB objects will only be exported through the traditional mode. Therefore, when DIRECT=Y, in addition to setting the RECORDLENGTH parameter, you also need to set the BUFFER parameter. Generally, it can be set to 104857600 ( 100M), which is ignored by many optimization articles on the Internet.
In addition, you can also modify the Oracle initialization parameter Multiple Block Read to submit the performance of reading data. After testing, the effect is not obvious. In the previous test environment, the time-consuming time can be reduced by about 10 minutes.
|
1
|
exp userid=sys/his@zyyy full=y direct=y recordlength=65535 buffer=104857600 file=F:\zyyy\exp20120218.dmp log=F:\zyyy\exp20120218.log feedback=10000
|
Expdp related parameters
The Parallel parameter indicates the degree of export parallelism, which is set according to the number of CPUs. The default is 1. If it is not set, the export performance is not much different from the direct path export mode of Exp, and its advantages are not exerted.
The Dumpfile parameter can be used with the Parallel parameter to specify multiple export files to reduce IO contention during concurrent writing.
|
1
|
expdp sys/his@zyyy full=y directory=dump_dir parallel=3 dumpfile=expdp_0225_1.dmp,expdp_0225_2.dmp,expdp_0225_3.dmp logfile=expdp_0225.log
|
Summary:
If you are still accustomed to using the Exp method, if you want to reduce the time-consuming, it is best to use the direct path method, and set the values of the RECORDLENGTH and BUFFER parameters, which can greatly improve the export performance. If you master the Expdp method, use the appropriate Parallel parameters to achieve fast export, and more importantly, the performance improvement of Impdp is the real advantage of the data pump mode. From the test results of some Oracle enthusiasts, it is indeed very different. .
Exp and Expdp export data
| Way | time consuming | illustrate |
|---|---|---|
| IMP regular path, not optimized | 3 hours 17 minutes | |
| Import speed of IMPdp | 3 hours 8 minutes | |
| Export with IMPdp, parameter optimization | 2 hours 8 minutes | parallel=3 |
The import operation of the data pump actually took 3 hours and 8 minutes, which is very close to the import speed of IMP. It seems that not all cases are as described by Oracle, and the import efficiency of data pump is greatly improved than that of ordinary import. .
In the test of the above article, it was found that the import speed of IMPDP is almost the same as that of IMP. When Oracle introduced the data pump, it mentioned that the import speed of IMPDP is up to 10 times that of IMP. Fortunately, IMPDP can still be optimized and adjusted, that is, by setting PARALLEL to improve the parallelism of IMPDP.