Oracle Data Pump uses rowid sharding to export and import lob large tables and its best practices

Author | JiekeXu

Source | Public account JiekeXu DBA road (ID: JiekeXu_IT)

If you need to reprint, please contact us for authorization | (Personal WeChat ID: JiekeXu_DBA)

Hello everyone, I am JiekeXu. I am very happy to meet you again. Today I will take a look at Oracle Data Pump's use of rowid sharding to export and import lob large tables and its best practices. Welcome to click on the blue text above "JiekeXu DBA Road" Follow my official account, mark it with a star or pin it to the top, and more useful information will arrive as soon as possible!

Why do you put an author source, authorization statement and public account at the beginning? This is because there has been a lot of noise about plagiarism in the past two days. I believe every blogger has encountered the situation of articles being crawled. When your works and articles are labeled as original and sold, you will be really disgusted. When I was writing articles in the past, I found that some bloggers read your articles and then experimented and posted them. Although it was a bit shameless, it was barely acceptable. In the past, there were training institutions that used my translated 051, 052, and 053 OCP materials as teaching materials for students, but I didn’t care about it. Later, various websites directly crawled my articles, and I saw a lot of them, but I couldn’t access them. After finding the person, I was too lazy to bother with it, but thinking about it made me sick. Last night I also found someone named "XX Operation and Maintenance" Website, I directly crawled Mo Tianlun and posted the articles of many bloggers verbatim. I found that many bloggers’ articles were included in it. Although the number of reads was zero, it still felt disgusting. Moreover, this kind of crawler website There are not a few, so the awareness of intellectual copyright in China is still very weak. The big guys have worked hard to publish books, write articles, and patrol scripts, and they will deal with many pirated ones in a few days. Look, there is nothing like this abroad. thing. I have no choice but to add a statement at the beginning and end.

Table of Contents
    Preface
    1. Large table information
    2. Export large table normally
    3. Use sharding to export a large lob table
    4. Import dmp into the target database respectively
    5. Other uses of data pump< /span>        Data Pump Best Practices        19c Data Pump New Features        Data pump component    7. Best practices for using data pump    6. Commonly used SQL statements in import and export        Use the attach parameter to check the import progress        Export some data
        Commonly used import and export statements in data pump

Preface

Sometimes, in order to shorten the import and export time of Data Pump, we usually choose to export the large table separately from other tables. Especially if the large table has lob fields, the export time will be slower. In this way, you need to export the large table and other tables separately, and exporting the large table alone will be very slow. You can use rowid sharding technology to separate the large table and export it at the same time to improve export efficiency. In addition, the commonly used export and import commands and best practices of Data Pump are also worth learning.

1. Large table information

There are only 1525536 table records, but the size is only 1.8G. There are two CLOB fields. The Lob size is 13.71G.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.15.0.0.0


SQL> select count(*) from PROD.T_CA_TX_HIS;


  COUNT(*)
----------
   1525536


SQL> set line 120
SQL> col OWNER for a30
SQL> col SEGMENT_NAME for a30
SQL> select OWNER,SEGMENT_NAME,BYTES/1024/1024/1024 from dba_segments where SEGMENT_NAME='T_CA_TX_HIS' and OWNER='PROD';


OWNER                          SEGMENT_NAME                   BYTES/1024/1024/1024
------------------------------ ------------------------------ --------------------
PROD                            T_CA_TX_HIS                    1.83007813


SQL> desc PROD.T_CA_TX_HIS
 Name                                                              Null?    Type
 ----------------------------------------------------------------- -------- --------------------------------------------
 SEQUENCE_NO                                                       NOT NULL NUMBER(18)
 PK_TX_HIS                                                         NOT NULL VARCHAR2(36)
 FK_USER_CCBSCF                                                             VARCHAR2(36)
 FK_PERSON_CCBSCF                                                           VARCHAR2(36)
 SOFT_CA_BRAND                                                     NOT NULL VARCHAR2(20)
 SOFT_CA_USER_ID                                                            VARCHAR2(50)
 TARGET_URL                                                                 VARCHAR2(300)
 TX_CODE                                                                    VARCHAR2(20)
 TX_REQUEST                                                                 CLOB
 TX_RESPONSE                                                                CLOB
 TX_ERROR_CODE                                                              VARCHAR2(50)
 TX_ERROR_MESSAGE                                                           VARCHAR2(500)
 CREATE_TIME                                                       NOT NULL DATE
 FK_USER_CREATE                                                    NOT NULL VARCHAR2(36)
 USER_NAME_CREATE                                                  NOT NULL VARCHAR2(60)
 
SQL> col SEGMENT_NAME for a30
SQL> select segment_name,BYTES/1024/1024/1024 from dba_segments where OWNER='PROD' group by segment_name,BYTES/1024/1024/1024 order by 2 asc;
.........省略........
SEGMENT_NAME                   BYTES/1024/1024/1024
------------------------------ --------------------
T_OPEN_MESSAGE_SMS_RECORD                    2.8125
SYS_LOB0000167075C00010$$                13.7119141


--查看某用户下 Lob 字段大小
SET LINE 345 PAGES 456 
COL OWNER FOR a20   
COL TABLE_NAME FOR A40 
COL SEGMENT_TYPE FOR A20  
col COLUMN_NAME FOR A35   
SELECT 
A.OWNER, 
B.TABLE_NAME, 
B.COLUMN_NAME,  
a.SEGMENT_TYPE,  
ROUND(SUM(A.BYTES/1024/1024/1024),2) G  
FROM DBA_SEGMENTS A  
LEFT JOIN DBA_LOBS B 
ON A.OWNER = B.OWNER 
AND A.SEGMENT_NAME = B.SEGMENT_NAME 
WHERE A.SEGMENT_TYPE='LOBSEGMENT' 
AND A.OWNER in('&OWNER')   
GROUP BY A.OWNER,B.TABLE_NAME,B.COLUMN_NAME,a.SEGMENT_TYPE 
ORDER BY 5 DESC; 


OWNER                TABLE_NAME                               COLUMN_NAME                         SEGMENT_TYPE                  G
-------------------- ---------------------------------------- ----------------------------------- -------------------- ----------
PROD                 T_CA_TX_HIS                              TX_RESPONSE                          LOBSEGMENT                13.71

2、正常导出大表

SQL> set linesize 9999
SQL> col OWNER for a10
SQL> col DIRECTORY_NAME for a30
SQL> col DIRECTORY_PATH for  a60
SQL> select * from dba_directories;


SQL> create or replace directory PUBLIC_DUMP as '/data/ora-share';
SQL> grant read,write on directory PUBLIC_DUMP  to public;


$ expdp PROD/proD_#31 directory=PUBLIC_DUMP LOGFILE=PROD.T_CA_TX_HIS.log dumpfile=PROD.T_CA_TX_HIS.dmp tables=T_CA_TX_HIS COMPRESSION=ALL CLUSTER=N


Export: Release 19.0.0.0.0 - Production on Wed Nov 29 15:00:36 2023
Version 19.15.0.0.0


Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.


Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
FLASHBACK automatically enabled to preserve database integrity.
Starting "PROD"."SYS_EXPORT_TABLE_01":  PROD/******** directory=PUBLIC_DUMP LOGFILE=PROD.T_CA_TX_HIS.log dumpfile=PROD.T_CA_TX_HIS.dmp tables=T_CA_TX_HIS COMPRESSION=ALL CLUSTER=N 
Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
Processing object type TABLE_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
Processing object type TABLE_EXPORT/TABLE/STATISTICS/MARKER
Processing object type TABLE_EXPORT/TABLE/PROCACT_INSTANCE
Processing object type TABLE_EXPORT/TABLE/TABLE
Processing object type TABLE_EXPORT/TABLE/GRANT/OWNER_GRANT/OBJECT_GRANT
Processing object type TABLE_EXPORT/TABLE/COMMENT
Processing object type TABLE_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
. . exported "PROD"."T_CA_TX_HIS"        3.710 GB 1524771 rows
Master table "PROD"."SYS_EXPORT_TABLE_01" successfully loaded/unloaded
******************************************************************************
Dump file set for PROD.SYS_EXPORT_TABLE_01 is:
  /data/ora-share/PROD.T_CA_TX_HIS.dmp
Job "PROD"."SYS_EXPORT_TABLE_01" successfully completed at Wed Nov 29 16:16:40 2023 elapsed 0 01:16:03


--排除大表导出其他表和对象 T_AUTH_ORIGINAL
nohup expdp \'/ as sysdba\'  directory=PUBLIC_DUMP dumpfile=expdp_prod_cc-2023-11-25_%U.dmp COMPRESSION=ALL  exclude=TABLE:\"IN \'T_CA_TX_HIS\'\" exclude=statistics parallel=4 cluster=no schemas=PROD_CC logfile=expdp_PROD_CC112521.log &

3、利用分片导出 lob 大表

--可以利用 rowid 切片方式导出 lob 大表


vi tableid.par


userid='/ as sysdba'
directory=PUBLIC_DUMP
content=ALL
compression=ALL
cluster=no
tables=PROD.T_CA_TX_HIS


nohup expdp parfile=tableid.par dumpfile=T_CA_TX_HIS_01.dmp logfile=tableid_01.log query=\"where mod\(dbms_rowid.rowid_block_number\(rowid\),8\)=0\" &
nohup expdp parfile=tableid.par dumpfile=T_CA_TX_HIS_02.dmp logfile=tableid_02.log query=\"where mod\(dbms_rowid.rowid_block_number\(rowid\),8\)=1\" &
nohup expdp parfile=tableid.par dumpfile=T_CA_TX_HIS_03.dmp logfile=tableid_03.log query=\"where mod\(dbms_rowid.rowid_block_number\(rowid\),8\)=2\" &
nohup expdp parfile=tableid.par dumpfile=T_CA_TX_HIS_04.dmp logfile=tableid_04.log query=\"where mod\(dbms_rowid.rowid_block_number\(rowid\),8\)=3\" &
nohup expdp parfile=tableid.par dumpfile=T_CA_TX_HIS_05.dmp logfile=tableid_05.log query=\"where mod\(dbms_rowid.rowid_block_number\(rowid\),8\)=4\" &
nohup expdp parfile=tableid.par dumpfile=T_CA_TX_HIS_06.dmp logfile=tableid_06.log query=\"where mod\(dbms_rowid.rowid_block_number\(rowid\),8\)=5\" &
nohup expdp parfile=tableid.par dumpfile=T_CA_TX_HIS_07.dmp logfile=tableid_07.log query=\"where mod\(dbms_rowid.rowid_block_number\(rowid\),8\)=6\" &
nohup expdp parfile=tableid.par dumpfile=T_CA_TX_HIS_08.dmp logfile=tableid_08.log query=\"where mod\(dbms_rowid.rowid_block_number\(rowid\),8\)=7\" &

Looking at the log, it takes about 22 minutes for each tableid. If rowid sharding is not used, it will take more than an hour to export 01:16:03.

more tableid_08.log
;;; 
Export: Release 19.0.0.0.0 - Production on Wed Nov 29 16:53:42 2023
Version 19.15.0.0.0


Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.
;;; 
Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
;;; **************************************************************************
;;; Parfile values:
;;;  parfile:  tables=PROD.T_CA_TX_HIS
;;;  parfile:  compression=ALL
;;;  parfile:  content=ALL
;;;  parfile:  directory=PUBLIC_DUMP
;;;  parfile:  userid=/******** AS SYSDBA
;;; **************************************************************************
FLASHBACK automatically enabled to preserve database integrity.
Starting "SYS"."SYS_EXPORT_TABLE_08":  /******** AS SYSDBA parfile=tableid.par dumpfile=T_CA_TX_HIS_08.dmp logfile=tableid_08.log query="where mod(dbms_rowid.rowid_block_number(rowid),8)=7" 
Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
Processing object type TABLE_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
Processing object type TABLE_EXPORT/TABLE/STATISTICS/MARKER
Processing object type TABLE_EXPORT/TABLE/PROCACT_INSTANCE
Processing object type TABLE_EXPORT/TABLE/TABLE
Processing object type TABLE_EXPORT/TABLE/GRANT/OWNER_GRANT/OBJECT_GRANT
Processing object type TABLE_EXPORT/TABLE/COMMENT
Processing object type TABLE_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
. . exported "PROD"."T_CA_TX_HIS"        471.9 MB  190827 rows
Master table "SYS"."SYS_EXPORT_TABLE_08" successfully loaded/unloaded
******************************************************************************
Dump file set for SYS.SYS_EXPORT_TABLE_08 is:
  /data/ora-share/T_CA_TX_HIS_08.dmp
Job "SYS"."SYS_EXPORT_TABLE_08" successfully completed at Wed Nov 29 17:15:40 2023 elapsed 0 00:21:50


Job "SYS"."SYS_EXPORT_TABLE_05" successfully completed at Wed Nov 29 17:15:40 2023 elapsed 0 00:21:54
Job "SYS"."SYS_EXPORT_TABLE_01" successfully completed at Wed Nov 29 17:16:26 2023 elapsed 0 00:22:42
Job "SYS"."SYS_EXPORT_TABLE_07" successfully completed at Wed Nov 29 17:17:50 2023 elapsed 0 00:24:01
Job "SYS"."SYS_EXPORT_TABLE_02" successfully completed at Wed Nov 29 17:17:27 2023 elapsed 0 00:23:42
Job "SYS"."SYS_EXPORT_TABLE_04" successfully completed at Wed Nov 29 17:16:45 2023 elapsed 0 00:22:59
Job "SYS"."SYS_EXPORT_TABLE_03" successfully completed at Wed Nov 29 17:16:24 2023 elapsed 0 00:22:38
Job "SYS"."SYS_EXPORT_TABLE_06" successfully completed at Wed Nov 29 17:16:03 2023 elapsed 0 00:22:15

4、目标库分别导入 dmp

When importing, serial import is performed one by one. Each dmp will lock the table. After the first dmp is imported, the table lock is released, and then the second dmp is imported. In fact, it is not particularly fast.

--创建好表空间和用户
CREATE TABLESPACE DT_CC_DATA DATAFILE '+DATA' SIZE 20g AUTOEXTEND ON NEXT 8192 MAXSIZE 32767M;


--查看原用户创建语句及权限
set long 9999
select dbms_metadata.get_ddl('USER',username) from dba_users where username='PROD';


--执行导入命令：
nohup impdp \'/ as sysdba\' directory=public_dump dumpfile=T_CA_TX_HIS_01.dmp logfile=T_CA_TX_HIS_01.log cluster=no REMAP_TABLESPACE=PROD_TBS:CC_OP_DATA REMAP_SCHEMA=PROD:CC_OP TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y TABLE_EXISTS_ACTION=append &
nohup impdp \'/ as sysdba\' directory=public_dump dumpfile=T_CA_TX_HIS_02.dmp logfile=T_CA_TX_HIS_02.log cluster=no REMAP_TABLESPACE=PROD_TBS:CC_OP_DATA REMAP_SCHEMA=PROD:CC_OP TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y TABLE_EXISTS_ACTION=append &
nohup impdp \'/ as sysdba\' directory=public_dump dumpfile=T_CA_TX_HIS_03.dmp logfile=T_CA_TX_HIS_03.log cluster=no REMAP_TABLESPACE=PROD_TBS:CC_OP_DATA REMAP_SCHEMA=PROD:CC_OP TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y TABLE_EXISTS_ACTION=append &
nohup impdp \'/ as sysdba\' directory=public_dump dumpfile=T_CA_TX_HIS_04.dmp logfile=T_CA_TX_HIS_04.log cluster=no REMAP_TABLESPACE=PROD_TBS:CC_OP_DATA REMAP_SCHEMA=PROD:CC_OP TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y TABLE_EXISTS_ACTION=append &
nohup impdp \'/ as sysdba\' directory=public_dump dumpfile=T_CA_TX_HIS_05.dmp logfile=T_CA_TX_HIS_05.log cluster=no REMAP_TABLESPACE=PROD_TBS:CC_OP_DATA REMAP_SCHEMA=PROD:CC_OP TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y TABLE_EXISTS_ACTION=append &
nohup impdp \'/ as sysdba\' directory=public_dump dumpfile=T_CA_TX_HIS_06.dmp logfile=T_CA_TX_HIS_06.log cluster=no REMAP_TABLESPACE=PROD_TBS:CC_OP_DATA REMAP_SCHEMA=PROD:CC_OP TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y TABLE_EXISTS_ACTION=append &
nohup impdp \'/ as sysdba\' directory=public_dump dumpfile=T_CA_TX_HIS_07.dmp logfile=T_CA_TX_HIS_07.log cluster=no REMAP_TABLESPACE=PROD_TBS:CC_OP_DATA REMAP_SCHEMA=PROD:CC_OP TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y TABLE_EXISTS_ACTION=append &
nohup impdp \'/ as sysdba\' directory=public_dump dumpfile=T_CA_TX_HIS_08.dmp logfile=T_CA_TX_HIS_08.log cluster=no REMAP_TABLESPACE=PROD_TBS:CC_OP_DATA REMAP_SCHEMA=PROD:CC_OP TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y TABLE_EXISTS_ACTION=append &

Note: The TRANSFORM parameter of impdp since Oracle 12c has been expanded to include the DISABLE_ARCHIVE_LOGGING option. The default value for this option is "N" and does not affect logging behavior. Setting this option to "Y" will cause the table and index to set the index attribute to NOLOGGING before importing, thereby reducing the generation of related logs during import, and then reset the log attribute to LOGGING after importing. If the target database has ADG, OGG and other replication software that enable force logging at the database level, then the "TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y" parameter will be invalid. A large number of archive logs are also generated.

Looking at the following log, the minimum import time is 7 minutes, the second dmp takes 14 minutes, and the last longest time is 54 minutes. It can be seen that the import is serial, and this does not shorten the time.

# more T_CA_TX_HIS_02.log
;;; 
Import: Release 19.0.0.0.0 - Production on Wed Nov 29 17:50:49 2023
Version 19.21.0.0.0


Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.
;;; 
Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Master table "SYS"."SYS_IMPORT_FULL_04" successfully loaded/unloaded
Starting "SYS"."SYS_IMPORT_FULL_04":  "/******** AS SYSDBA" directory=public_dump dumpfile=T_CA_TX_HIS_02.dmp logfile=T_CA_TX_HIS_02.log cluster=no REMAP_TABLESPACE=PROD
_SCFOP_TBS:CC_OP_DATA REMAP_SCHEMA=PROD:CC_OP TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y TABLE_EXISTS_ACTION=append 
Processing object type TABLE_EXPORT/TABLE/PROCACT_INSTANCE
Processing object type TABLE_EXPORT/TABLE/TABLE
Table "CC_OP"."T_CA_TX_HIS" exists. Data will be appended to existing table but all dependent metadata will be skipped due to table_exists_action of append
Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
. . imported "CC_OP"."T_CA_TX_HIS"         475.4 MB  189956 rows
Processing object type TABLE_EXPORT/TABLE/GRANT/OWNER_GRANT/OBJECT_GRANT
Processing object type TABLE_EXPORT/TABLE/COMMENT
Processing object type TABLE_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
Processing object type TABLE_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
Processing object type TABLE_EXPORT/TABLE/STATISTICS/MARKER
Job "SYS"."SYS_IMPORT_FULL_04" successfully completed at Wed Nov 29 18:45:39 2023 elapsed 0 00:54:44


Job "SYS"."SYS_IMPORT_FULL_06" completed with 1 error(s) at Wed Nov 29 18:05:29 2023 elapsed 0 00:14:31
Job "SYS"."SYS_IMPORT_FULL_04" successfully completed at Wed Nov 29 18:45:39 2023 elapsed 0 00:54:44
Job "SYS"."SYS_IMPORT_FULL_07" completed with 1 error(s) at Wed Nov 29 18:32:05 2023 elapsed 0 00:41:05
Job "SYS"."SYS_IMPORT_FULL_10" completed with 1 error(s) at Wed Nov 29 18:39:14 2023 elapsed 0 00:48:06
Job "SYS"."SYS_IMPORT_FULL_05" completed with 1 error(s) at Wed Nov 29 18:12:09 2023 elapsed 0 00:21:12
Job "SYS"."SYS_IMPORT_FULL_11" completed with 1 error(s) at Wed Nov 29 18:18:57 2023 elapsed 0 00:27:48
Job "SYS"."SYS_IMPORT_FULL_08" completed with 1 error(s) at Wed Nov 29 18:25:49 2023 elapsed 0 00:34:48
Job "SYS"."SYS_IMPORT_FULL_09" completed with 1 error(s) at Wed Nov 29 17:58:31 2023 elapsed 0 00:07:28

5、数据泵其他用法

expdp help=y
expdp 参数


DIRECTORY：用于转储文件和日志文件的目录对象。
DUMPFILE：指定导出备份文件的命名。
LOGFILE：指定导出备份日志的命名。里面记录了备份中的信息。
FULL：导出整个数据库 （默认是N，就是默认只会导出登录用户的所有数据）。
SCHEMAS：要导出的方案的列表 （指定想要导出哪个用户下的数据）。
EXCLUDE：排除特定对象类型。(表名要大写)
SAMPLE：要导出的数据的百分比。
TABLESPACES：标识要导出的表空间的列表。
VERSION：指定导出数据库的版本，一般用于高版本数据库的数据要导入到低版本数据库中时用到。
PARALLEL：更改当前作业的活动 worker 的数量。
REUSE_DUMPFILES：覆盖目标转储文件 (如果文件存在) [N]。
TABLES：标识要导出的表的列表。例如, TABLES=HR.EMPLOYEES,SH.SALES:SALES_1995。
QUERY：用于导出表的子集的谓词子句。例如, QUERY=employees:"WHERE department_id > 10"。
JOB_NAME：要创建的导出作业的名称。




impdp参数
impdp help=y


DIRECTORY             供转储文件, 日志文件和 sql 文件使用的目录对象。
DUMPFILE              要从 (expdat.dmp) 中导入的转储文件的列表,
LOGFILE               日志文件名 (import.log)。
FULL                  从源导入全部对象 (Y)。
SCHEMAS               要导入的方案的列表。
EXCLUDE               排除特定的对象类型, 例如 EXCLUDE=TABLE:EMP。
JOB_NAME              要创建的导入作业的名称。
TABLESPACES           标识要导入的表空间的列表。
REUSE_DATAFILES       如果表空间已存在, 则将其初始化 (N)
PARALLEL              更改当前作业的活动 worker 的数目。
QUERY                 用于导入表的子集的谓词子句。
VERSION               要导出的对象的版本, 其中有效关键字为:
TABLES                标识要导入的表的列表。
TABLE_EXISTS_ACTION   导入对象已存在时执行的操作。有效关键字: (SKIP)跳过, APPEND附加, REPLACE 替换和 TRUNCATE清空表后在添加。
REMAP_TABLESPACE      将表空间对象重新映射到另一个表空间。
REMAP_SCHEMA          将一个 schema 中的对象加载到另一个 schema。
REMAP_TABLE           将表名重新映射到另一个表。例如, REMAP_TABLE=EMP.EMPNO:REMAPPKG.EMPNO。

数据泵常用导入导出语句

--Data Mining and Real Application Testing options
--UDE-00010: multiple job modes requested, schema and tables.
schemas 和 tables 不能同时出现。


--仅导出元数据表结构
nohup expdp \'/ as sysdba\' directory=PUBLIC_DUMP SCHEMAS=DT_CC CONTENT=METADATA_ONLY exclude=STATISTICS LOGFILE=DT_CC0916.log dumpfile=expdp_METADATA_ONLY_DT_CC-2023-09-16_%U.dmp COMPRESSION=ALL PARALLEL=4 &


--SQL语句
SQL> select to_char(current_scn) from v$database;


TO_CHAR(CURRENT_SCN)
----------------------------------------
14898415798


--导出某 schema 的数据
nohup expdp \'/ as sysdba\' directory=PUBLIC_DUMP SCHEMAS=PROD  FLASHBACK_SCN=14898415798 exclude=STATISTICS LOGFILE=expdp_PROD0420.log dumpfile=expdp_PROD_2022-04-20_%U.dmp COMPRESSION=ALL CLUSTER=N PARALLEL=8 &  


--导出某用户下以 T_PRICE 开头的表。
nohup expdp PROD/'rop_P98#' directory=PUBLIC_DUMP LOGFILE=prod_t_price63Tables.log dumpfile=expdp_prod_t_price63Tables.dmp tables=t_price%  flashback_scn=9759642727  COMPRESSION=all exclude=statistics parallel=4 cluster=no &


--按条件导出大表的一部分数据
expdp PROD/'rop_P98#' directory=PUBLIC_DUMP LOGFILE=prod.T_AUTH_ORIGINAL.log dumpfile=expdp_prod.T_AUTH_ORIGINAL.dmp tables=T_AUTH_ORIGINAL COMPRESSION=ALL query="where DIGEST_TIME '>=' to_date\('2023-01-01','yyyy-mm-dd'\)"


--使用 parfile 导出表的一部分数据
expdp  scott/tiger  directory=exp_dir parfile=emp_main.par


vim emp_main.par
tables=emp_main
dumpfile=emp_main.dmp
logfile=emp_main.log
query="where sendtime between to_date('20220101','yyyymmdd') and to_date('20220401','yyyymmdd')"


--利用 SQLfile 参数生成创建索引，触发器，约束的 SQL 语句,该参数可以用于 impdp,主要作用是未真实在目标端执行导入的情况下,生成 sql 文件包含该 dmp 文件的所有 ddl 语句,使用语法为
impdp \'/ as sysdba \' directory=PUBLIC_DIR dumpfile=expdp_FULL_T2_CC_2022-10_17_%U.dmp logfile=T2_index.log sqlfile=t2_cre_index.sql include=INDEX  include=TRIGGER  include=CONSTRAINT


nohup impdp \'/ as sysdba\'  directory=EXPDP_DIR dumpfile=expdp_prod-2021-08-17_%U.dmp parallel=4 logfile=impdp_PROD_081720.log logtime=ALL TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y &
注意：在不管是非归档还是归档情况下使用 DISABLE_ARCHIVE_LOGGING 都会减小导入时间，减少归档量，但是需要注意如果数据库是 force logging 情况下，DISABLE_ARCHIVE_LOGGING 参数会无效。


--导入到其他用户
nohup impdp \'/ as sysdba\' directory=PUBLIC_DUMP LOGFILE=impdp_D2_CC0826.log dumpfile=expdp_prod_cc-2021-08-25_%U.dmp REMAP_SCHEMA=prod_cc:D2_CC REMAP_TABLESPACE=CC_DATA:D2_CC_DATA,CC_INDEX:D2_CC_INDEX PARALLEL=4 &


--查看表的统计信息
SQL> alter session set nls_date_format='yyyy-mm-dd hh24:mi:ss';
SQL> select TABLE_NAME,OWNER,NUM_ROWS,LAST_ANALYZED from dba_tables where table_name like 'ACT_GE_%' and owner='OP_DEMO';


--收集用户统计信息
SQL> exec dbms_stats.gather_schema_stats('OP_DEMO')
SQL> exec dbms_stats.gather_schema_stats(ownname => 'D5_CC',options => 'GATHER AUTO',estimate_percent => dbms_stats.auto_sample_size,method_opt => 'for all columns size repeat',degree => 5)


--只导出表结构和数据，排除索引和统计信息
nohup expdp \'/ as sysdba\'  directory=PUBLIC_DUMP SCHEMAS=prod exclude=STATISTICS,INDEX LOGFILE=prod0712.log dumpfile=onlydata_expdp_prod-2021-07-12_%U.dmp  COMPRESSION=ALL PARALLEL=4 CLUSTER=N &


--导入用户元数据
impdp \'/ as sysdba\' directory=DUMP_DIR LOGFILE=New_imp_T2_APP.log dumpfile=T2_APP-2021-05-31_NEW.dmp REMAP_SCHEMA=T2_APP:T1_APP REMAP_TABLESPACE=T2_APP_TBS:T1_APP_TBS 


--直接导入
nohup impdp \'/ as sysdba\' directory=EXPDP_DIR dumpfile=expdp_prod-2021-08-16_%U.dmp  parallel=4  logfile=impdp_PROD_0817.log logtime=ALL TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y &


--使用 dblink 不落地导入
nohup impdp system/Oracle_19C@TEST directory=EXPDP_DIR NETWORK_LINK=PROD_LINK flashback_scn=9010004930 exclude=statistics parallel=4 cluster=no schemas=PROD,PROD_CC logfile=impdp_PROD.log logtime=ALL TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y &
注意：logtime=ALL，Oracle 12c 以后的新参数，记录导入导出的时间，将时间信息输出到控制台和日志里。


--导出序列
SQL> SELECT ' CREATE SEQUENCE '||SEQUENCE_NAME|| ' INCREMENT BY '|| INCREMENT_BY ||' START WITH '||LAST_NUMBER||' MAXVALUE '|| MAX_VALUE ||'  CACHE '||CACHE_SIZE||' ORDER NOCYCLE ;' 
FROM user_SEQUENCES;


--导出序列
SQL> select dbms_metadata.get_ddl('SEQUENCE',u.object_name) from user_objects u where object_type='SEQUENCE';


--导出序列
nohup expdp \'/ as sysdba\' directory=PUBLIC_DUMP LOGFILE=prod.log dumpfile=expdp_prod-2021-05-21_%U.dmp SCHEMAS=prod COMPRESSION=ALL  PARALLEL=4 CLUSTER=N &


nohup impdp \'/ as sysdba\' directory=PUBLIC_DUMP  include=sequence LOGFILE=imp_D4_CC_SEQ.log dumpfile=expdp_prod_scfop-2021-07-09.dmp REMAP_SCHEMA=prod:D4_CC REMAP_TABLESPACE=CC_DATA:D4_CC_DATA PARALLEL=2 &


SQL> select sequence_name from user_sequences;
--SQL 大全  https://www.modb.pro/db/45337

导出部分数据

Migrating a full production database with TB-level data is time-consuming and labor-intensive. However, when creating a test environment, we often do not need to use all the data, but only use part of the data for functional testing. In this regard, Data Pump provides two methods for exporting part of the data. One method is QUERY, which is export based on conditions, similar to where in the query statement. For example, to export the first 5000 rows of data in each table under a business user, the command is as follows:

expdp \'/ as sysdba\'  parfile=expdp.par


vim expdp.par
directory=EXPDIR
parallel=8
CLUSTER=N
dumpfile=jieke_%U.dmp
logfile=jieke_1130.log
schemas=
('PROD','CC','APP','PROD_CC')
query="where rownum<=5000"

Another way is SAMPLE, which is the data sampling percentage. For the three modes of full database, user and table, we can use this parameter to set it when exporting. Examples of usage are as follows. Export 20% of the data in each table under the SCOTT user, the command is as follows:
Export 20% of the data in each table under the SCOTT user, the command is as follows:

expdp \'/ as sysdba\'  directory=EXPDIR dumpfile=scott_data.dmp logfile=scott_data.log schemas=scott sample=20

导出 SCOTT 用户下的所有表，但只对大表 emp 抽取 20% 的数据，命令如下：

expdp \'/ as sysdba\'  directory=EXPDIR dumpfile=scott_data.dmp logfile=scott_data.log schemas=scott sample=scott.emp:20

使用 attach 参数查看导入进度

impdp  system/Oracle attach=SYS_IMPORT_SCHEMA_02


expdp PROD/prod attach=SYS_EXPORT_TABLE_01


Export: Release 19.0.0.0.0 - Production on Wed Nov 29 15:32:47 2023
Version 19.15.0.0.0


Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.


Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production


Job: SYS_EXPORT_TABLE_01
  Owner: PROD                     
  Operation: EXPORT                         
  Creator Privs: FALSE                          
  GUID: 0B46179E09764E60E06347DD000CB607
  Start Time: Wednesday, 29 November, 2023 15:00:39
  Mode: TABLE                          
  Instance: jdb1
  Max Parallelism: 1
  Timezone: +00:00
  Timezone version: 32
  Endianness: LITTLE
  NLS character set: AL32UTF8
  NLS NCHAR character set: AL16UTF16
  EXPORT Job Parameters:
  Parameter Name      Parameter Value:
     CLIENT_COMMAND        PROD/******** directory=PUBLIC_DUMP LOGFILE=PROD.T_TX_HIS.log dumpfile=PROD.T_HIS.dmp tables=T_TX_HIS COMPRESSION=ALL CLUSTER=N 
     COMPRESSION           ALL                                     
     TRACE                 0
  State: EXECUTING                      
  Bytes Processed: 0
  Current Parallelism: 1
  Job Error Count: 0
  Job heartbeat: 1851
  Dump File: /data/ora-share/PROD.T_TX_HIS.dmp
    bytes written: 57,344
  
Worker 1 Status:
  Instance ID: 1
  Instance name: jdb1
  Host name: rac19c
  Object start time: Wednesday, 29 November, 2023 15:01:48
  Object status at: Wednesday, 29 November, 2023 15:01:48
  Process Name: DW00
  State: EXECUTING                      
  Object Schema: PROD
  Object Name: T_TX_HIS
  Object Type: TABLE_EXPORT/TABLE/TABLE_DATA
  Completed Objects: 1
  Total Objects: 1
  Completed Rows: 914,779
  Worker Parallelism: 1


Export> status

6. Commonly used SQL statements in import and export

col JOB_NAME for a30 
col OWNER_NAME for a20 
col OPERATION for a20 
col JOB_MODE for a20 
col STATE for a20
select * from dba_datapump_jobs;


OWNER_NAME           JOB_NAME                       OPERATION            JOB_MODE                       STATE                              DEGREE ATTACHED_SESSIONS DATAPUMP_SESSIONS
-------------------- ------------------------------ -------------------- ------------------------------ ------------------------------ ---------- ----------------- -----------------
SYSTEM               SYS_IMPORT_SCHEMA_02           IMPORT               SCHEMA                         EXECUTING                               4                 1                 6
SYSTEM               SYS_IMPORT_SCHEMA_01           IMPORT               SCHEMA                         NOT RUNNING                             0                 0                 0
查询该JOB涉及的对象：


col OBJECT_NAME for a30 
SELECT a.object_id, a.object_type, a.owner, a.object_name, a.status FROM dba_objects a, dba_datapump_jobs j 
WHERE a.owner=j.owner_name AND a.object_name=j.job_name and j.job_name='SYS_IMPORT_SCHEMA_01';
OBJECT_ID OBJECT_TYPE             OWNER                          OBJECT_NAME                    STATUS
---------- ----------------------- ------------------------------ ------------------------------ -------
    110320 TABLE                   SYSTEM                         SYS_IMPORT_SCHEMA_01           VALID
  
查询的结果显示：是一个系统在导入时生成的表：
select owner,object_name,subobject_name, object_type,last_ddl_time from dba_objects where object_id=110320;


OWNER                          OBJECT_NAME                    SUBOBJECT_NAME                 OBJECT_TYPE             LAST_DDL_TIME
------------------------------ ------------------------------ ------------------------------ ----------------------- -------------------
SYSTEM                         SYS_IMPORT_SCHEMA_01                                          TABLE                   2021-08-16 17:53:33


确定问题所在，接下来我们只需要删除IMPDP时产生的表：
drop table SYSTEM.SYS_IMPORT_SCHEMA_01 purge;


检查
select * from dba_datapump_jobs;


set line 200 pages 1000
col MESSAGE for a86
col TARGET for a35
select sid,SERIAL#,TARGET,START_TIME,ELAPSED_SECONDS,TIME_REMAINING,MESSAGE,ROUND(SOFAR/TOTALWORK*100,2) "%_COMPLETE"
from v$session_longops
where 1=1 
--and sid=2983 
--and SERIAL#=123
and TIME_REMAINING>0;

7. Best practices for using data pump

Refer to the video of Oracle Public Welfare Classroom B;

https://www.bilibili.com/video/BV1264y1b7a8

Basic concepts of exp/imp EXPDP/IMPDP

EXPDP and IMPDP are called Oracle Data Pump, which are import and export tools that began to appear in ORACLE 10G. Compared with Data Pump, EXP and IMP are called the original Export and Import utilities. Generally speaking, ORACLE recommends using Data Pump, because it supports all new features after ORACLE10G, but traditional import and export tools do not support it. Both types of import and export tools support cross-operating system platforms and cross-ORACLE versions.

The main differences between exp/imp EXPDP/IMPDP

1) EXPDP/IMPDP data pump is a server-side tool. It can only be used on the server side and not on the client side, nor can it be used on the DG library.

2) exp/imp can be used on the server and client, or on the read-only DG library

When using traditional import and export tools:

1. Need to import files generated by EXP

2. You need to export files that will be imported by IMP, such as exporting data from ORACLE 10G, and then importing it into a lower version of the database.

ORACLE data pump technology can transfer data and metadata between two libraries at very high speed. This technology is only available in ORACLE10.1 and later versions.

Data Pump Component

Data Pump consists of three parts:

1. Command line client (expdp and impdp)

2. DBMS_DATAPUMP package (that is, Data Pump API)

3. DBMS_METADATA package (also known as Metadata API)

The command line client is very similar to the traditional exp and imp, but it performs import and export naming by using the procedures provided by the DBMS_DATAPUMP package. All data pump execution processes are performed on the server side, which means that all unauthorized users need to ask the DBA to create a DIRECTORY to read and write data pump files. For authorized users, a default DIRECTORY is available. The DBMS_DATAPUMP and DBMS_METADATA packages can be used independently of expdp and impdp, just like other ORACLE packages.

Import and export transfer modes

1) exp export is divided into table mode, user mode and complete mode. Corresponding to the export table, export objects under the entire user and export all objects under the entire library.

2) EXPDP export is divided into table mode, user mode, database mode, and transportable table space mode.

Source and target libraries

1) The source database refers to the database that provides data sources

2) The target database refers to the database into which data needs to be imported.

19c Data Pump new features

Support cloud object storage as a storage target for data loading
Test mode for moveable tablespaces
Allow tablespaces to remain read-only during TTS import
Support resource usage limits
Exclude ENCRYPTIONN clause during import operation
Support wildcards for object storage files

Data Pump Best Practices

METRICS
Report additional job information to the import log file [NO].

Report additional job information to the import log file [NO].

LOGTIME
Specifies that messages displayed during import operations be timestamped.
Valid keyword values are: ALL, [NONE], LOGFILE and STATUS.

Specifies the timestamp of the message displayed during the import operation. Valid keyword values include: ALL, [NONE], LOGFILE, and STATUS.

To estimate the export file size, run EXPDP with the "ESTIMATE_ONLY=Y" option. When importing Oracle 12c and above, it is recommended to use the "TRANSFORM=DISABLE_ARCHIVE_LOGGING:Y" parameter to not generate archive logs in archive mode.

For data pump, it is recommended that you remember the "expdp/impdp -help" command. The usage of several key parameters is as follows.

·CLUSTER: The default is Y in RAC, which uses cluster resources to work in parallel. It is recommended to turn it off (CLUSTER=N).

·CONTENT: Export content, the default is ALL, you can choose metadata [METADATA_ONLY] or data only [DATA_ONLY].

·INCLUDE: The object specified when exporting/importing.

·EXCLUDE: Mutually exclusive with INCLUDE, objects excluded when exporting/importing.

·NETWORK_LINK: Remote database DBLINK connection.

·PARFILE: Parameter file to avoid the impact of different escape characters under different operating systems.

·QUERY: Specific object filtering conditions, similar to the where condition in select.

·SAMPLE: Data sampling ratio, generally used to build a test environment.

·SQLFILE: Generate SQL text statements for imported content.

·REMAP_SCHEMA/REMAP_TABLE/REMAP_TABLESPACE: used to redirect target import objects.

·COMPRESSION: Export file compression. You can use this parameter when space is tight. You can choose all content [ALL], metadata [METADATA_ONLY] or only data [DATA_ONLY].

·PARTITION_OPTIONS: used to migrate partition tables, NONE means to be consistent with the source, DEPARTITION means to create each partition table and sub-partition table as an independent table, and use the combination of the table and partition (sub-partition) name as its own name , MERGE means merging all partitions into one table.

To confirm whether the character sets are consistent, you need to ensure that the character set on the source library is the same as the character set on the target library, otherwise expdp/impdp (exp/imp) will easily report errors and lose data.

select userenv('language') from dual;


USERENV('LANGUAGE')
----------------------------------------------------


AMERICAN_AMERICA.AL32UTF8

The full text is complete. I hope it can help you who are reading this. If you think this article is helpful to you, you can share it with your friends and colleagues. Share it with anyone you care about, and learn and make progress together~~~

Welcome to follow my public account [JiekeXu DBA Road] and learn new knowledge together as soon as possible! You can find me at the following three addresses. The other addresses are all pirated and infringing articles that have been crawled from me, and the code formats, pictures, etc. are all messed up, making it inconvenient to read. Welcome to my official account or Mo Tianlun address to follow me, and I will be the first to do so. Get the latest news.

———————————————————————————
Public name: JiekeXu DBAnoji< /span>Motiendar: https://www.modb.pro/u/4347———————————————— ———————————腾讯云：https://cloud.tencent.com/developer/user/5645107
CSDN: https://blog.csdn.net/JiekeXu

Share several database backup scripts

Oracle table fragmentation check and defragmentation solution

OGG|Oracle GoldenGate 基础2022 年公众号历史文章合集整理

Several problems encountered by Oracle 19c RAC

OGG|Oracle 数据迁移后比对一致性