Due to the needs of the project, 60 million pieces of data need to be imported into the database. Existing resources are txt text (data separated by ",").
Option 1: Convert it to an insert statement, but the execution efficiency is too low, so give up.
Option 2: Use the plsql tool (tool-text import), the efficiency is about 500,000 per hour. Not because of the urgency of time.
Option 3: Using sqlldr, this efficiency is not bad, about 3.5 million per hour. The following highlights.
tool: sqlloader
Prerequisite: Install oracle server
Environment: 60 million data, i5cpu , 4g memory, oracle10g
step:
The data during execution is as follows:
1. Write a ctl file ( risk.ctl in this example ):
OPTIONS(skip_index_maintenance=TRUE,direct=true,BINDSIZE=20971520,READSIZE=20971520,ERRORS=-1,ROWS=500000)
unrecoverable
load data
CHARACTERSET AL32UTF8
infile 'C:\Users\LHB\Desktop\TaxmodelData\PureRiskPremiumItem.txt'
insert into table pureriskpremium
Fields terminated by ','
trailing nullcols
(MODELCODE,
BASICRATECODE,
AREACODE,
BASICRISKPREMIUMS,
BIZVERSION,
EFFECTIVETIME,
OBTAINCREATETIME
);
explain:
ROWS=500000 every 500000 commits
C:\Users\LHB\Desktop\TaxmodelData\PureRiskPremiumItem.txt' needs to deal with the real
Fields terminated by ',' separator, in this case the separator is ","
pureriskpremium table name
2. Open cmd
sqlldr jyrluser/[email protected]:1521/xasccxdb1 control=C:\Users\LHB\Desktop\TaxmodelData\risk.ctl
log=c:\jrimp.log
Explanation: sqlldr username / password @ real ip : port number / instance name control= control file address log= log (log only when it fails)
3. After pressing Enter, if no error is reported, it means that it is being executed. Each time a certain number of executions will be prompted.
4. Debug the error message:
SQL*Loader-2026: Loading terminated because SQL Loader could not inherit
SQL*Loader-925: uldlfca: error in OCIStmtExecute(ptc_hp)
ORA-03114: not connected to ORACLE
At that time, when the execution reached about 3000, an error was reported, and then the error log was queried. It can be seen from the log that the field definition length was too short, resulting in 30 million errors reported, and then the field was expanded. Execute it again and it works fine.
C:\Users\Administrator>sqlldr xinjiatao/[email protected]:1521/ORCL control=F:\New_Data\A\test.ctl log=c:\xin.log data=F:\New_Data\A\score_20170715.csv