The BUG of Datastage has been corrected officially announced by IBM,
http://www-01.ibm.com/support/docview.wss?uid=swg21438499
Personally, I think that there are indeed a lot of problems with DataStage. If you don’t believe me, you can see the bug list on the official website;
The following is a summary of some component problems and error-prone operation problems encountered when using DataStage in the following work.
1. The operating system where the DS is located
The environment adopted by DS in this project (Galaxy ACRM) is based on the experience of CIC Data Center, so Red Hat Linux As 4 Update 8 x86_64 + Datastage 7.5.2.1 is chosen.
Later, in the process of solving the problem, I found such a sentence "Windows/Linux platforms are only supported on Intel x86 32-bit Processor" from the release notes. It was
verified that the choice to install DS on Linux X86_64 this time caused subsequent some problems!
After installing the software, be sure to read the release notes of the software! If it is suitable for 32bit system, then don't choose 64bit operating system!
Recommended: Red Hat Linux As 4 Update 8 x86 + Datastage 7.5.2.1
2.Complier selection
You won't have this problem on a 32-bit system!
Because the 64-bit system is selected, but Datastage 7.5.2.1 only supports 32-bit systems, the compiler part in the environment variable is adjusted as follows:
IBM's official configuration:
APT_COMPILEOPT: -O -fPIC -Wno-deprecated -c -m32
APT_COMPILER:g++32 -m32
APT_LINKER:g++32 -m32
APT_LINKOPT:-shared -32 -Wl,-Bsymbolic,--allow-shlib-undefined
My own config:
APT_COMPILEOPT: -O -fPIC - Wno-deprecated -c
APT_COMPILER:g++32 -m32
APT_LINKER:g++32 -m32
APT_LINKOPT:-shared -Wl,-Bsymbolic,–allow-shlib-undefined
3. Selection of Oracle lib
Because it is a 64-bit operating system, the 64-bit client of Oracle 10g is also installed.
4. The problem of Oracle character set conversion
Set NLS_LANG in the operating system environment variable and DS environment variable, and also need to do character set mapping,
add in /home/dsadm/Ascential/DataStage/PXEngine.752.1/etc/oracle_cs.txt:
GBK ZHS16GBK
ZHS16GBK GBK
first The column is the built-in character set of DS, and the second column is the character set of Oracle. According to this statement, the second line is not needed. I still add it here!
5.DS shared memory tuning
In the version before DS8.1, the command SHMTEST is only valid on Linux 32-bit, select infinite loop on Linux 64-bit, IBM has fixed it in 8.1!
BUG fix address: http://www-01.ibm.com/support/docview.wss?rs=14&uid=swg1JR32113 , no more excerpts here!
6.DS connect to Oracle RAC
If the Oracle database uses RAC, which used to be called (Oracle Parallel Server, OPS), the database user used for data extraction must have access to SYS.GV_$INSTANCE in addition to other privileges.
In addition, you must also add the Project environment variable, APT_ORACLE_NO_OPS = 1, in the Datastage Administrator.
7. Permissions of Oracle users in DS
When using oracle, DS needs to grant permissions to the following views
DBA_EXTENTS
DBA_DATA_FILES
DBA_TAB_PARTITONS
DBA_OBJECTS
ALL_PART_INDEXES
ALL_PART_TABLES
ALL_INDEXES
SYS.GV_$$INSTANCE (Only if Oracle Parallel Server is used or RAC)
This is explained in the online help document that comes with datastage,
We suggest that you create a role that has the appropriate SELECT privileges, as follows:
CREATE ROLE DSXE;
GRANT SELECT on sys.dba_extents to DSXE;
GRANT SELECT on sys.gv_$instance to DSXE;
GRANT SELECT on sys.dba_data_files to DSXE;
GRANT SELECT on sys .dba_tab_partitions to DSXE;
GRANT SELECT on sys.dba_objects to DSXE;
GRANT SELECT on sys.all_part_indexes to DSXE;
GRANT SELECT on sys.all_part_tables to DSXE;
GRANT SELECT on sys.all_indexes to DSXE;
Once the role is created, grant it to users who will run DataStage jobs, as follows:
GRANT DSXE to ;
8.DS boot off
Before closing DS with uv -admin -stop in datastage, to exit all client connections, you must use ps -ef| grep ds to view the processes connected to DS, find the PID and kill it with the kill command,
or clear it in the director Disconnect all connections, and then use uv -admin -stop to close the DS, otherwise close the DS if there is a DS connection. When starting the DS with uv -admin -start,
dsrpcd may not be able to start. If the dsrpcd cannot be started after several attempts, please Refer to the article "Datastage's dsrpc tracking method in Linux environment"; the last resort is to restart the machine!
9. The problem of slow extraction
Datastage 7.5.2 Extracting Oracle data is too slow. Check the JOB log and find that the default.apt file in the /home/dsadm/Ascential/DataStage/Configurations directory is checked when the JOB is running.
This file is used to specify the node of the DS, which is generally Each CPU corresponds to one node, and there is only one node in my file. Increase the number of nodes according to the number of CPUs and restart DS.
During this extraction, monitoring JOB found that multiple nodes were used at the same time, and the extraction speed increased sharply!
10. TEXT type conversion in MS SQL SERVER
When Datstage obtains data from MS SQL Server through ODBC, the default text type field is Longvarchar, and there is an error when viewing data. Use convert in the custom SQL statement to convert text to varchar (4000),
and change the field type to varchar , otherwise an error occurs: DSP.Open GCI $DSP.Open error -100.
11. When DS extraction extracts Oracle data, spaces are converted to NULL by default
In Datastage 7.5.2.1, when Oracle Stage in DS extracts data from Oracle, spaces are converted to NULL by default, even if APT_ORACLE_PRESERVE_BLANKS in the DS environment variable is set to True, it does not help.
After searching, IBM officially believes that this is a bug, and it is recommended to upgrade and patch.
Patch address: http://www-01.ibm.com/support/docview.wss?rs=14&uid=swg1JR33670
Before patching, please back up the entire DS directory before patching!
When I applied the patch myself, I closed all DS-related processes, but the installation prompted an error, so I asked the system administrator to find it, but no errors were found in /home/dsadm/Ascential/patches/patch_JR35216/log.
The error message during installation:
Upgrading the Stage …
An error has occurred during DataStage initialization
Please contact the system administrator
Error code: 2 2
Done!
Fortunately, restart the DS, everything is normal, and by setting the APT_ORACLE_PRESERVE_BLANKS in the DS environment variable, You can control the conversion of spaces to NULL!
Fortunately, it is short-lived. After patching, the JOB that extracts SQL Server data from ODBC reports an error. It seems that it is because of the patch problem!
Error message:
main_program: Fatal Error: Fatal: Shared library (drsenu.so) failed to load: errno = (2),
system message = (/home/dsadm/Ascential/DataStage/DSEngine/lib/drsenu.so: undefined symbol: DSCUVTextMark)
Although there is a backup before patching, there is no problem in restoring it, so think of a way to solve it first!
I found a solution on the Internet, the same error, but others encountered $DSHOME/lib/dsdb2enu.so: undefined symbol: DSCUVTextMark, then try it!
Address: http://www.docin.com/p-42820466.html
First back up the $DSHOME/etc/operator.apt file, edit the file with VI, and change dscapiop dscapiop 1 to dscapiop @dscapiop 1, save and exit!
Close DS, reboot! I don't know why it does this! Searching through the official documents did not find any relevant content, the solution given on the Internet, the current problem is solved, but what is the basis?
Will this cause other potential problems? !
Despise DS here!
12.JOB troubleshooting, enable detailed log output
The log prompts a warning, but you don't know which stage prompts the warning, then set APT_DUMP_SCORE = TRUE, you can see the stage name in the log warning message.
In addition, there are some options for log output.
Memory usage, APT_PM_PLAYER_MEMORY = TRUE
CPU usage time, APT_PM_PLAYER_TIMING = TRUE
data extraction, APT_RECORD_COUNTS = TRUE
13.DS performance tuning
See the official documentation for details.
APT_DEFAULT_TRANSPORT_BLOCK_SIZE = 131072 changed to APT_DEFAULT_TRANSPORT_BLOCK_SIZE = 1048576
APT_BUFFER_MAXIMUM_MEMORY = 3145728 (3 MB) changed to APT_BUFFER_MAXIMUM_MEMORY = 6291456 (6 MB)
APT_BUFFER_FREE_RUN = 0.5 RUN = 0.5 APT_BUFFER
14. Problems with Oracle DB found during DS operation
When DS is running, the LOG log reports errors, which are all TNS and ORA errors, especially when multiple JOBs are running in parallel, and some JOBs fail.
Please seek DBA assistance to optimize the database!
15. The handling of special characters in Oracle in DS (for example: # $)
DS_ENABLE_RESERVED_CHAR_CONVERT = TRUE handles special characters #, $
16. Substr function in Transformer stage
The substr function in the Transformer stage intercepts characters according to the character object when extracting data, but when it is loaded into the target library, the length is calculated in bytes.
Especially when intercepting fields containing Chinese, the last character of some Chinese characters is cut into garbled characters, and an error occurs when loading to the target table, which
can be found in the bad file under the /home/dsadm/Ascential/DataStage/Scratch folder Bad data entry. ,
Recommendation:
For fields containing Chinese, the field length of the target table should be as consistent as possible with the field length of the source table.
17. When extracting data from MS SQL Server 2005 to Oracle, Chinese garbled characters are resolved
The NLS is GBK, and the view data is normal, but when it is extracted to Oracle, the Chinese characters are garbled.
In the Transformer, set the extend of the Chinese field to unicode.
- Original address: http://www.bi168.cn/ >> http://www.bi168.cn/thread-370-1-1.html
18. Service update does not take effect
Before the service is redeployed, it needs to be undeployed, and then edited and deployed again;
19. Failed to pass in service parameters
If the parameter is more than one (more than one) or the parameter value is more than one, it must be checked;
20. The service only returned the first piece of data
If the number of output data is more than one line (more than one line), you must check Return to Array;
21. Chinese garbled data returned by the service
If the output content of DataStage is Chinese, you need to set the Extended property of the Chinese column to Unicode;
22. An error is reported when running, the empty data column is not set to run as empty
If the DataStage output column contains null data, set the Nullable property of the column to Yes;
23. Compilation error, not set to run multiple instances and enable information service
When creating a job in DataStege, you need to select the job properties –> General –> (allow multiple instances) Allow Multiple Instance and (enable information services) Enabled for Information Services properties;
24. Compilation error, model field not found
When defining the query data output model, the defined data output model must be consistent with the data returned by the query SQL, including the column name, data type, data length, and data precision;
25. The operation fails, there is an empty data column in the data output, and the mapping entity cannot be found.
Use NullToValue in the Transformer control to process the empty data column. If there is empty data in a column or row in the output data, you must use this function. columns are processed.
NullToValue (input column, null replacement value) applies to all types,
NullToEmpty (input column) is used for character type, default output is empty data column,
NullToZero (input column) is used for numeric type, default output is 0.00;
26. The service continuously calls an error for a short time (within 10 seconds) (axis2ns1:Server)
For service deployment, it is generally not necessary to set the program properties. It is generated by default. However, due to the stress test of the IIS deployment service in the early stage, it was found that this was incorrect, and the continuous invocation of the service failed. Since the modified configuration here is gray, it cannot be edited. If If you need to modify it, you need to export the deployed service to the local, modify the configuration file, and then import it into the project for redeployment.
27. The imported table definition has Chinese garbled characters
Due to a little problem in the branch office environment, after importing, all Chinese characters will be garbled, and all of them are (?) question marks, so at present, it is necessary to manually modify the corresponding Chinese garbled characters;