simulates the whole process of data collection, records the time used in the whole collection process, and the time used by each collection sub-process, monitors the resource utilization of TOMCAT and the application server, and finds out the performance bottleneck. Adjust various configuration parameters to optimize performance.
1.1 Source library XML package:
<?xml version="1.0" encoding="UTF-8"?>
<MHC_BabyVisitInfo>
<PregnantID de="DEX04.01.001.01">
0000000000026888</PregnantID>
<BabyID de="DEX04. 04.001.01">
0000000000046310</BabyID>
<BabyName de="DE02.01.039.00" />
<BabyNO de="DEX04.04.001.02" />
<BabySex de="DE02.01.040.00" display=" Male" localCode="1"
localText="male">1</BabySex>
<BabyBirth de="DE02.01.005.01">2011-09-25</BabyBirth>
<FatherName de="DE02.01.039.00">fdf</FatherName>
<FatherJob de="DEX04.04.001.13" />
<FatherPhone de="DE02.01.010.00">13067916410</FatherPhone>
<FatherBirth de="DE02.01.005.01">1959-02-12</FatherBirth>
<MotherName de="DE02.01.039.00">ss</MotherName>
<MotherJob de="DEX04.04.001.13" display="不便分类的其他从业人员"
localCode="Y" localText="不便分类的其他从业人员">Y</MotherJob>
<MotherPhone de="DE02.01.010.00">ss</MotherPhone>
<MotherBirth de="DE02.01.005.01">1981-06-17</MotherBirth>
<MotherCardNO de="DE02.01.030.00">330102198706172121</MotherCardNO>
<Gestation de="DE02.10.006.00" />
<PregnancyDisease de="DEX04.04.001.07" />
<OtherDisease de="DEX04.04.001.08" />
<DeliveryUnit de="DEX04.04.001.09" />
<BirthStatus de="DEX04.04.001.05" />
<OtherStatus de="DEX04.04.001.06" />
<Asphyxia de="DEX04.04.001.10" display="无" localCode="1"
localText="无">1</Asphyxia>
<MalforMation de="DEX04.04.001.12" display="未发现"
localCode="2" localText="未发现">2</MalforMation>
<MalforMationDescription de="DE02.10.025.00" />
<HearingTest de="DEX04.04.001.00" />
<Weight de="DEX03.02.001.15" />
<Length de="DEX03.02.001.16" />
<FeedWay de="DEX03.10.001.28" />
<InputUnit de="DEX03.05.001.16">33010500222111</InputUnit>
<InputUser de="DEX06.04.001.05">gsmsw052222111</InputUser>
<InputDate de="DEX06.05.001.16">2010-10-25</InputDate>
<LastModifyUser de="DEX01.01.002.12">gsmsw052222111</LastModifyUser>
<LastModifyDate de="DEX01.01.002.13"> 2010-10-25</LastModifyDate>
<MPIID de="DEX01.01.001.24">ec433954197a4cf6a42012a12121bc3f3d432</MPIID>
</MHC_BabyVisitInfo>
1.2 Example: Parse 8 fields
InputUnit, InputDate, Weight, Length, FeedWay in XML package , BabyBirth, BabyID, MalforMation
2 Program deployment
2.1 Data collection process
The entire data collection process is divided into one stage:
2.2 Server deployment
The entire data collection process is deployed on the following 2 servers:
3.1 Hardware configuration
3. 2 Software configuration
4 Test time
Time 2013-01-25 to 2013-02-02
5 Test plan Unpack the
acquisition program and save the index data to the corresponding index table
6 Test results
6.1.1 Plan 1 Test result record
Plan 1,
Single thread, one script collects 113W data.
Test results:
single-threaded collection of 113W data, as shown in the figure, the total time is 25.40 seconds, 741.5 per second. The operation process is relatively stable, basically at the average speed, but when it runs to 1/4, there is a period of slow speed, which may be caused by network conditions. As shown
in the figure As shown in the figure: Performance monitoring for process execution Figure
1. The situation of resource monitoring when the collection script is just run
2. The situation of resource monitoring when a single thread collects 10W data
3. When a single thread collects 50W data, the Monitoring situation
4.113W collection to complete service resource monitoring
6.1.2 Solution 2 test result record
Multi- threaded test, execute 10 scripts at the same time, each script collects about 10W of data, and saves the data to the library.
Test results:
Multi-threaded collection of 113W data, 10 threads each collect about 10W of data, as shown in the figure, the average time is 290 seconds, and the 23W time is 363 seconds. The operation process is relatively stable, basically at the average speed. As shown in the figure
Scheme data volume time server resource remarks
Average time average 0W 50W 113W
Scheme one 1130000 290 seconds 350 pieces/second Cpu: within 0.51%
Memory 36%
Network: 426KB
I/O: 42 kilobytes per second Cpu: within 6.63%
Memory : within 39%
Network: 587KB
I/O: 52 kilobytes per second Cpu: within 0.51%
Memory : within 37%
Network: 0KB
I/O: 112 kilobytes per second
As shown in the figure: Data collection situation
1. When multi-threading is just starting to execute:
2. Multi-threading collects data halfway, about 1 minute, and resource monitoring is as follows:
3 . It takes about 3 minutes after the multi-threaded data collection is completed. The resource monitoring is as follows:
6.1.3 Test result record of scheme 3
Multi- threaded test, execute 10 scripts at the same time, each script collects 100W of data, and saves the data to the library .
Test results:
Multi-thread collection of 1000W data, 10 threads each collect about 100W of data, as shown in the figure, the average time is 4000 seconds. The operation process is relatively stable, basically at the average speed. When the dividing line is collected each time (such as 5W, 10W, 15W, etc.), the CPU utilization will increase, the program will print a prompt for the baseline, and the program is in the process of releasing memory and reading new data. According to the statistical results, it can be found that the multiple scripts of the ETL service collect the same service, the collection efficiency is average, and the difference between the actual collection time of 10W and 100W is not very large. The data collection pressure on the application server is not large, so it is inferred that the performance pressure should be on the database server. As shown in the figure
Scheme data volume time server resource remarks
Average time average 10W 50W 100W
scheme 1 1000W 4000 seconds 260 pieces/second Cpu: within 0.51%
Memory : within 9%
Network: 9%
I/O: 42 kilobytes per Cpu per second: within 0.9%
Memory : within 11%
Network: 9%
I/O: 52 kilobytes per second Cpu: within 1.53%
Memory : within 14%
Network: 9%
I/O: 112 kilobytes per second
1. 10 threads collect about 5 minutes, resource service usage:
2. 10 threads collect 10W data, resource service usage:
3. 10 threads About 10 minutes of collection, resource service usage:
4. 10 threads collect about 15 minutes, resource service usage:
5. 10 threads collect about 50W, resource service usage:
6. 10 threads collect about 100W , the usage of resource services:
6.1.4 Option 4: Recording of test results
Multi- threaded test, execute 25 scripts at the same time, each script collects 40W of data, and saves the data to the library.
Test results:
Multi-threaded collection of 1000W data, 25 scripts each collect about 40W of data, as shown in the figure, the average time is 10000 seconds, and the average speed is 40 pieces/second. The operation process is relatively stable, basically at the average speed. According to the statistical results, the collection time of each script in Scheme 3 is basically the same and still exists. In the resource monitoring of the database server, the CPU and memory will increase during the initial collection. The memory usage is about 7.2G, and the CPU usage is 4%. No abnormality was found during the collection process, and the CPU and memory usage were relatively stable. The CPU usage is kept at 2%, the highest is 4%, and the memory usage is kept at around 7G. As shown in the figure,
plan data volume time server resource remarks
Average time average 10W 35W 40
plan 1 1000W 10000 seconds 40 pieces/second Cpu: within 0.48%
Memory : within 9%
Network: 9%
I/O: 42 kilobytes per Second Cpu: within 0.9%
Memory: within 10%
Network : 7%
I/O: 52 kilobytes per second Cpu: within 0.64%
Memory 11%
Network: 8%
I/O: 112 kilobytes per second
1. 25 threads per second collect 10W About, the usage of resource services:
2. 25 threads collect about 35W, the usage of resource services:
3. 25 threads collect about 40W, the usage of resource services:
4. The usage of resource services for 25 threads database services Situation:
6.1.5 Option 5 Test result record
Multi- threaded test, execute 50 scripts at the same time, each script collects 20 data volumes, and saves the data to the library.
Test results:
Multi-threaded collection of 1000W data, 50 scripts each collect about 20 data, as shown in the figure, the average time is 19000 seconds, and the average speed is 21 pieces/second. The operation process is relatively stable, basically at the average speed. In the database server segment memory, the CPU is relatively stable, the CPU is kept at 2%, and the memory usage is kept at about 7.02. The application server also found no exceptions. As shown in the figure
1. The usage of the resource service of the 50-thread application server:
2. The usage of the resource service of the 50-thread database server:
6.1.6 Scenario 6 test result record
Multi- thread test, execute 100 scripts at the same time, Each script collects 10W of data and saves the data to the library.
Test Results:
Multithreading collects 1000W of data, and each script collects about 10W of data for 100 scripts. The application server is relatively stable during the collection process. When the database server just starts the script, the CPU and memory usage is relatively high. The average CPU usage remains at 12%, the highest is 17%, and the memory remains at 7.35G. As shown in the figure
1. The usage of the resource service of the 100-thread application server:
2. The usage of the resource service of the database server after 100 threads are collected for 5 minutes:
3. The resource service of the database server during the 100-thread data collection process Usage:
6.1.7 Option 7 Test result record
Multi- threaded test, execute 150 scripts at the same time, each script collects 10W of data, and saves the data to the library.
Test results:
150 scripts each collect about 10W of data. When the database server just starts the script, the CPU and memory usage is relatively high by about 25%, and the CPU number is relatively large during the collection process. The average CPU usage remains at 17%, the highest is 40%, and the memory remains at 7.4G. As shown in the figure
1. The usage of the resource service of the
100-thread database: 2. The usage of the resource service of the 100-thread database:
3. The usage of the resource service of the 100-thread application server:
4. The application of 100 threads The usage of the resource service of the server:
5. The usage of the resource service of the 100-thread application server:
7 Summary of the test results
Multi- threaded collection, the server resource utilization is more fully utilized, and the resource monitoring diagram shows that I/O is found when the multi-threaded test is performed. , Memory, network, etc. have not found obvious fluctuations, and the collected data is relatively stable.
In the second stage, during the test, it was found that when the dividing line was collected each time (such as 5W, 10W, 15W, etc.), the CPU utilization rate would increase, the program would print a prompt for the baseline, and the program would be releasing memory and reading new data. According to the statistical results, it can be found that the multiple scripts of the ETL service collect the same service, the collection efficiency is average, and the difference between the actual collection time of 10W and 100W is not very large. During the test, the data collection pressure of the application server is not great, and TOMCAT sets 2G memory to collect data at 1500W. In the process of increasing the business scripts of the database server, the CPU and memory usage continue to increase. When 150 scripts are executed at the same time, the CPU usage changes greatly, up to 40%. It is inferred that the performance pressure should be on the database server.