An item found performance problems, performance test results affect whether on-line, emergency external team of technical experts in the APT (Application Performance Testing application performance testing) before the on-line process.
Because they belong to different projects, only the information provided by the project team for analysis.
The first round of judging
Now like
The theme APT performance monitoring, and the left is the number of concurrent users CPU usage, CPU usage right is a separate case. APT test scenarios for the number of concurrent users increases at 5 minute period, the number of users 25, 40 reach a final 200 minutes after 1 hour concurrent users 200 and user activity state.
It is clear that the user stable CPU usage continues to grow, suspected memory leaks.
analysis
We got the first round of the API source code, which outOfMemory error log, check again find API to create a controller method HttpClient instance. He asked the group to develop responses are different because each request URL, based on experience recommended HttpWebRequest replace tune HttpClient. We investigated whether a memory leak problem caused by the HttpClient.
By Google search results HttpClient does have a memory leak report, we have the handwriting of a relatively small program HttpClient HttpWebRequest and performance, and performance testing locally using JMeter.
HttpClient and HttpWebRequest performance than the more
test environment
Test the URL: https://www.baidu.com/#{number} (Random Number variable is incremented or HTTP cache to prevent interference performance comparison results)
Rhythm Test: 10 seconds cycle, concurrent users increased from 1-200
The test results (memory usage)
- HttpClient: 455MB
- HttpWebRequest: 242MB
图1: HttpClient (GetAsync)
FIG 2: HttpWebRequest (Ignore projection occurs at the end of a network connection problem)
in conclusion
HttpClient will replace the use of HttpWebRequest, but according to HttpClient memory usage increased sharply magnitude estimates there are other issues to inform the development team continues to investigate and provide relevant information.
The second round of judging
Now like
Memory Usage compare IIS and SQL Server memory usage found the same growth trend, guess bottlenecks in the database query (if it is denied access to the database will not increase memory, must have been connected to a database and generate inquiries lead to).
Recommended to increase query log for query execution time, install the database performance monitoring tool (available with Windows Server comes with the Performance Monitor or third-party tools such as AppDynamic), view the execution plan for the query (Execution Plan).
Figure 3: SQL Server - Memory -% committed Bytes (memory usage)
FIG 4: MS IIS - memory -% Committed Bytes (memory usage)
After getting to know while API (API nested) API call will be upstream of the system, so consider checking the log whether there is any cause bottlenecks upstream API call fails, resulting in abnormal block the request process. There will be the same phenomenon, the development team set out to investigate.
API upstream problem by adjusting the system configuration to eliminate bottlenecks, but SQL Server performance issues still no clue. We can only arrange a time with the review and the corresponding database query performance issues.
The third round of review
Background :
The total test duration 1 hour and 40 minutes: 1 minute from the increase of five concurrent users beginning at intervals of 40 minutes was added to 200 concurrent, then maintained unchanged concurrent users, and perform one hour
Phenomenon :
Remained relatively stable ratio after 200 is reached after 20 minutes concurrent, the CPU utilization rate of 30%, but in the process which occurred after the execution request queued queues curve, sudden growth curves appears Thread Count, and with a response time doubling.
The last report no HTTP 5xx error, HTTP 5xx error this occurs, the surge in the number of errors on the 27th.
There program extracts data from the data generating one million cascade selection (drop down menu) UI elements.
Procedures used Http Cache, but no lock processing logic Cache data preparation, data preparation time is too long, it will cause a large number of requests to access the database during.
SQL Server performance test report also showed periodic reach 100% CPU utilization, triggering a database connection timeout, while the application appears GATEWAY_TIMEOUT.
System.Data.Entity.Core.EntityException: The underlying provider failed on Open. ---> System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
analysis
Because the assessment has been done two, the development team to provide comprehensive information, there is logic in using LDAP (Windows Active Directory Lightweight Directory Access Protocol Light Directory Access Protocol) traversing the group under the lander through code review all Active Directory discovery .
This logic can create a bottleneck ADFS query request, block the request.
同时发现部分被查询的表中没有创建索引（这个是在最初问过开发团队但得到的回答是肯定的）并进行索引优化，另外，API采用.NET Entity Framework编写，由SQL查询监控看出因无索引导致EF生成相同表的嵌套（笛卡尔积）查询，是导致表遍历和产生表锁的主要原因。
对于大量出现的5xx错误，因上次性能测试没有，主要针对修改的代码进行评审发现增加了EF查询逻辑，这也是导致循环嵌套的原因。
结论
表索引优化是主要解决办法。EF查询最终改为明文查询并SQL查询优化。
总结
性能问题首先由测试入手，根据测试报告发现错误和性能瓶颈，主要以解决错误消除瓶颈为主要原则。对于有数据库存在的情景，注意查询和索引优化，保证连接池有效支持应用并发。问题调研期间要了解整体架构（本次API嵌套问题和LDAP的使用都是后期由开发组告知）和所使用的技术，才能给出全面建议。从开发组角度应该训练发现性能问题的敏感度。所有问题的发现和解决以测试事实数据说话，不会存在莫须有的性能问题，所有问题皆有因缘。