I. Background drainage test
Most of the work is routine tests in a test environment to validate the system by simulating user behavior, including functionality and performance. In this process, you may encounter the following problems:
-
User access behavior is complex, difficult and consistent simulation of user behavior, simulation not true;
-
Line simulation scenarios under limited circumstances covering insufficiency may occur.
Appeared drainage test is to solve the above problems, it is the essence : by copying the real flow line to the next line environment to address environmental simulation test is not true, or not comprehensive enough to cover the problem.
II. Common drainage test program
At present many companies to drainage tests were conducted to explore the practice, common are the following four kinds of drainage way (limited):
| Drainage way | Method to realize | advantage | Shortcoming |
| Nginx layer traffic Copy | Installation lua-nginx-module, ngxdevelkit, headers-more-nginx-module module, the replication request to the test environment through lua script. | Also relatively simple to install and deploy. | We need to develop lua script, and resources have a certain occupation. |
| Based business code drainage layer | The call service request object code into a package, an asynchronous write cache. Drainage tool by reading the cache request, send a request to the test environment. | Scalable, customizable high. | Need to develop tools to copy flow, a larger workload, and high performance requirements of the job itself. |
| Based on the access log replay | Online system records access logs, log analysis tools drainage, initiates a request to the test environment. | Offline, the impact on small online. | We need to develop a log analysis tool, and analytical cost will be more. |
| Based drainage TCP / IP layer | Currently open source tools TcpCopy | Install, deploy, use are relatively simple. | Only for stateless service, and the service request needs to be filtered. |
Several of these drainage methods have advantages and disadvantages, some need to develop their own appropriate tools to support. And to tell you today is another an easy to use, low-cost learning tool for drainage Gor.
Three recommend a drainage tool:. Gor
Gor, also known as GoReplay, is the use of an open source HTTP traffic in real-time replication tool written Golang. It only needs to be performed on a server LB or Varnish entry process, you can copy the production environment to flow anywhere, such as Staging environment, Dev environment. Commonly used in the pressure sensing line and recurring problem.
Note: Golang is achieved using Gor and open source, which means you can easily integrate into their architecture, can be used in the stress test platform, real-time traffic analysis, application-layer firewall.
Four. Gor workflow
Capture http traffic monitor server and sends it to the server or saved to a file replay, replay server to forward traffic to a given address.
Note: Gor not an agent, does not require third-party tools into the critical path. Instead, Gor just quietly analyze traffic of your application does not affect the program itself.
Five. Gor support common functions
1, Gor supports zoom in and out of traffic, frequency limit, so do not need to build a consistent environment and production server cluster can also be tested properly.
2, Gor also supports traffic filtering based on regular expressions, which means that you can test an API service alone.
3, Gor can also modify the HTTP request header, such as the replacement User-Agent, or add some HTTP Header.
4, Gor can also request to record a file for playback and analysis. Gor support and ElasticSearch integration, will flow into ES real-time analysis.
Six. Gor download and install
Download the official website address: https: //github.com/buger/goreplay/releases download the latest Gor binary files (supports Windows, Linux x64, and Mac OS provide pre-compiled binary files), or may be self- compiled.
Get gor version: https://github.com/buger/goreplay/releases
Source Address: https://github.com/buger/goreplay
Use official document: https://github.com/buger/gor/wiki
Seven. Gor usage of common commands
Available Input:
-
--input-raw- for capturing HTTP traffic, you should specify the IP address or port interfaces and applications. About the capture and replay traffic for more information. -
--input-file- file used before accepting--output-file. More about saving and replay from a file -
--input-tcp- If you decide to forward traffic from multiple transponders Gor instance to it, by Gor polymerization instance. Read about using Aggregator-forwarder settings .
Available output:
-
--output-http- Replay HTTP traffic to a given endpoint, receiving basic URL. Read [more information about it] (replay HTTP traffic) -
--output-file- Record incoming traffic to a file. More about saving and replay from a file -
--output-tcp- the incoming data Gor forwarded to another example, and used therewith--input-tcp. Read about Aggregator-forwarder settings for more information. -
--output-stdout- for debugging output all the data to stdout.
More parameters can be found in the official website of the document.
Eight. Gor common practice scene
1, real-time replication drainage flow rate (--input-raw knockdown output port with --output-http), such as copying HTTP traffic port 80 of the machine in real time targer_server: 8080.
gor --input-raw :80 --output-http "http://target_server:8080"
2, if the destination server using the online bank and like machines, and only requires drainage Get method request.
gor --input-raw :80 --http-allow-method GET --output-http "http://target_server:8080"
3, when it is necessary for the overall performance of the online service pressure measurement, the line may be requested N times expansion, for drainage, the request will be doubled, may be reduced, to adjust the "|" to the percentage of the latter.
gor --input-raw :80 --http-allow-method GET --output-http "http://target_server:8080|200%"
4, just copy a URL request, - http-allow-url parameters, -http-allow-url parameter can be a regular expression (--output-http-url-regexp has expired gor 0.16, use --http- allow-url instead).
gor --input-raw :8080 --http-allow-method GET --output-http "http://target_server:8080" --http-allow-url mall.*hotword
5, multi-flow target server replication drainage, nginx is somewhat similar to the mirror.
gor --input-raw :80 --output-http "http://target_server:8080" --output-http "http://target_server2:8080"
6, forward traffic to multiple endpoints, by default, it will send the same traffic to all outputs.
gor --input-tcp :28020 --output-http "http://staging.com" --output-http "http://dev.com"
7, the same traffic to be sent to multiple sites, and bisects all traffic.
gor --input-raw :80 --output-http "http://staging.com" --output-http "http://dev.com" --split-output true
8, HTTP timeouts, default, http request and response timeout time is 5 seconds. You can cover it like this:
gor --input-tcp replay.local:28020 --output-http http://staging.com --output-http-timeout 30s
9, Basic authentication, if your development or staging environment is protected by basic authentication, you can inject these credentials during playback:
gor --input-raw :80 --output-http "http://user:[email protected]"
10, a pressure test performance
Traffic can be copied to a file, and then for them to play back. Playback time, traffic will maintain the original time interval. If you use a percentage rate limit, then the playback rate will correspondingly increase or decrease. With this rate limiting, gor can be used to pressure test.
gor --input-file "requests.gor|200%" --output-http "staging.com"At present, input-fileonly supports this feature only if based on a percentage of the limiter. Limiter different from the default input-file, it will not reduce the rate request, the acceleration request can slow or emission.
11, recording and playback
./gor --input-raw :8000 --output-file=requests.gorAfter executing the recording command will create a new file and continue to write to all requests for capture.
12, the flow rate playback, the requested file from the replay.
./gor --input-file requests.gor --output-http="http://localhost:8001"You should see all the records to http: // localhost: 8001 request, and they will replay the same order, and recorded exactly the same time.
gor usage is far more than these, more applications test scenarios mentioned, also requested the reader to dig.