The Way of Testing - Alibaba's Eight Years of Dedication from Testing Experts
Abstract: I have been engaged in testing for nearly eight years. At first, I did not understand the test, doubted the test, believed in the test, and then determined the test. The bitterness and suffering I experienced are beyond words. In the eight years that I have been engaged in testing, some people have questioned it, and some people have pursued it. There is no end to it. It seems that testing is always a role on the cusp of public opinion.
1. Introduction
I have been engaged in testing for nearly eight years. From the beginning, I did not understand the test, doubted the test, believed in the test, and then determined the test. The bitterness and suffering I experienced are beyond words. In the eight years that I have been engaged in testing, some people have questioned it, and some people have pursued it. There is no end to it. It seems that testing is always a role on the cusp of public opinion. This article is written by Xia Zhixue. It is my high-level summary of the test. It aims to help everyone understand the test, so that newcomers can better engage in the test work, and the elderly can conduct test discussions and exchange ideas. In order to let as many people understand the test as possible, this article focuses on the narration and less on the technique of the test. I believe that after reading it, you will make your own judgments, and the merits and demerits will be left to the judges.
2. The universal model of testing
Why are you talking about this? The model of the test is not only my high-level modeling of the test cognition, but also the starting point to help everyone understand the test and understand my point of view. Yes, the road is simple. After understanding the core idea and then looking at the point of view, there will be an argument. This is just like cultivating martial arts. The foundation determines the height of martial arts in the future. It's eloquent, but everyone doesn't know what I'm talking about.
There are three realms in Buddhist practice: seeing mountains as mountains and water as water; seeing mountains as not mountains and seeing water as not water; seeing mountains as mountains and water as water. From my experience and cognition of the test, it is very consistent. When I first started to do the test, I felt that the test work was boring, boring, and not too technical. I thought this was the test. However, with the increase of work experience, I feel that testing is getting more and more difficult. Faced with various systems under test, I really cannot use a general method or general tool to meet all testing needs. So I started desperately learning the implementation of various systems, trying to understand my system under test. The systems I tested were developed in Java, C++, and other languages, so I had a certain understanding of various languages and started to study how to test them. As time passed and my understanding of testing gradually deepened, I found that all testing concepts were the same. Gradually, I realized the universal testing model: y= f(x). Yes, you read that right, it is the function expression we have learned.
x is our input, y is the output of f(x), which represents the function of the system under test. The idea of the test is: select an appropriate x, substitute it into f(x), get y, and compare it with my expected result y', so as to determine whether the tested process is pass or fail. Graphical representation:
For testers, SUT (System Under Test, system under test) is a black box. Testers generally do not pay attention to the specific implementation logic of f(x), but only focus on the function of f(x). For example, assuming f(x)= 2^x, the program can be implemented by "multiplying x by 2" or by "shifting left". As a tester, the focus is only on "Is there a correct implementation of the requirements? ", "How about the performance after the function is satisfied? Are there any security issues?", the focus is not on "how to implement", but on "how to implement" (this is also the essential difference between testing thinking and development thinking). Therefore, understanding the business and understanding the product requirements is the premise of testing.
Some people may ask, it is not so simple, the system is so complicated, how can it be summed up with only one y= f(x)? You only have one request and one response here, how complicated is the system, database, cache, asynchronous message, log, etc. It must be emphasized here that x, y represent not only request and response, but generalized input and output. What is the difference? request and response are only one type of input and output. For SUT, as long as the data read is considered input, for example: the function of user login, when I fill in a user name to log in, my input is not only filled in on the page Enter the "username" and "password", this user record must also exist in the DB (of course, the user does not exist is also a test scenario), so this user record in the DB is also considered an input, even if the login system process To query the security system, the "user security policy" returned by the security system is also considered an input. Therefore, the input here is a generalized input, including the "username/password" filled in the user page, the "user record" in the DB, and the "user security policy" returned safely. Similarly, the output is also generalized, including "DB write", "request to other systems", "printed log", "put to cache", "asynchronous message sent", etc. So our test universal formula can evolve into the following: y1,y2,y3,…,yn= f(x1,x2,x3,…,xn). Our test work is actually to determine the value range of each x, and then select the appropriate combination of data from x1 to xn (a set of data is actually a test case), substitute it into f, and then match the obtained y1...yn with the expected y1' ...yn' to compare, so as to judge the correctness of the tested scene. Represented by a diagram:
Therefore, a qualified test must clarify "the function of the SUT", "all the inputs of the SUT", "the value range of each input", "all the outputs of the SUT", "deduce each one according to the function" expected value of the output".
It should also be emphasized here that the SUT here is very particular. In my opinion, except for the static walking code, which is considered a white box test, all other tests are considered black boxes, but the size of the "box" is different. That's it. The "box" in unit testing is relatively small, that is, one or several methods; the "box" of interface testing will expand to the application level; the "box" of integration testing will expand to the system level.
Once you understand the test model, you can begin to analyze the key points of the test.
Third, the purpose of
testing The purpose of testing is to avoid bugs. Why use "avoid" instead of "find"? Because for all test cases, not every one can detect the bug, can you say that the test work has no value for the execution of the use case that fails to detect the bug? Obviously not, for testers, before the test is not executed, the premise is that all the tested processes are in an unknown state, and the process state becomes known only after the corresponding test case is executed - pass or fail, for We have found a bug in the fail test case, but we have not and cannot find a bug in the pass test case, so no matter whether the pass or fail, the test execution work is valuable. State the purpose of the test work.
Fourth, the test steps Let's
take a look at the model diagram of the test:
As mentioned above, the test work is actually to determine the value range of each x, and then select the appropriate combination of data from x1 to xn (a set of data is actually a test case. ), substitute f, and then compare the obtained y1...yn with the expected y1'...yn' to judge the correctness of the tested scene. From this, it can be concluded that the test work steps are:
"Determine the combined data of x1 to xn"
"Transfer each set of data into the SUT"
"Determine the expected output results y1' to yn' generated after each set of input data is input according to the requirements"
"Compare the expected results with the actual results y1, y2, ..., yn to draw conclusions"
5. Difficulties of testing
For the above four steps, points 3 and 4 are relatively easy. The main difficulty of testing lies in points 1 and 2:
1. How to find all x and y?
To find all the x and y, you must be very familiar with the system, very familiar with the process. How about system dependencies? Process call, how does the system handle and interact? What reactions occur? Typical inputs are: call request, read DB data, read cache data, return data from the dependent system, received asynchronous messages, etc.; typical outputs are: write DB, write cache, write log, call the request of the dependent system, Asynchronous messages sent, etc. So this requires you to be very, very familiar with your system under test and process.
2. How to determine a suitable combination of x1 to xn?
First of all, you need to be familiar with the possible values of each x. In addition to normal values, there are also abnormal values, which are very demanding for test engineers. In order to understand all possible values of x:
not only need, you are very, very familiar with business and business data. Note that here I separate "business" and "business data" and refer to two different things. "Business" refers to the functions provided by the product. For example, you can log in with your account and password, or you can log in with your mobile phone scan code. "Business data" refers to the data generated after the product runs around the clock online. For example, for the registration function, there are Taobao users registered through Taobao, Alipay users registered through Alipay, and even data that is synchronized from other places. Other user data, you should be very familiar with these business data.
Also, you need to be very familiar with the interfaces that the systems depend on. This is mainly to figure out your SUT's call to the dependent system, which call request parameters are legal? Which are illegal? What possible data or responses will the relying party send back to you, preferably with a standardized interface document (unfortunately, it is scarce now).
You also need to have a deeper understanding of the data types of other input methods. For the system I am responsible for, mainly the first two aspects, of course, it varies according to different system conditions, and this has to be analyzed in detail.
Secondly, when all the possible values of x are determined, the professional test case design method is used here to design the combination of x1 to xn. Design methods include: equivalence class, boundary value, causal diagram, decision table, orthogonal method, etc. These are described in detail in many software testing books, and I will not make a detailed table here. Those who are interested can refer to them by themselves.
3. How are x1 to xn passed to the SUT?
This is precisely described in one word as "driving". How to drive your testing process? In fact, this reflects the level of test engineers. If you use manual testing, you must build a test environment, make your SUT run, and then preset various data and call your process. If you are testing automation, there are actually two ways here:
deploying the system under test, simulating the client to send requests to drive;
directly relying on the code of the system under test and driving it by calling from local code.
In general, there are two ways of "driving", which are independent of language and common to all languages:
code-driven. There is nothing to say about this, it is to directly rely on the code of your SUT and drive it by calling the code directly.
Protocol driven. This also includes some RPC calls in a broad sense, mainly http, https, ftp, telnet, hsf, dubbo, etc.
6. Proposition of the concept of the test system
As mentioned above, the steps of the test work are:
determine the combined data of x1 to xn, input
each group of data into the SUT
, and determine the expected output results y1' to yn after inputting each group of input data according to the requirements '
Compare the expected results with the actual results y1, y2, ..., yn to draw a conclusion
We analyze the output of the above steps. Points 1 and 3 produce "data", and points 2 and 4 produce "logic". The logic of point 4 is very fixed, that is, the comparison logic between the expected output result and the actual output result, while the "driving" logic of point 2 has code-driven and protocol-driven, and there are many kinds of protocol-driven, but because of the involved It is the interface and protocol, which is relatively fixed and not easy to change. It is very suitable for making 2 and 4 applications. Let's imagine that if there is a test system that can complete the test of various SUTs according to the data passed to it, wouldn't it mean that test engineers only need to output data (test cases). The idea is completely feasible, because the test case is essentially a "description," a description of "what kind of data to use, what process to call, and what results are expected to be produced". This description can be in Chinese, English, xml format, or script, as long as the semantics can be clearly described, but we definitely need to formulate some format specifications for this description to ensure the testing system. be able to recognize this description. In this way, our test system can integrate use case management, test execution, bug submission, and test report, and become a perfect turn of the test center (I don't know if the word is correct). I boldly draw a schematic diagram of the architecture of this test system:
I am currently doing some research in this area, and there are some substantial outputs (driver module and use case management module), but it has yet to be perfected, if anyone is interested, welcome Discuss together.
Seven, the core value of testers People
often ask, what are the core values of testers? What is the difference between PD and development? If you don't go through the above series of analysis, it's really hard to answer when someone is bombed. But today is different. Today I will talk about the core values of the testers I know. These are the things that every test leader must know, otherwise how do you give direction to your team's growth and answer questions for your team members?
The core value of testing is: for any system under test, bugs can be avoided comprehensively and efficiently—finding, locating, and solving. Note that there are four elements here: any system under test, comprehensive, efficient, bug
1. Element one: any system under test
The diversity of the system may confuse your eyes, just as people are often easily lost in this colorful world, unable to realize what is really worth pursuing and what is real wealth. With the above analysis as a foreshadowing, this is very simple, in fact, it is to solve the "driving problem". There are always people who are afraid of the construction of the test framework and the construction of the test environment. If you understand this principle, you will become indomitable. There are only two kinds of drivers. , but we have already seen the direction of the beacon, are we still afraid?
2. Element 2: Comprehensive
This is actually the design problem of test cases. This has been analyzed very clearly and will not be repeated. Please refer to the settings of the combination data of x1, x2,...,xn above.
3. Element 3: Efficiency
This is mainly reflected in three aspects: data preparation services, automated testing, and test maintenance and inheritance.
At present, the most done and the most mature is the data preparation service. Basically, each product line has its own data preparation tools, such as data factory, TAP, etc.
Automated testing is also the main means to improve testing efficiency, but manual testing cannot be completely replaced. Automated testing has its applicable scenarios: manual testing cannot be performed; functions are stable and not easy to change; frequent regressions. Even if it cannot be fully automated, it is necessary to find a way to semi-automate it. In short, the conditions allow us to automate, but the conditions do not allow us to create conditions to automate, and everything that can be done by the computer must not be done by people, so the degree of automation also reflects the ability level of a test engineer.
The maintenance and inheritance of testing, this is the place where labor is most likely to be wasted. "I would rather rewrite all the code than change other people's code" is a common problem for engineers. For development engineers, this problem is better. After all, you can't open an application alone, and you have to change it in the original application, but for test engineers This problem is particularly acute. The independence of test scripts determines that the styles of automation scripts written by each person are different. Once a person is replaced, later people can write their own scripts and will not maintain scripts written by others. You can also reuse and extend the code you wrote yourself, but it is also difficult for others to reuse your code, and then replace the code and continue the vicious circle. In the final analysis, there is no unified plan for test scripts, not only no unified style, but also no unified structure. It is indeed necessary and necessary to formulate some script coding standards, plan the structure of test scripts, and make test scripts maintainable and reproducible. Useful, extensible, and precipitate some test services for test use. On the other hand, people who have just graduated are writing scripts, and those who have been working for five or six years are also writing scripts. If you don’t believe me, look at them, there is still a big gap between the scripts written by the two. The person who just wrote the script will put all the logic in a testcase, and a veteran will have a certain architectural awareness, the abstraction of the abstract, the encapsulation of the package. Therefore, the unified planning of test scripts also provides a growth direction for new testers, which is conducive to the rapid growth of new testers. Another idea is to use the "test system" mentioned above to solve this problem. Everyone only needs to write use cases according to fixed specifications, and hand over the test execution to the system. This should be the most perfect solution to the inheritance problem, but The premise is that the "test system" needs to be stable and powerful enough.
4. Element 4: Bug
What is a bug? Anything that doesn't meet expectations can be called a bug. Therefore, bugs are also generalized bugs, which can be divided into functional bugs, performance bugs, security bugs, and even process bugs. For a bug, a good test engineer should be able to locate the cause of the bug and give a solution.
Not much to say about functional bugs, where most of the test engineer's time is spent. The main method of bug positioning is to look at the log and debug.
For non-functional bugs, it's a bit more complicated and can't be generalized, but there are ways. For example, in the performance test, if the program is found to be stuck, you will guess whether there is a thread deadlock. For java applications, you need to use some jvm tools to view the thread stack and make judgments based on the thread status. As long as you master some non-functional bug positioning methods, there are traces to follow, and finally you can do it with ease. The location and resolution of bugs test not only the technical depth of testers, but also the breadth of knowledge, so this is also an important aspect of judging the ability level of a test engineer.
In addition, for some process problems, the test is the communication and coordination ability of test engineers. Because it is really difficult, the dominant power is in PD and development. As the last link of the test, sometimes it is really necessary to use some communication skills and cultivated personality charm to persuade and advance.
8. The nature of testing positions
To sum up, testing is the last part of software quality control, and the quality of testing directly determines the quality of software. In history, there are many cases of heavy losses due to poor testing: for example, program bugs have caused huge losses, do you want to shoot programmers? At the same time, testing is also a supporting position. Although it does not directly produce code, the requirements for testers are not only not low, but also very high. Many test experts in the industry first become developers and later The logic of turning it into a test is very simple, because after a person's ability reaches a very high level, how to copy his ability to others becomes a knowledge. The easiest and most direct way is to evaluate other people's code and point out Other people's code, architecture problems. The test is a simple entry, the more difficult it is, and even the final requirements for people are extremely demanding. Test management is also very difficult to do. In reality, everyone is responsible for different needs. It is difficult for you to judge the pros and cons of two test engineers, because the depth of testing is reflected in ideas. Maybe you can learn from test cases. Find some clues above, or look for it from communication, or use it as a reference from the bugs found, or find it from the faults generated online.
Nine, talk about the status quo of testing
Testing is a job that is easily misunderstood. The complexity and comprehensiveness of the testing work itself determines that the growth of testers is not as fast as the development and PD of single-dimensional operations, so that many people have misunderstood the testing position, and also Can't blame the occasional "do we need full-time QA" war of words that many testers themselves start to wonder about. This is caused by the lack of understanding of the nature of the test. The test is a bit like practicing Neijiaquan, and it is difficult to cultivate. Some people even can’t get in after three years of practice. This can’t be blamed for so many people who quit halfway through and few people with firm beliefs. . In a word, too few people understand the test. There are also many departments that force testers to become developers. Is it true? I have never doubted the value of testing, and I firmly believe that testing cannot be cut off. How about those who are forced to convert testing into development, how is the product quality after conversion? How many of them are doing testing work under the title of development? Of course, I have not investigated in detail, and those who know can talk about it.
The development of the testing work requires a standardized cooperation process. For the development process that is not strictly managed, the development of the testing work appears to be constrained everywhere. Ali is a result-oriented company, and many teams neglect the management of the process: project delays have no impact on performance; as long as there are no major faults online, even small faults continue to have no impact on performance; The big deal is to roll back. In such an environment, the quality awareness of development has also reached a low point, all kinds of reviews are omitted, all kinds of reviews are not called tests, all kinds of development are finished and come to test to verify, all kinds of compression test time, even I have encountered projects There is no test plan in the manager's project plan, and the development is still unwilling to test (because it can't pass the smoke), coupled with the advocacy of development self-test, development can completely bypass the test, test it by yourself, release the code online, If there is a problem, let's find the test regression again. According to historical experience, there have been several serious major failures, most of which are caused by bypassing tests or developing self-tests.
10. Test and life
Isn't life like this, just imagine what we understand very well in life? A lot of things are a "black box" to us, you can't understand why, but you know how to use it. Do you know the principles of Chinese medicine? Our ancestors have not used it to cure diseases for thousands of years; people are also a "black box", how do you know what kind of people are around you? It is not through many things in daily life to test, understand him, let you make close friends, let you know people and make good use of them, and lead a good team; when the CEO chooses a successor, he will definitely let the candidate go through different departments, through good times and bad times. Various tests are conducted to examine their performance in different environments, and finally determine whether to let them take the top position; after the year-end performance is completed, submit it to the big boss for approval. How does the big boss approve it? This performance evaluation form has not been tested through the internal correlation of each set of indicators, the overall proportion and other dimensions. There are so many examples of this, life is full of tests, and we are all testers.
I love testing, testing has taught me a lot, it has taught me business, taught me technology, and more importantly, it has taught me the ability to think about the overall situation, the way of thinking about the root cause of things, and how to manage well , taught me how to behave.