Some Thoughts on the automated test cases failed retries

Automated test case fails re-run will help improve the stability of automation use cases, then we look at, python and java in ecology What are the specific approach?

How to do it

If the python is ecological, the test driver to do with pytest, it may be realized by pytest plug pytest-rerunfailures failure cases with heavy run, the specific use, there are two, one is designated by the command line pytest --reruns 2 - -reruns-delay 1, reruns represents the repetition number of runs, reruns-delay represents the delay time is run repeatedly. Another way is through @ pytest.mark.flaky (reruns = 2, reruns_delay = 1), this way usually use, do not want a global all test cases are re-run, but need to run a specific test cases, it is in particular the test methods to use this mark.


If the java ecological, the test driver to do with junit, junit5 for annotations @RepeatTest (2), can try to repeat the tests or the test class methods, may be customized by implementing a TestRule interfaces, to control test operation.

public class MyTestClass {

    @Rule
    public RepeatRule repeatRule = new RepeatRule();

    @Test
    @Repeat(10)
    public void testMyCode() {
        //your test code goes here
    }
}

import static java.lang.annotation.ElementType.ANNOTATION_TYPE;
import static java.lang.annotation.ElementType.METHOD;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention( RetentionPolicy.RUNTIME )
@Target({ METHOD, ANNOTATION_TYPE })
public @interface Repeat {
    int value() default 1;
}

import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;

public class RepeatRule implements TestRule {

    private static class RepeatStatement extends Statement {
        private final Statement statement;
        private final int repeat;    

        public RepeatStatement(Statement statement, int repeat) {
            this.statement = statement;
            this.repeat = repeat;
        }

        @Override
        public void evaluate() throws Throwable {
            for (int i = 0; i < repeat; i++) {
                statement.evaluate();
            }
        }

    }

    @Override
    public Statement apply(Statement statement, Description description) {
        Statement result = statement;
        Repeat repeat = description.getAnnotation(Repeat.class);
        if (repeat != null) {
            int times = repeat.value();
            result = new RepeatStatement(statement, times);
        }
        return result;
    }
}

If there is a maven can use to add a rerunFailingTestsCount parameters, but this is to control all of the use cases.

Why should it fail to run with heavy cases
because automation usually in a test environment or other non-line environment, due to environmental instability could lead to inexplicable failure test cases, cases with stability greatly reduced. Re-run this time adding the failure mechanism to improve the stability of test cases within a certain range, make more output.

What kind of automation use cases fail to be re-run

Interface is recommended for automated testing can be added to this failure heavy run, and an interface for automated UI interfaces, failed re-run, then, think of little significance, since it is often used when patients fail, either because the interface elements are not loaded out, either is a logical example of a problem, either unexpected pop influence, this time should be allowed to throw the error as soon as possible, as quickly as possible to repair, rather than where he kept retry, futile. UI Automation should make explicit and implicit wait.

What kind of use case should be re-run defeat

In the test frame, it is best to distinguish what kind of abnormality when the abnormality service, what is the frame anomaly test itself, the service may be suitably exception retry, no abnormality to the frame re-run directly thrown. Assertion failed and certainly do not need to re-run. So when control test case execution, not all children fall into the re-run, selective, it is necessary to ensure stability, but also to ensure efficiency, allow automation play value.

to sum up

Test to be targeted, doing the right thing at the right time, the value of automated testing is because it can quickly check the system, because if the retry lead to run time multiplied, there is no sense of, not as good as thrown wrong, as soon as possible to resolve. And also automatic test run sequence control, traffic in front of the interface is possible start and, in the rear of the service interface as possible after the run. For example, under a single interface, and interfaces landing, landing interface belongs to the business front, the next single is rearward, in general, when a single test interface to be initialized state landed, this time landing will call the interface, batch execution of test cases time, you can let login interface to start and test cases, if this interface has a problem, then the other interfaces need to login with use cases will all fail, and that in such cases the latter would not run, it will save a lot of time.

Welcome everyone to my blog Chou Chou, there are more details about the test of combat Oh! !