Example 2: Adverse Conditions
This web page presents details specific to running tests in adverse conditions. A general explanation of the screens is given on Example 1: Ordinary Execution.
To run a test in adverse conditions using the basic algorithm, select the algorithm Yes, all events in same, and one or more event types. This will trigger each of the selected event types at every injection site that is reached.
Note: some event types are experimental. For testing the life-cycle, use Pause-Resume, Pause-Stop-Restart, or Pause-Stop-Destroy-Create.
The image below shows the result of executing the test testCalculatorPlus in adverse conditions (specifically, the Pause-Stop-Destroy-Create event, which is often triggered during device orientations).
Contrary to in ordinary test executions (see Example 1: Ordinary Execution), the test fails.
By clicking on the test in the aggregation overview report (above), an individual test report incl. the specific failure is shown.
In this case, an assertion checking that 2+2=4 failed, because the Pause-Stop-Destroy-Create event caused the calculator display to be reset (as can be seen from the snapshot).
From the above image it can be seen that testCalculatorPlus lead to two test executions with ID's aggressive-0 and aggressive-4, respectively. (Hence the significant increase of execution time, compared to Example 1: Ordinary Execution: 19 seconds vs 7 seconds.)
This is because Thor re-runs tests in case of failures, potentially revealing additional failures later in the same test (see the paper "Systematic Execution of Android Test Suites in Adverse Conditions").
The mode can be disabled by checking the Stop on error option.
Here, the ID's are interpreted as follows:
-
aggressive-0: adverse conditions were injected from injection site #0 -
aggressive-4: adverse conditions were injected from injection site #4
From this we can conclude that testCalculatorPlus fails due to an injection of Pause-Stop-Destroy-Create at injection site #0, #1, #2, or #3 (or multiple of them). A specific injection site that causes the problem can be detected by Failure Isolation.
A number of other algorithms also trigger adverse conditions:
-
Yes, all events in different: Triggers the selected event type in all injection sites, but performing only one injection per test case, and hence cause a test case with X injection sites to be executed X times. -
Yes, single: Triggers the select event type in the specified injection site.
When executing many tests, a lot of redundant event injections are potentially being performed. The redundancy reduction extension is a heuristic that attempts to ignore injections that are redundant (see "Systematic Execution of Android Test Suites in Adverse Conditions"), and can be enabled by checking the Abstract state cache option.
The default state abstraction is the option Using view hierarchy, that uses an abstraction of the view hierarchy for comparing states.
For comparison, it is also possible to rely on actual screenshots (experimental), by checking Using bitmap.
Snapshots (see also Example 1: Ordinary Execution) are helpful for debugging in adverse conditions, since it is easy to identify what the impact of an injection was (in cases where the effect of an injection is manifested on the UI).
When triggering adverse conditions, edges corresponding to the injected events are also present in the graph. Injection that cause the UI to change will be red, and those that leave the UI unchanged will be grey.
Hence, red edges indicate potential problems due to adverse conditions, and are a good source for debugging tests that fail in adverse conditions.
