mutate: assess assertions in test code

[christoffer-nylen]

Assume

• The user assesses test quality by ensuring that each test kills some mutant.
• The user has written a test case that contains both correct and incorrect assertions.
• When running mutation testing, some assertions kill mutants.

Problem: Incorrect assertions will not be detected.

Example 1: Incorrect test evaluation

EXPECT_EQ(some_actual_value_returned_from_the_sut, some_expected_value_set_by_the_test); // OK
EXPECT_EQ(some_expected_value_set_by_the_test, some_expected_value_set_by_the_test); // NOT OK

Example 2: Incorrect pre-condition evaluation

ASSERT_EQ(some_pre_condition, some_pre_condition); // NOT OK

Example 3: Overgenerelized test code

if (some_cond) {
  EXPECT_EQ(...); // Is this executed?
}
if (some_other_cond) {
  EXPECT_EQ(...); // Is this executed?
}
if (some_third_cond) {
  EXPECT_EQ(...); // Is this executed?
}

Example 4: Complex test code

for (cond){
  for (cond) {
    if (some_cond) {
      if (some_other_cond) {
        if (some_third_cond) {
          EXPECT_EQ(...); // Is this executed?
        } else
          EXPECT_EQ(...); // Is this executed?          
        }
      }
    }
  }
}

Example 5: Test code that can be removed

// TODO: Remove depricated EXPECT_EQ when feature X is removed
Feature_X = getDepricatedFeature();
if (Feature_X.active == True){
  EXPECT_EQ(...); // Is this executed?
}

Suggestion:

Add functionality that lets the user verify that each assertion in the test is able to detect a bug.