mutate: handling unproductive mutants

[christoffer-nylen]

Background

The user wants to generate mutants that enforce effective testing. Some mutants are killable but unproductive. They draw attention from the productive mutants.

Example 1 - Logging

Assume:

• There's a project Foo with 19494 LoC.
• Foo contains 359 invocations of a function addLoggMessage(param, param) that is used for debugging purposes.
• The addLoggMessage usage (e.g., invokation order and contents of argument) is highly coupled to the Foo implementation.
• The correctness of the addLoggMessage() usage is not critical -> The user does not intend to write tests to verify its correctness.

Problem:

• The 359 reported addLoggMessage(param, param) statements draw attention from critical problems.

Evaluation criteria

The below suggestions will be evaluated to the following criteria:

Criterion	Priority (TBD)	Motivation (TBD)
Efficiency: Is the solution quick or tedious for the user to use? Does it increase/decrease complexity to the tool usage?	High	?
Lock-in: Does the solution couple the source code to a specific tool?	Low	Code annotations can easily be changed later if need be.
Readability: Does the solution advance code quality or knowledge? Does it add noise?	Medium	?
Complexity: Is the solution simple or hard to implement and maintain?	?	?

Suggested solutions

The suggested solutions are based on the below comments.

Approach 0: Do nothing

Use //NOMUT annotations to suppress each addLoggMessage() invokation.

Pros:

• Complexity: Nothing needs to be implemented.

Cons:

• Efficiency: The user needs to put //NOMUT to 359 places.
• Lock-in: Tool specific annotations couples the project to a specific tool.
• Readability: Some code maintainers might argue that this adds "noise" to the code.

Meta Approach 1: Line suppression

Example:
dextool mutate admin --suppress 52

Pros:

• Complexity: Relatively simple to implement?
• Lock-in: No need to add annotations to the code base.
• Readability: No need to add "noise" to the code.

Approach 1.1: Line suppression + No automation

Cons:

• Efficiency: The user needs to input 359 lines.

Approach 1.1: Line suppression + User script

• Update readme.md with information how to write a script that detects lines containing some unproductive mutant (for example addLoggMessage).
• Use the script to suppress the lines.

Pros:

• Efficiency: The user only needs to write a small script with a grep command etc.

Cons:

• Efficiency: When and how does the user integrate the script? This adds "many" steps to readme.md that might be perceived as complicated?

Meta Approach 2: Filter

• Add some kind of filtering function so that the user can ignore all recognized patterns.

Pros:

• Efficiency: Only needs to be specified once.
• Lock-in: No need to update the code base.
• Readability: No need to add "noise" to the code.

Cons:

• TODO

Approach 2.1: Function call filter

Adding something like this to the config-file:

# Specify which functions you would like to exclude from the mutation testing
# Note: this will affect all kinds of mutants surrounding those functions.
# For example: SDL won't remove the call to the function, or the function-body.
suppress-functions = ["addLoggMessage", "print", "randomFunctionToSuppress"]

Pros:

• Simple syntax

Cons:

• Complexity: ?

Approach 2.2: Super expressive filter

TODO: Update background example to motivate the need for a more expressive filter.

Cons:

• Complexity: ?

[joakim-brannstrom]

Yes this is coupled to how to improve NOMUT. Suppressing mutants.
I have had the idea to let the user specify uninteresting functions.

[joakim-brannstrom]

For now what exist is to mark those lines with NOMUT

[peatersson]

The idea I've started implementing, but put on hold, was that the user specified mutantID.
Example:
dextool mutate admin --suppress 52

I realized that this quickly became problematic in terms of scale. A user cant be expected to write that commando for every single mutant, if there are 359 as in your example.

I know I wouldn't.

Are we maybe thinking of an addition to the config-file? Something like:

# Specify which functions you would like to exclude from the mutation testing
# Note: this will affect all kinds of mutants surrounding those functions.
# For example: SDL won't remove the call to the function, or the function-body.
suppress-functions = ["addLoggMessage", "print", "randomFunctionToSuppress"]

And then we add the ability to run admin-mode and suppress those now added to the config?

[joakim-brannstrom]

The idea I've started implementing, but put on hold, was that the user specified mutantID.

Did you consider suppressing a whole line?

Example:
dextool mutate admin --suppress 52

I realized that this quickly became problematic in terms of scale. A user cant be expected to write that commando for every single mutant, if there are 359 as in your example.

I agree. Another problem is that they constantly change. The checksum is more stable.

Are we maybe thinking of an addition to the config-file? Something like:

# Specify which functions you would like to exclude from the mutation testing
# Note: this will affect all kinds of mutants surrounding those functions.
# For example: SDL won't remove the call to the function, or the function-body.
suppress-functions = ["addLoggMessage", "print", "randomFunctionToSuppress"]

Yes, in that direction. But probably have to allow the user to be more precis?
This is why I think it is hard to do. How to specify the mutants in a sane way that the user wont cry rivers over?
A line number is kind a easy. A bit brute force but doable.

And then we add the ability to run admin-mode and suppress those now added to the config?

Shouldn't need to run in admin-mode. Being in the config file should be enough.

[peatersson]

Did you consider suppressing a whole line?

Yes, I started with that. But I remembered (maybe wrongly) that the user who requested this said that maybe not ALL mutants on a row needed to be suppressed. I therefore went with the mutantID-solution first. But as I said, it was put on hold early in the phase.

I agree. Another problem is that they constantly change. The checksum is more stable.

Oh definitely.

@christoffer-nylen Extremely good edits on the issue! :)

[peatersson]

User feedback:
Working with inspected code makes it difficult to use current NOMUT. The ability of marking mutants by ID in a command is a valid option.

I am thinking that I can start with implementing this, so that it is atleast possible to do something. We can then evaluate and improve it with more options in the config-file, for example marking certain function-calls.

[joakim-brannstrom]

I discussed this with a user. How to use this to mark a mutant as unproductive in a pull request.
The simplest way we could come up with is to use the configuration file in the PR to mark it in the database.
If the approach we are talking about here makes that possible then we have another tool for our box to realize "CI mode".

[joakim-brannstrom]

#946 changes how mutants are calculated which is one outcome of the discussions on this issue.

[peatersson]

#953 intend to solve parts of this problem by allowing the user to mark mutants manually by id via an admin command. There is obvious issues with this approach, one of them being that MutationId is very unstable and changes when the analyze phase is run again.

This was however a request from a user, so atleast providing the ability to do this increases short term usability. From here, we can work towards a more complete solution or alternative (both can still be available in the tool).