Expectation Over Transformation Wrapper #719
Conversation
|
Great, thanks for preparing this! Would you mind adding unit tests for the new module, too? |
|
By looking at the other test files, I have added some tests to assert that wrapping the model with EoT does not change clean / robust accuracy (when no random transforms are applied). Don't know if this is correct. Also, I am not really sure how to add more complex tests (with random transformations applied to the input). |
tests/test_eot_attack.py
Outdated
There was a problem hiding this comment.
Let's rename this file to something like test_eot_wrapper.py.
tests/test_eot_attack.py
Outdated
| from conftest import ModeAndDataAndDescription | ||
|
|
||
|
|
||
| def test_eot_attack( |
tests/test_eot_attack.py
Outdated
| epsilons = (5000.0, L2(50.), Linf(1.)) | ||
|
|
||
| for attack, eps in zip(attacks, epsilons): | ||
|
|
There was a problem hiding this comment.
Remove unnecessary whitespace.
|
I have uploaded a refactored version, which should now be compatible with |
Codecov ReportAll modified and coverable lines are covered by tests ✅
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## master #719 +/- ##
=======================================
Coverage 98.46% 98.47%
=======================================
Files 56 56
Lines 3529 3546 +17
=======================================
+ Hits 3475 3492 +17
Misses 54 54 ☔ View full report in Codecov by Sentry. |
| @@ -21,3 +21,29 @@ def __call__(self, inputs: T) -> T: | |||
| y = ep.where(x < self._threshold, min_, max_).astype(x.dtype) | |||
| z = self._model(y) | |||
| return restore_type(z) | |||
|
|
|||
|
|
|||
| class ExpectationOverTransformationWrapper(Model): | |||
There was a problem hiding this comment.
Sorry for not mentioning this earlier, but can you please add docstrings to the class & module like its done for the other classes in the project?
There was a problem hiding this comment.
Added it to the models.rst file, taking the other wrapper module as an example.
However, I am not sure how to fix the version error for sphinx. In my local machine, I tried to build html docs with sphinx version 5 and it works, but the requirements.txt file in the docs folder wants:
sphinx==4.5.0
sphinx-autobuild==2021.3.14
sphinx_rtd_theme==1.0.0
sphinx-typlog-theme==0.8.0
Closes #563 and #191
My approach for implementing EoT.
I tried to follow the suggestions of @zimmerrol.
Basically, I created a new ExpectationOverTransformation Wrapper class that computes the forward pass
n_stepstimes (default to 16) and returns the average prediction.I also created a new example for pytorch
resnet-18model, which seems to work.Potential Issue: since the forward pass is executed
n_stepstimes, the attack takes a long time for execution.