Feature/symbolic explanations #101

sarah-segel · 2024-01-03T15:35:35Z

Plugins

Add symbolic explanations plugin (New plugin: Add symbolic explanations #46).

Enhancements

Update author email.
Add an exit button that first deletes running jobs and then terminates DeepCave (otherwise jobs such as the symbolic regression would continue running after DeepCave termination).
Nicer handling of Keyboard Interrupt (avoid long error message in terminal).
Disable debug mode (to avoid error messages being shown directly in the UI, especially upon exit of DeepCave).

Bug-Fixes

Fix varying configspace index order in PDP and symbolic explanations.

Others

Disabled existing loggers again to suppress redis logs

…/symbolic-explanations

helegraf

Looks nice overall and the tests are working 💪 I have some minor remarks. It would also be nice if you could run the pre-commit hooks so everything is formatted properly and address the errors in the new files (some minor things like missing docstring I think).

Also is it correct that I am getting DEBUG console output without changing any settings?

Furthermore, I am running into some problems running the plugin but maybe I am using it wrong. When I select the smac-pendigits run, the plugin does not seem to produce any explanations.

Server Log

SymbolicExplanations (DEBUG): Button pressed.
SymbolicExplanations (DEBUG): Enqueued SMAC-pendigits-015-0-0 (69666054159540351ed1f6fbf825f2cd).
RandomForestSurrogate (DEBUG): RandomForestSurrogate: Initial Seed = 0
RandomForestSurrogate (DEBUG): RandomForestSurrogate: Final Seed = 248400791
PDP (DEBUG): PDP: Initial Seed = 0
PDP (DEBUG): PDP: Final Seed = 1543538891
ICE (DEBUG): ICE: Initial Seed = True
ICE (DEBUG): ICE: Seed = Use existing
ICE (INFO): Recalculating ICE...
    |   Population Average    |             Best Individual              |
---- ------------------------- ------------------------------------------ ----------
 Gen   Length          Fitness   Length          Fitness      OOB Fitness  Time Left
   0    12.62          181.416       11         0.014005              N/A     17.39s
   1     8.73          20.5379       11         0.014005              N/A     14.85s
   2     7.70          0.39384       10         0.014005              N/A     12.92s
   3     4.63           0.2092       10         0.014005              N/A     11.07s
   4     2.15         0.086137        3        0.0140051              N/A      8.63s
   5     2.02        0.0345978        2        0.0140079              N/A      6.89s
   6     2.04        0.0389469        2        0.0140079              N/A      5.37s
   7     2.03        0.0395785        2        0.0140079              N/A      3.46s
   8     2.05        0.0408348        2        0.0140079              N/A      1.72s
   9     2.03         0.039717        2        0.0140079              N/A      0.00s
SymbolicExplanations (DEBUG): Job 69666054159540351ed1f6fbf825f2cd-48ad708a7a708995d8fdf94b34a2dca3 for run_id 69666054159540351ed1f6fbf825f2cd.
SymbolicExplanations (DEBUG): Job 69666054159540351ed1f6fbf825f2cd-48ad708a7a708995d8fdf94b34a2dca3 cached.
SymbolicExplanations (DEBUG): Job 69666054159540351ed1f6fbf825f2cd-48ad708a7a708995d8fdf94b34a2dca3 deleted.

helegraf · 2024-02-09T10:10:32Z

Makefile

@@ -2,7 +2,7 @@
 # are usually completed in github actions.

 SHELL := /bin/bash
-VERSION := 1.1.3
+VERSION := 1.2


This says version 1.2, but in the Changelog the change is under Version 1.1.4

Edited to 1.2 in Changelog.

helegraf · 2024-02-09T10:14:02Z

deepcave/__init__.py

 description = "An interactive framework to visualize and analyze your AutoML process in real-time."
 url = "automl.org"
 project_urls = {
    "Documentation": "https://automl.github.io/DeepCAVE/main",
    "Source Code": "https://github.com/automl/deepcave",
 }
 copyright = f"Copyright {datetime.date.today().strftime('%Y')}, {author}"
-version = "1.1.3"
+version = "1.2"


Same as the previous comment about inconsistent versions

Same as above.

helegraf · 2024-02-09T10:15:10Z

docs/plugins/symbolic_explanations.rst

+
+This plugin is capable of answering similar questions as the Partial Dependencies plugin, i.e.:
+
+* How does the objective change wrt one or two hyperparameters? For example, does the accuracy


I would just write "with respect to"

Changed that in all cases of wrt

helegraf · 2024-02-09T10:18:01Z

docs/plugins/symbolic_explanations.rst

+Symbolic Explanations
+====================
+
+This plugin is capable of answering similar questions as the Partial Dependencies plugin, i.e.:


I personally think an explanation of what symbolic explanations are in the first sentence would be best; so just adding one more sentence before this one.

Added a sentence in the beginning.

helegraf · 2024-02-09T10:26:48Z

deepcave/cli.py

@@ -1,5 +1,4 @@
-from typing import Any, List
-
+import os


os is unsused

helegraf · 2024-02-09T10:27:18Z

deepcave/config.py

@@ -49,6 +49,9 @@ def PLUGINS(self) -> Dict[str, List["Plugin"]]:
        from deepcave.plugins.budget.budget_correlation import BudgetCorrelation
        from deepcave.plugins.hyperparameter.importances import Importances
        from deepcave.plugins.hyperparameter.pdp import PartialDependencies
+        from deepcave.plugins.hyperparameter.symbolic_explanations import (
+            SymbolicExplanations,
+        )


Why the brackets? :)

Not sure how they got there, removed :)

helegraf · 2024-02-09T10:29:34Z

deepcave/utils/symbolic_regression.py

+    try:
+        # Simplification can fail in some cases. If so, use the unsimplified version.
+        symb_simpl = sympy.simplify(symb_conv)
+    except:


Is there a specific exception that could be caught here instead?

I unfortunately didn't track that and cannot find an example now, thus I just changed it to actually print the exception (so if I should see one at some point, I can edit it).

helegraf · 2024-02-09T10:31:44Z

deepcave/utils/symbolic_regression.py

+
+def convert_symb(symb, n_decimals: int = None, hp_names: list = None) -> sympy.core.expr:
+    """
+    Convert a fitted symbolic regression to a simplified and potentially rounded mathematical expression.


This line is over 100 characters; however, I am not 100% sure what our desired setting for the linter would be. 100 is admittedly really short.

Not sure either, but I think most parts of the code would be fine with 100. I limited the length here.

helegraf · 2024-02-09T10:32:21Z

deepcave/utils/symbolic_regression.py

+    return function_set
+
+
+def convert_symb(symb, n_decimals: int = None, hp_names: list = None) -> sympy.core.expr:


Type of symb?

helegraf · 2024-02-09T10:45:15Z

deepcave/plugins/hyperparameter/symbolic_explanations.py

+                                [
+                                    dbc.Label("Parsimony coefficient"),
+                                    help_button(
+                                        "Penalizes the complexity of the resulting formulas."


Maybe add a line if higher or lower will mean less complexity

True, added.

sarah-segel · 2024-02-16T10:43:27Z

Looks nice overall and the tests are working 💪 I have some minor remarks. It would also be nice if you could run the pre-commit hooks so everything is formatted properly and address the errors in the new files (some minor things like missing docstring I think).

Also is it correct that I am getting DEBUG console output without changing any settings?

Furthermore, I am running into some problems running the plugin but maybe I am using it wrong. When I select the smac-pendigits run, the plugin does not seem to produce any explanations.

Server Log

SymbolicExplanations (DEBUG): Button pressed.
SymbolicExplanations (DEBUG): Enqueued SMAC-pendigits-015-0-0 (69666054159540351ed1f6fbf825f2cd).
RandomForestSurrogate (DEBUG): RandomForestSurrogate: Initial Seed = 0
RandomForestSurrogate (DEBUG): RandomForestSurrogate: Final Seed = 248400791
PDP (DEBUG): PDP: Initial Seed = 0
PDP (DEBUG): PDP: Final Seed = 1543538891
ICE (DEBUG): ICE: Initial Seed = True
ICE (DEBUG): ICE: Seed = Use existing
ICE (INFO): Recalculating ICE...
    |   Population Average    |             Best Individual              |
---- ------------------------- ------------------------------------------ ----------
 Gen   Length          Fitness   Length          Fitness      OOB Fitness  Time Left
   0    12.62          181.416       11         0.014005              N/A     17.39s
   1     8.73          20.5379       11         0.014005              N/A     14.85s
   2     7.70          0.39384       10         0.014005              N/A     12.92s
   3     4.63           0.2092       10         0.014005              N/A     11.07s
   4     2.15         0.086137        3        0.0140051              N/A      8.63s
   5     2.02        0.0345978        2        0.0140079              N/A      6.89s
   6     2.04        0.0389469        2        0.0140079              N/A      5.37s
   7     2.03        0.0395785        2        0.0140079              N/A      3.46s
   8     2.05        0.0408348        2        0.0140079              N/A      1.72s
   9     2.03         0.039717        2        0.0140079              N/A      0.00s
SymbolicExplanations (DEBUG): Job 69666054159540351ed1f6fbf825f2cd-48ad708a7a708995d8fdf94b34a2dca3 for run_id 69666054159540351ed1f6fbf825f2cd.
SymbolicExplanations (DEBUG): Job 69666054159540351ed1f6fbf825f2cd-48ad708a7a708995d8fdf94b34a2dca3 cached.
SymbolicExplanations (DEBUG): Job 69666054159540351ed1f6fbf825f2cd-48ad708a7a708995d8fdf94b34a2dca3 deleted.

Regarding the debug logging, I changed the plugin (and other) specific logging settings in logging.yml to INFO.

For the pre-commit hooks, I edited the files I added similar to the changes in the pre-commit branch.

Regarding the problem with the smac-pendigits run, this seems to be related to the problem I ran into with one of Daphne's runs (#111). I added the example there.

@helegraf would be great if you could re-check if everything is addressed :)

alexandertornede

Suggested some small adaptations that might be nice.

alexandertornede · 2024-02-19T08:28:53Z

deepcave/utils/logging.yml

  handlers: [console]
-disable_existing_loggers: false
+disable_existing_loggers: true


What is the reason for us setting this to true?

alexandertornede · 2024-02-19T08:29:26Z

docs/plugins/configurations.rst

@@ -13,7 +13,7 @@ Since configurations are used throughout the application, you might find links a
 plugin. This plugin is capable of answering following questions:

 * Where is the configuration coming from?
-* How are the objective values wrt the budgets?
+* How are the objective values with respect to the budgets?


What are the ....

…lanations

Sarah Krebs added 22 commits September 8, 2023 16:33

Draft for symbolic explanations plugin

35eea9b

Format

aded257

Filter out categorical HPs for symbolic explanations

252ea68

Handling sympy timeouts

a510db8

Resolve merge conflict

e7db918

Resolve merge conflict

b1a5a97

Merge branch 'development' of github.com:automl/DeepCAVE into feature…

72df8cd

…/symbolic-explanations

Solve merge conflict

6a47a7f

Improvements for symbolic explanations plugin

60580c5

Add exit button and delete jobs on exit

4f2636e

Update author email

803cfc3

Add symb expl docs and handle long formulas

cd5b1c5

Edit exit callbacks

1da5915

Disable debug mode

d56b82f

Update changelog

1e75d3c

Remove signal runtime alarm from symbolic conversion

87b02bd

Update version number

205f672

Update changelog

95473b6

Reset inputs for symb expl when changing runs

d8bb6a7

Edit symb expl input settings

77e2246

Fix varying configspace index order in PDP and symbolic explanations

7777934

Allow longer expression

b3723aa

helegraf mentioned this pull request Jan 18, 2024

Error message overflow #100

Closed

Sarah Krebs added 5 commits January 23, 2024 10:39

Add pdp plot to symbolic explanation output

933a137

Add pdp plot to symbolic explanation output and add paper to docs

b62462e

Solve merge conflict

4252994

Disable existing loggers again to suppress redis logs

e7b2e9c

Solve merge conflict

ab0164a

sarah-segel requested a review from helegraf February 9, 2024 08:10

Revert incomplete fix of configspace index order in pdp and symb. expl.

208ebfa

helegraf reviewed Feb 9, 2024

View reviewed changes

Sarah Krebs added 3 commits February 16, 2024 08:41

Resolve merge conflict

04b660a

Fixes from review

da987e8

Fixes from review

bc8bfd4

Fixes from review

d4101f6

alexandertornede approved these changes Feb 19, 2024

View reviewed changes

sarah-segel merged commit e6c1f2e into development Feb 21, 2024
2 checks passed

sarah-segel deleted the feature/symbolic-explanations branch February 21, 2024 08:48

github-actions bot pushed a commit that referenced this pull request Feb 21, 2024

Sarah Segel: Merge pull request #101 from automl/feature/symbolic-exp…

fa5246b

…lanations

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/symbolic explanations #101

Feature/symbolic explanations #101

sarah-segel commented Jan 3, 2024 •

edited

Loading

helegraf left a comment

helegraf Feb 9, 2024

sarah-segel Feb 16, 2024

helegraf Feb 9, 2024

sarah-segel Feb 16, 2024

helegraf Feb 9, 2024

sarah-segel Feb 16, 2024

helegraf Feb 9, 2024

sarah-segel Feb 16, 2024

helegraf Feb 9, 2024

sarah-segel Feb 16, 2024

helegraf Feb 9, 2024

sarah-segel Feb 16, 2024

helegraf Feb 9, 2024

sarah-segel Feb 16, 2024

helegraf Feb 9, 2024

sarah-segel Feb 16, 2024

helegraf Feb 9, 2024

sarah-segel Feb 16, 2024

helegraf Feb 9, 2024

sarah-segel Feb 16, 2024

sarah-segel commented Feb 16, 2024

alexandertornede left a comment

alexandertornede Feb 19, 2024

alexandertornede Feb 19, 2024


		This plugin is capable of answering similar questions as the Partial Dependencies plugin, i.e.:

		* How does the objective change wrt one or two hyperparameters? For example, does the accuracy

		return function_set


		def convert_symb(symb, n_decimals: int = None, hp_names: list = None) -> sympy.core.expr:

Feature/symbolic explanations #101

Feature/symbolic explanations #101

Conversation

sarah-segel commented Jan 3, 2024 • edited Loading

Plugins

Enhancements

Bug-Fixes

Others

helegraf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sarah-segel commented Feb 16, 2024

alexandertornede left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sarah-segel commented Jan 3, 2024 •

edited

Loading