Skip to content
This repository has been archived by the owner on Jan 6, 2022. It is now read-only.

How to change measure and resampling and maximize kappa value? #6

Open
GegznaV opened this issue Jul 28, 2017 · 2 comments
Open

How to change measure and resampling and maximize kappa value? #6

GegznaV opened this issue Jul 28, 2017 · 2 comments

Comments

@GegznaV
Copy link

GegznaV commented Jul 28, 2017

In a HyperControl object, is there a way to change either measure or resampling (or both) and leave other settings at their defaults?

This code results in error:

makeHyperControl(resampling = cv3,  measures   = kappa)
## Error in makeHyperControl(resampling = cv3, measures = kappa) : 
## Assertion on 'mlr.control' failed: Must have class 'TuneControl', but has class 'NULL'.

And manual modification of fields result in unexpected outcome:

set.seed(1)
# Create the object:
h_ctrl <- generateHyperControl(task = sonar.task, learner = "classif.ranger")
# Manually modify the fields:
h_ctrl <- makeHyperControl(h_ctrl$mlr.control,
                           resampling = cv3, 
                           measures   = kappa,
                           par.config = h_ctrl$par.config)
# Tune
h_params <- hyperopt(sonar.task,
                     learner = "classif.ranger",
                     hyper.control = h_ctrl,
                     show.info = FALSE)

h_params
## Op. pars: mtry=1; min.node.size=2
## kappa.test.mean=-0.64

It seems that it was searched for the minimum kappa value

@GegznaV GegznaV changed the title How to change measure and resampling? How to change measure and resampling and maximize *kappa* value? Jul 28, 2017
@GegznaV GegznaV changed the title How to change measure and resampling and maximize *kappa* value? How to change measure and resampling and maximize kappa value? Jul 28, 2017
@GegznaV
Copy link
Author

GegznaV commented Jul 28, 2017

One more example with iris dataset:

h_ctrl <- generateHyperControl(task = iris.task, learner = "classif.ranger")

set.seed(1)
h_ctrl <- makeHyperControl(h_ctrl$mlr.control,
                           resampling = cv3, 
                           measures   = kappa,
                           par.config = h_ctrl$par.config)
h_params <- hyperopt(iris.task,
                     learner = "classif.ranger",
                     hyper.control = h_ctrl,
                     show.info = FALSE)
set.seed(1)
h_ctrl2 <- makeHyperControl(h_ctrl$mlr.control,
                           resampling = cv3, 
                           measures   = mmce,
                           par.config = h_ctrl$par.config)

h_params2 <- hyperopt(iris.task,
                     learner = "classif.ranger",
                     hyper.control = h_ctrl2,
                     show.info = FALSE)
h_params
## Tune result:
## Op. pars: mtry=4; min.node.size=5
## kappa.test.mean=-0.938

h_params2
## Tune result:
## Op. pars: mtry=1; min.node.size=7
## mmce.test.mean=0.0533

"Good" mmce value, but "bad" kappa for the same task seems suspicious.

@jakob-r
Copy link
Owner

jakob-r commented Jul 31, 2017

Thanks for spotting that bug. I have to fix it in mlr. But it is just an output bug. Measures that are to be maximized are transformed with -1 because the optimizer always minimizes.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants