Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test predicting for out-of-domain time points #273

Merged
merged 1 commit into from
Dec 5, 2023
Merged

Conversation

hfrick
Copy link
Member

@hfrick hfrick commented Dec 4, 2023

closes #10

I've taken "out of domain" to mean eval time points not within the observed range of time points. This PR tests that this does cause censored to error and leaves the prediction value to the individual engines.

@hfrick hfrick requested a review from EmilHvitfeldt December 4, 2023 15:13
Copy link
Member

@EmilHvitfeldt EmilHvitfeldt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking good. Found two discrepancies in output. might not be isolated to out-of-domain time pints

fit(Surv(time, status) ~ ., data = lung)

expect_no_error(
preds <- predict(mod, obs_without_NA, type = "survival", eval_time = eval_time_obs_max_and_ood)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This prediction differs in output format than then other ones with this one being

str(preds)
tibble [1 × 1] (S3: tbl_df/tbl/data.frame)
 $ .pred: list<tibble[,2]> [1:1] 
  ..$ : tibble [2 × 2] (S3: tbl_df/tbl/data.frame)
  .. ..$ .eval_time    : num [1:2] 1022 2000
  .. ..$ .pred_survival: num [1:2] NA NA
  ..@ ptype: tibble [0 × 2] (S3: tbl_df/tbl/data.frame)
  .. ..$ .eval_time    : num(0) 
  .. ..$ .pred_survival: num(0) 

opossed to partykit which produces

str(preds)
tibble [1 × 1] (S3: tbl_df/tbl/data.frame)
 $ .pred:List of 1
  ..$ : tibble [2 × 2] (S3: tbl_df/tbl/data.frame)
  .. ..$ .eval_time    : num [1:2] 1022 2000
  .. ..$ .pred_survival: num [1:2] 0.0499 0.0499

fit(Surv(time, status) ~ ., data = lung_orsf)

expect_no_error(
preds <- predict(mod, obs_without_NA, type = "survival", eval_time = eval_time_obs_max_and_ood)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same issue as above

@hfrick
Copy link
Member Author

hfrick commented Dec 5, 2023

Thanks for highlighting that! That's not related to out-of-domain eval times, so there are two new issues: #274 and #275

@hfrick hfrick merged commit e4e2497 into main Dec 5, 2023
8 checks passed
@hfrick hfrick deleted the pred-out-of-domain branch December 5, 2023 10:42
Copy link

This pull request has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.

@github-actions github-actions bot locked and limited conversation to collaborators Dec 20, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make sure all models can handle out of domain survival probability predictions
2 participants