-
Notifications
You must be signed in to change notification settings - Fork 27
NaN error with high dropout rate
Using the default dropout rate range, 0.1 to 0.9, when doing the hyperparameters search, will sometimes results in NaN values in the network weights and output. 0.9 as the maximum dropout rate is too large, which may result in a layer dropping all its units in a forward pass, causing a NaN value in training. In older version of lifelines, NaN values are incorrectly converted to True when computing c-index. Hence masking the error. However, since version 0.21.0, such errors will be raised, breaking the demo script.
Therefore to reproduce the 4Dsurvival paper, where dropout search range was 0.1 to 0.9, and NaN values were converted to True, use lifelines version of 0.14.6. This corresponds to docker image lisurui6/4dsurvival-gpu:1.0
.
To further develop the code, we would suggest to change the dropout range to 0.1 to 0.5, and use the latest lifelines version (0.21.0+). This corresponds to docker image lisurui6/4dsurvival-gpu:1.1
.