Entropy-based stopping criterion #151
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alliepiper
requested changes
Jan 8, 2024
alliepiper
reviewed
Jan 10, 2024
alliepiper
reviewed
Jan 11, 2024
alliepiper
approved these changes
Jan 12, 2024
1 task
7 tasks
rapids-bot bot
pushed a commit
to rapidsai/rapids-cmake
that referenced
this pull request
Mar 4, 2024
This PR bumps the nvbench version to fetch the latest feature added in NVIDIA/nvbench#151 which should largely reduce nvbench runtime. Authors: - Yunsong Wang (https://github.com/PointKernel) Approvers: - Robert Maynard (https://github.com/robertmaynard) URL: #549
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #150 and #147.
This PR adds new command line option
--stopping-criterion <criterion>with two predefined criteriastdrelandentropyalong with API for customization of the stopping criterion. Thenvbench/examples/custom_criterion.cuillustrates how custom criteria can be added on per-run basis. This opens possibilities for performance CI improvements. One can now develop criteria that, for instance, collects large sample, store the sample size and then on each re-run of performance CI loads this number, leading to better stability.Apart from new API,
entropycriterion is introduced. To enable it, it's sufficient to write--stopping-criterion entropy. The criterion computes cumulative entropy of the sample and stores it in an entropy window. Then, linear regression on the cumulative entropy window is computed. If the angle of the linear regression is small enough and coefficient of determination (R^2) is large enough, criterion believes that new samples will not introduce any new information and the sample is representative. Entropy criterion addresses concerns from #150 and #147 as well as significantly reduces variation of sample size, which is important for performance CI. Below is a plot of sample size distribution forstdrelandentropycriteria collected onnvbench/examples/throughput.cuthat illustrates this point:Below is an example where
stdrelnoticed small variance and decided to stop, butentropynoticed that entropy grows and kept sampling, discovering new modes:Other times,
entropynotices that new measurements do not introduce anything new to the sample and stops earlier:Each criterion has its own set of parameters. Parameters like
--max-noiseand--min-timeonly affectstdrelcriterion, whereas--max-angleand--min-r2are parameters ofentropy.For now,
stdrelstays as default criterion. Decision on switching the default criterion will be made after some field experience.