ArgIntervalExcl #303
Comments
|
Looks like an overflow of statrs/src/distribution/gamma.rs Line 118 in 41555a1 (we know that This is not nice from an API perspective.. maybe add |
|
A different error is thrown if either of the parameters are invalid, so they are definitely both positive. But (presumably owing to my gradients being wrong) both parameters are infinitesimally small. So while the gamma function does instantiate, the inverse_cdf can't handle it. I'm sure it's only in my trial-and-error with something else that I've run into it. As it's possibly unlikely to be encountered in any real applications, it's probably not a high priority bug. |
|
What would the behavior be if As alternative to a failure, we can look at a way to expose gamma function evaluating these cases,
The underflow implementation could naively use the lowest order that has all the terms separately without underflowing multiply |
Ah, I missed that possibility. That same term can also underflow, which should mean that the result (=param passed to gamma_lr) is |
|
This opens up a bit of a question on how to handle cases like this. Is it unreasonable to expect that the API will handle special cases like this? Catching over/underflows like this may not be trivial. E.g. it's evidently not enough to check
because
is not enough because |
In a general sense, I think a human directly evaluating the density and distribution functions should be a little bit more wary or willing to take fault out of misunderstanding, but evaluation at fairly unpredictable points but a numerical iteration scheme for fitting or root finding would run across these cases. I think since we have checking on constructing the distribution, then an I'd expect evaluating a function near a finite boundary of its support would be considered "reasonable input" - my expectation is that there are usually ways one can implement a "small input" regime with approximations to the function that are more numerically stable so I'd expect them to work for normal floats near a function's edge of support at zero. Specific to Gamma, Maybe the question is where to reject constructions of "unreasonably" small beta? Then we can do some numerical tricks to ensure that reasonable accuracy for normal float input to trait methods. After all, in terms of the cumulative distribution function, the scale parameter is exactly a scale, and it's easy to argue that if a scale parameter is to be that small, maybe the user should consider moving some of that scale into the "units" of the random variable so that an iterative search scheme has some freedom to hop around inputs. They'd need to know that a search is being used internally. Would probably require similar changes to checks in other distribution constructors. |
|
@rm-minus-r-star was this underflow issue what helped you identify the optimizer was incorrectly implemented? |
No, I was writing debugging messages to stdout in the closures that I fed into the optimiser, and could see it making much bigger leaps in parameter values than I thought it should ... into the negative range. I then implemented taking the .exp() of the parameter inputs to keep them positive, and set the starting parameters to the correct answer. My debugging output on tests tracks them moving away from those correct initial parameters, all the way down into the infinitesimally small range. So actually it's the same behaviour from the optimiser for reasons I still can't explain -- the optimiser keeps pushing the parameters it knows more and more negative, which become infinitesimally small upon taking the .exp(). in my closure functions. Whereas the negative parameter case was caught appropriately at the point of setting the function ("Bad distribution parameters"), the .exp()/underflow case of panics subsequently when the .inverse_cdf() function assumes success with .unwrap(). |
|
At the moment, I'm leaning toward rejecting values of beta too small at ctor. As we should require that If this is done well - for phenomena that are not multi-scale - then x should be I'm trying to think of a counterexample where I have appropriate units and extreme values of beta are still useful. Obtaining one and discussing where this change affects it would be useful. |
|
I've run into this again, and this time I'm not quite sure why. My function that draws on the statrs crate is thread 'tokio-runtime-worker' panicked at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/statrs-0.17.1/src/function/beta.rs:101:31: I guess there are reasons why the beta distribution is being called, but I can't find any pathological |
|
The repeatability concern immediately raises a flag in my head about memory. What platform are you running this on and are you using musl or glibc for floating point? Suggestions:
|
|
I did a quick dive into this. Now, what is In your code snippet, you set The "variable-inserted" code (see StudentsT::cdf) would be let k = (x - 0.0) / 1.0;
let h = self.freedom / (self.freedom + k * k);I think that even with FP math and its weirdness, this should always be in
EDIT: This should not be as big of a surprise to users as it is. |
|
NaN is a tricky topic and it doesn't have a well defined role within defensive programming. Scipy has a NaN policy that can be set globally or at the call site. There are different stages at which the policy could be set, least granular is as a compilation option, the policy would be on the consumption of NaN, so we could still return NaN and that could allow us to identify bugs in statrs. More granular is to offer more functions at the API,
Are you, @FreezyLemon, wanting to move toward enforcing a policy consistency or do you want to figure out a policy we should strive for first? |
|
I'm not sure... it might make sense to talk about NaN in general. |
I guess I've hit an edge case that was never expected to fail ...
While trying to (I think) get my gradients right in optimization_engine to find a best-fit gamma distribution under the statrs crate, I get
The RUST_BACKTRACE=full is
The text was updated successfully, but these errors were encountered: