#747: 9th order prolongation test fails with REAL_PRECISION=4 ---------------------+------------------------------------------------------ Reporter: hinder | Owner: eschnett Type: defect | Status: new Priority: major | Milestone: Component: Carpet | Version: Resolution: | Keywords: ---------------------+------------------------------------------------------
Comment (by hinder):
I looked up numeric_limits, and according to http://www.cplusplus.com/reference/std/limits/numeric_limits, numeric_limits<T>::min() is of type T and is the minimum positive value that T can take. i.e. it would be 1.17549435e-38F for single precision and 2.2250738585072014e-308 for double precision. Do you mean numeric_limits<T>::epsilon() instead? This would be 1.19209290e-07F for single precision and 2.2204460492503131e-16 for double precision. eps^3/4^ would be correspondingly 6.4e-6 and 1.8e-12, which sound more like what we want. In the code, with single precision, I see:
INFO (CarpetLib): numeric_limits<RT>::min() = 1.1920928955078125e-07 INFO (CarpetLib): good::abs (res - y0) = 1.52587890625e-05
Is this what you mean by borderline? Should we use eps^4/7^? This gives (1.19209290e-07)^(4/7)^ = 1.1e-4 as the tolerance for single precision and 1.13e-9 for double precision.