[inayatku]

Fixes #370.

The example in #370 (COBYLA on (2 - cos(x0) + x1^2)^2 from (0, 0)) never returns with maxeval 668 or more, even with maxtime set; with maxeval 667 it returns. The cause is a nan step in trstlp.

When rho becomes very small, trstlp can compute stpful = dd / (temp + sd) as 0/0 = nan. There is a check after it from 2010:

if (nlopt_isinf(step)) return NLOPT_ROUNDOFF_LIMITED;

It does not catch nan. Then, at the end of the iteration, step == stpful is false for nan, so the code falls through to L480, "switches to stage two" again and repeats this forever. No function evaluation happens in this loop, so maxeval and maxtime are never checked. I found this by counting the passes through the labels in the quadratic example below: one trstlp call went through L480 more than 10^7 times with step and stpful equal to nan and rho = 7.5e-155 (so rho*rho is subnormal).

In other cases the nan step leaves trstlp, and cobylb evaluates the objective at nan points until maxeval. With the six-hump camel function on [-3,3] x [-2,2] from (0.5, 0.5) and maxeval 1000, master calls the objective 32 times with x = (nan, nan) after evaluation 690.

This PR makes the check !nlopt_isfinite(step), so a nan step also returns NLOPT_ROUNDOFF_LIMITED, as an infinite step already did. This is also the code COBYLA returns when rho reaches rhoend = 0.

Results with this change:

• COBYLA freezes (though maxeval and maxtime are given) #370 example: maxeval 668 and 2000 both stop after 668 evaluations with roundoff-limited and f = 1 (the minimum). maxeval 667 is unchanged.
• (x0-1)^2 + 10 (x1-2)^2 from (0, 0) with default tolerances and maxeval 2000: master never returns; with this change it stops after 844 evaluations at (1, 2).
• testopt with COBYLA on all 22 objectives, 3 seeds and 4 stopping rules (264 runs): 216 runs are identical. In the other 48 runs (objectives 14, 16, 17, 18, all 2-D), master runs to the evaluation limit of 1000 and returns NLOPT_MAXEVAL_REACHED; with this change the run stops after 692 to 725 evaluations with NLOPT_ROUNDOFF_LIMITED and the same minimum. These are the runs where master evaluated at nan points.

So users who get NLOPT_MAXEVAL_REACHED in such a case will get NLOPT_ROUNDOFF_LIMITED instead, which the C++ and Python interfaces raise as an exception; the best point and value are the same.

New test test/t_cobyla_nan_step.cxx runs the two examples above. On master it does not return. ctest with MSVC 2022 on Windows: 78 of 78 tests pass.

The reproducer in the 2020 comment on #370 calls a modified internal trstlp with rho about 1e6, so it may be a different cycle; I did not test it.