Repository navigation
Warn when newtonMemoryBudget leaves a Laplace fit on coordinate descent - #118
Merged
Merged
Conversation
proxFitsDense() records the largest prox-Newton work it refuses (24|W|^2 + 12 bytes per working-set entry) in the convergence report, exported as fit$convergence$budgetBytes. fitCyclopsModel() warns when that left the fit without any Newton step, naming the memory needed and the budget. Such fits can crawl: a 500,000-person CCW Cox censoring model ran 1,700 coordinate-descent cycles in 25 minutes unconverged under the 1024 MB default, and converged in 24 iterations and 105 s with newtonMemoryBudget = 4096. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #118 +/- ##
===========================================
+ Coverage 85.13% 85.15% +0.02%
===========================================
Files 73 73
Lines 15765 15774 +9
===========================================
+ Hits 13421 13433 +12
+ Misses 2344 2341 -3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Under
algorithm = "auto"(or"newton"), a Laplace fit takes prox-Newton steps only while their dense work fitscreateControl(newtonMemoryBudget = ): 24·|W|² for the working-set Hessian plus 12 bytes per entry of the working-set columns (proxFitsDense()). When the budget is exceeded, the fit falls back to coordinate descent without any notice. On correlated designs that can be far slower.A 500,000-person Cox censoring model from CloneCensorWeight (1.2·10⁸ covariate values, so about 1.4 GB of prox-Newton work):
newtonMemoryBudgetAt 10⁶ persons with 8192 MB it took 24 iterations and 187 s.
This PR makes that fallback visible:
proxFitsDense()records the largest prox-Newton work it refuses in the convergence report, exported asfit$convergence$budgetBytes(NAwhen nothing was refused). It is also added to theCyclopsApiconvergence numbers.fitCyclopsModel()warns when the budget left the fit without any Newton step (budgetBytesfinite andnewtonIterations == 0). The warning gives the memory needed and the budget, and points tocreateControl(newtonMemoryBudget = ). It followswarnings = FALSE.createControl()docs and NEWS item 42 are updated.Fits that take Newton steps, and fits whose budget is enough, are unchanged; no results change.
Tests
test-proxNewton.R: the existing tiny-budget test now expects the warning and checksbudgetBytes. The default-budget fit has no warning andbudgetBytesisNA.warnings = FALSEsilences the warning.NOT_CRAN=true, in the package namespace) has 1,924 expectations and 0 failures. Eight test files error at load (bootstrap, conditionalPoisson, dataConversion{Stratified,Unstratified}, fisherEvents, smallCLR, smallFineGray, weighting). They error the same way on unmodified develop 4306300 on the same machine, so the cause is the environment, not this change.🤖 Generated with Claude Code