Skip to content

Warn when newtonMemoryBudget leaves a Laplace fit on coordinate descent - #118

Merged
msuchard merged 1 commit into
developfrom
newton-budget-warning
Nov 19, 2026
Merged

msuchard merged 1 commit into
developfrom
newton-budget-warning

Conversation

@msuchard

Copy link
Copy Markdown
Member

Summary

Under algorithm = "auto" (or "newton"), a Laplace fit takes prox-Newton steps only while their dense work fits createControl(newtonMemoryBudget = ): 24·|W|² for the working-set Hessian plus 12 bytes per entry of the working-set columns (proxFitsDense()). When the budget is exceeded, the fit falls back to coordinate descent without any notice. On correlated designs that can be far slower.

A 500,000-person Cox censoring model from CloneCensorWeight (1.2·10⁸ covariate values, so about 1.4 GB of prox-Newton work):

newtonMemoryBudget solver result
1024 MB (default) coordinate descent 1,700 cycles in 25 min, not converged
4096 MB prox-Newton 24 iterations, 105 s

At 10⁶ persons with 8192 MB it took 24 iterations and 187 s.

This PR makes that fallback visible:

  • proxFitsDense() records the largest prox-Newton work it refuses in the convergence report, exported as fit$convergence$budgetBytes (NA when nothing was refused). It is also added to the CyclopsApi convergence numbers.
  • fitCyclopsModel() warns when the budget left the fit without any Newton step (budgetBytes finite and newtonIterations == 0). The warning gives the memory needed and the budget, and points to createControl(newtonMemoryBudget = ). It follows warnings = FALSE.
  • The createControl() docs and NEWS item 42 are updated.

Fits that take Newton steps, and fits whose budget is enough, are unchanged; no results change.

Tests

  • test-proxNewton.R: the existing tiny-budget test now expects the warning and checks budgetBytes. The default-budget fit has no warning and budgetBytes is NA. warnings = FALSE silences the warning.
  • The full suite (NOT_CRAN=true, in the package namespace) has 1,924 expectations and 0 failures. Eight test files error at load (bootstrap, conditionalPoisson, dataConversion{Stratified,Unstratified}, fisherEvents, smallCLR, smallFineGray, weighting). They error the same way on unmodified develop 4306300 on the same machine, so the cause is the environment, not this change.

🤖 Generated with Claude Code

proxFitsDense() records the largest prox-Newton work it refuses
(24|W|^2 + 12 bytes per working-set entry) in the convergence report,
exported as fit$convergence$budgetBytes. fitCyclopsModel() warns when
that left the fit without any Newton step, naming the memory needed and
the budget. Such fits can crawl: a 500,000-person CCW Cox censoring
model ran 1,700 coordinate-descent cycles in 25 minutes unconverged
under the 1024 MB default, and converged in 24 iterations and 105 s
with newtonMemoryBudget = 4096.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.90909% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 85.15%. Comparing base (4306300) to head (30f631a).

Files with missing lines Patch % Lines
R/ModelFit.R 75.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #118      +/-   ##
===========================================
+ Coverage    85.13%   85.15%   +0.02%     
===========================================
  Files           73       73              
  Lines        15765    15774       +9     
===========================================
+ Hits         13421    13433      +12     
+ Misses        2344     2341       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@msuchard
msuchard merged commit 22bffdb into develop Nov 19, 2026
26 checks passed
@msuchard
msuchard deleted the newton-budget-warning branch November 19, 2026 03:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant