Repository navigation
Add Taylor linearization and R-parity features, validated against the R survey package - #330
Open
ayushpatnaikgit wants to merge 40 commits into
Open
ayushpatnaikgit wants to merge 40 commits into
ayushpatnaikgit wants to merge 40 commits into
Conversation
… could be removed.
…hed to the footnotes.
Co-authored-by: Nadia <76887318+nadiaenh@users.noreply.github.com>
Co-authored-by: Nadia <76887318+nadiaenh@users.noreply.github.com>
Co-authored-by: Nadia <76887318+nadiaenh@users.noreply.github.com>
Co-authored-by: Nadia <76887318+nadiaenh@users.noreply.github.com>
Co-authored-by: Nadia <76887318+nadiaenh@users.noreply.github.com>
Co-authored-by: Nadia <76887318+nadiaenh@users.noreply.github.com>
Co-authored-by: Nadia <76887318+nadiaenh@users.noreply.github.com>
Co-authored-by: Nadia <76887318+nadiaenh@users.noreply.github.com>
Added co-author
JuliaCon Proceedings Author's Guide says we should only edit these 3 files because other files might be overwritten later: `paper.tex` `ref.bib` `paper.yml`
Co-authored-by: Nadia <76887318+nadiaenh@users.noreply.github.com>
added Shikhar Co-authored-by: Ayush Patnaik <ayushpatnaik@gmail.com>
Add paper for JuliaCon proceedings
Add Iulia's name
The CairoMakie versions allowed by the current compat bounds fail to precompile on Julia 1.12 (removed Core.TypeName.mt internal), which broke every workflow before tests could run. Pin to 1.11 until the Makie stack is upgraded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # src/Survey.jl # src/mean.jl # src/quantile.jl # test/quantile.jl # test/reg.jl # test/runtests.jl # test/total.jl
The domain estimation examples printed standard errors that are mathematically zero, so their floating-point noise differed between architectures and failed the docs build on CI. Use stype domains (all meaningfully nonzero) and round the post-stratified totals instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Preview deployment from pull_request events fails with the repository's deploy key (Permission denied), while regular push deploys work. PR runs now build the docs and doctests and skip deployment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR closes most of the everyday-usage gap with R's
surveypackage and adds a testsuite that validates every deterministic estimate against reference values computed by
R (survey 4.5), to ~1e-9 relative tolerance.
Taylor linearization (the foundation)
SurveyDesignestimators now compute standard errors by Taylor linearization — thedefault variance method of R's
svydesign— including finite population correctionwhen the design is constructed with
popsize, and support for stratified one-stagecluster designs with nested PSU ids (validated on NHANES):
mean,total,ratioonSurveyDesignreturnSEcolumns matchingsvymean/svytotal/svyratioto machine precisionmean(:y, :domain, design), etc.) now computes SEs via scorevariables on the full design, matching
svyby. The previous subset-based approachdropped PSUs without domain members and produced incorrect design-based SEs.
degf(design)— degrees of freedom (PSUs − strata), as in RNew features
mean(:stype, d)/total(:stype, d)svymean/svytotalon a factor (proportions/counts)var(:y, d),std(:y, d)svyvarconfint(est),confint(est; dof=degf(d))confint,confint(..., df=degf(d))svytable(d, :a, :b)svytablesvyttest(:y, :group, d)svyttestsvychisq(d, :a, :b)(:Fand:Chisq)svychisq(Rao–Scott)quantile(:y, d, p; ci=true)oldsvyquantile(..., ci=TRUE)(Woodruff CIs)trimweights(d; lower, upper, strict)trimWeightspoststratify(d, :var, population)postStratifyrake(d, margins, populations)rakebrrweights(d; fay_rho)as.svrepdesign(type="BRR"/"Fay")poststratifyandrakeon aReplicateDesignadjust every set of replicate weightsseparately (as in R), so replicate SEs fully reflect the calibration. On a plain
SurveyDesignthe adjusted weights match R exactly, but linearized SEs treat them asfixed (conservative) — documented in the docstrings.
BRR uses a Sylvester Hadamard matrix by default (identical to R's choice for the
tested stratum counts) and accepts a
hadamard_matrixargument to reproduce R for anystratum count. Fay's method is a new
BRRReplicatesinference type.Validation against R
test/r_reference/generate_fixtures.Rcomputes reference values with R survey 4.5and writes them to CSVs (committed), so CI needs no R installation. Regenerate with
Rscript generate_fixtures.R.test/r_comparison.jl— 527 assertions across 16 testsets. Since theapi,scdand
nhanesdatasets shipped here are identical to R's, deterministic results(linearized SEs, jackknife, BRR with matching Hadamard matrix, weights from
trimming/post-stratification/raking) agree to machine precision.
Behaviour changes
mean/total/ratioonSurveyDesigngained anSEcolumn.oldsvyquantile(ties="discrete", linearinterpolation of the inverse weighted CDF): e.g. the apisrs median is now 658
(R's answer) instead of 659.
Fixes along the way
test/total.jl,test/reg.jl) thatfail on Julia 1.11 because the RNG stream changed across Julia versions — verified
pre-existing on a clean checkout; tolerances corrected.
Documenter.doctest.current compat bounds fail to precompile on Julia 1.12 (this broke CI before any
tests ran, independent of this PR's content). Upgrading the Makie stack is left as
a follow-up.
Note: this branch also contains the working-paper commits (
paper/), which predatethe feature work.
🤖 Generated with Claude Code