Copy States Optimization

[Angeladadd]

Description

This PR introduces a two-phase optimisation to address communication bottlenecks in the copy_states routine during distributed resampling:

1. Deduplication & Local Replication:

• Transmit each unique particle only once across ranks.
• Reconstruct duplicates locally with multi-threaded replication.

2. Communication-Efficient Redistribution:

• Reformulates resampling redistribution as a lightweight rank-level transportation problem.
• Minimises unnecessary cross-rank transfers caused by global index ordering.
• Solved efficiently with HiGHS, adding negligible overhead relative to communication savings.

Issue

#116

Testing

• Added integration test 7 for covering the cases using optimised copy states and resampling function
• Added mpi test for optimised copy states function
• Added unit test for optimised resampling function
• Added slurm scripts for
  • running end-to-end run_particle.jl
  • running mpi test for optimised copy states function

[codecov]

Codecov Report

❌ Patch coverage is 96.59091% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.64%. Comparing base (99b7804) to head (b4cc2ea).
⚠️ Report is 59 commits behind head on main.

Files with missing lines	Patch %	Lines
src/utils.jl	96.51%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #297      +/-   ##
==========================================
+ Coverage   94.49%   94.64%   +0.14%     
==========================================
  Files           9        9              
  Lines         654      728      +74     
==========================================
+ Hits          618      689      +71     
- Misses         36       39       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tkoskela]

Hi Chenge, I hope you are doing well. Sorry it's taken so long to review your code after marking the report. I think this is very good work and should definitely get merged into the upstream repo.

I'd like to have a few clarifying comments in some places that I've highlighted in my review comments. I think overall the optimisations you made should be the default behaviour, rather than something the user has to switch on. This would especially simplify the copy_states! and copy_states_dedup! functions that duplicate some code at the moment.

If Matt and Mose could also take a look at this, that would be great!

[tkoskela]

Something I noticed while going through the GitHub Actions logs. Can we suppress the output from HiGHS in tests? The output from julia-runtest is long

[matt-graham]

Thanks @Angeladadd - this looks great in general. I've added a couple of minor comments / suggestions, but these are not blockers.

[matt-graham]

Thanks @Angeladadd for the further updates. Just to say from my perspective this looks good to merge but will let @tkoskela and @giordano a chance to also look over things

[tkoskela]

This looks good to me now! Thanks for all your hard work @Angeladadd