Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
98 commits
Select commit Hold shift + click to select a range
99dc47f
Add dry-run memory resources for allocation profiling without real me…
achirkin Feb 18, 2026
695a8a3
First batch of dry-run guards
achirkin Feb 18, 2026
42d8ad4
Dry run compliance for raft::linalg namespace
achirkin Feb 19, 2026
6db7ec8
Update developer guide with the dry run protocol
achirkin Feb 19, 2026
d91a1c6
BREAKING CHANGE: replaced pinned_container with host_container using …
achirkin Feb 19, 2026
1a114f6
Dry run compliance for raft::matrix namespace
achirkin Feb 19, 2026
dec5e95
Dry run compliance for raft::random namespace
achirkin Feb 19, 2026
f84d9a9
Dry run compliance for raft::solver namespace
achirkin Feb 19, 2026
44793cd
Dry run compliance for raft::sparse namespace
achirkin Feb 20, 2026
d566fe9
Dry run compliance for raft::spectral namespace
achirkin Feb 20, 2026
fc3bde6
Dry run compliance for raft::stats namespace
achirkin Feb 20, 2026
b0ddbc8
Add a little bit more tests
achirkin Feb 20, 2026
15c07a1
Add the Dry Run Protocol Overview
achirkin Feb 20, 2026
1c57abb
Fix C++ example in the docs
achirkin Feb 20, 2026
d916b45
Merge branch 'main' into fea-dry-run-protocol
achirkin Feb 20, 2026
9d24480
Add a few more tests and fix a missed CUDA call in QR algorithm
achirkin Feb 20, 2026
7577e56
Fix excess subsample doing work in dry run
achirkin Feb 20, 2026
99faf68
Add dry run compliance to the raft::copy on mdspans
achirkin Feb 20, 2026
b859894
Merge branch 'main' into fea-dry-run-protocol
achirkin Feb 20, 2026
57d4c19
Revert changing includes from public to detail namespace to avoid bre…
achirkin Feb 23, 2026
694ec63
Merge branch 'main' into fea-dry-run-protocol
achirkin Feb 23, 2026
a2dd18c
Merge rapidsai/main into fea-dry-run-protocol
achirkin Feb 24, 2026
45e2d49
Rename device_uvector_policy -> device_container_policy and add non-i…
achirkin Feb 26, 2026
65d4570
Declare the new resources in raft handle
achirkin Feb 26, 2026
d86638f
Renamed managed policy
achirkin Feb 26, 2026
d6788f6
Add raft::resources for pinned and managed resources and the type-era…
achirkin Feb 26, 2026
e7bea48
Updated container policies
achirkin Feb 26, 2026
2514621
All but host memory resource are done
achirkin Feb 26, 2026
49735a5
Simplify the implementation
achirkin Feb 27, 2026
22b4048
Make the host container policy use the resource concept
achirkin Feb 27, 2026
557cc8c
Settle down with raft::mr::*et_default_host_resource()
achirkin Feb 27, 2026
cc7a4b0
Add some thread-safety
achirkin Feb 27, 2026
e77fe2a
Merge branch 'main' into fea-unify-memory-resources
achirkin Feb 27, 2026
8922b8f
Merge branch 'main' into fea-dry-run-protocol
achirkin Feb 27, 2026
866211e
C++17 backwards-compatibility
achirkin Feb 28, 2026
c171d84
Merge branch 'main' into fea-unify-memory-resources
achirkin Feb 28, 2026
268eb1b
newline
achirkin Feb 28, 2026
5c718d6
Add raft::mr::device_resource wrapper for cuda::mr::any_resource
achirkin Mar 1, 2026
c5ab9c4
Copy semantics and return resource refs
achirkin Mar 2, 2026
6af142e
Rework workspace resources to avoid nesting bridge layers
achirkin Mar 2, 2026
ece1990
Fix the argument order in tests
achirkin Mar 2, 2026
3c17e3e
Merge branch 'main' into fea-dry-run-protocol
achirkin Mar 2, 2026
4dd256b
Merge branch 'main' into fea-unify-memory-resources
achirkin Mar 3, 2026
a26357d
Add explicit conversion through cuda::mr refs to rmm ref
achirkin Mar 3, 2026
2a90680
Switch from rmm host and host_device resource reference wrappers to r…
achirkin Mar 4, 2026
59c3793
Merge branch 'main' into fea-unify-memory-resources
achirkin Mar 4, 2026
3a40d22
Prefer rmm::mr::get_current_device_resource_ref() over rmm::mr::get_c…
achirkin Mar 4, 2026
cce4f45
Remove raft pinned and managed memory resources in favor of cuda::mr …
achirkin Mar 4, 2026
fb56025
Merge branch 'main' into fea-dry-run-protocol
achirkin Mar 4, 2026
ff20962
Merge fea-unify-memory-resources into fea-dry-run-protocol
achirkin Mar 5, 2026
e76bf7c
Adapt to fea-unify-memory-resources
achirkin Mar 5, 2026
2d3f8fc
Refactor dry_run_resources as a child of raft::resources to better ke…
achirkin Mar 5, 2026
d2cf85e
Merge branch 'main' into fea-dry-run-protocol
achirkin Mar 9, 2026
e86b56d
Merge branch 'main' into fea-dry-run-protocol
achirkin Mar 14, 2026
d9a0abf
Fix style after merge commit
achirkin Mar 16, 2026
16324fb
Merge branch 'main' into fea-dry-run-protocol
achirkin Mar 18, 2026
ced0e6e
Fix merge commit typo
achirkin Mar 18, 2026
c9bf618
Merge branch 'main' into fea-dry-run-protocol
achirkin Mar 19, 2026
1acf6cf
Fix some sparse routines not being dry-run compliant
achirkin Mar 23, 2026
2326061
Unify the looks of the three custom raft::resources
achirkin Mar 24, 2026
d4ff16e
Expand test coverage Part 1
achirkin Mar 24, 2026
fee4b62
Expand test coverage Part 2
achirkin Mar 24, 2026
f1b7aca
Update docs to reflect unify memory resources PR changes
achirkin Mar 24, 2026
156a437
Fix segfault in sparse tests caused by invalid thrust exec policy
achirkin Mar 25, 2026
51c0b16
Better allocation estimates in the sparse namespace
achirkin Mar 30, 2026
c99b879
Fixing more failing tests
achirkin Apr 1, 2026
e535124
Fixing last failing tests
achirkin Apr 1, 2026
9971c71
Merge branch 'main' into fea-dry-run-protocol
achirkin Apr 1, 2026
b682d46
Fix not initialize the mdarray scalars only in dry run mode
achirkin Apr 1, 2026
69543a1
Clarify that all workspace resources are actually counted independent…
achirkin Apr 2, 2026
5db5727
Rename the dry_run_resources header file for conistency
achirkin Apr 2, 2026
d1cf594
Dry-run compliance for coo_sort
achirkin Apr 2, 2026
e47c41f
Fix the expected minimum allocation calculation
achirkin Apr 2, 2026
3d93b0f
Merge branch 'main' into fea-dry-run-protocol
achirkin Apr 3, 2026
e60a048
Merge branch 'main' into fea-dry-run-protocol
achirkin Apr 8, 2026
f8754d9
Merge branch 'main' into fea-dry-run-protocol
achirkin Apr 10, 2026
0f65503
Merge branch 'main' into fea-dry-run-protocol
achirkin Apr 13, 2026
969b868
Merge remote-tracking branch 'rapidsai/main' into fea-dry-run-protocol
achirkin Apr 22, 2026
217ca58
Fix tests after rmm breaking change
achirkin Apr 22, 2026
e72e872
Store the device resources by values to safely keep them alive while …
achirkin Apr 23, 2026
1c4deb8
Switch to owning semantics for both host and per-device resources
achirkin Apr 24, 2026
8fdf194
Don't let allocations cross dry-run/normal scopes
achirkin Apr 29, 2026
1a7501c
More thorough tests for bitset/bitmap in dry run mode
achirkin Apr 29, 2026
65dda3b
Merge branch 'main' into fea-dry-run-protocol
achirkin May 6, 2026
a710e97
make bitset.count() dry-run-compliant
achirkin May 7, 2026
3a65638
Merge branch 'main' into fea-dry-run-protocol
achirkin May 7, 2026
7f1210e
Merge branch 'main' into fea-dry-run-protocol
achirkin May 11, 2026
592b8e0
Merge branch 'main' into fea-dry-run-protocol
achirkin May 15, 2026
0f5641e
Merge branch 'main' into fea-dry-run-protocol
achirkin Jun 10, 2026
205cffa
Make randomized SVD dry run compliant (adopting new features to dry run)
achirkin Jun 10, 2026
ec1a794
Merge branch 'main' into fea-dry-run-protocol
achirkin Jun 15, 2026
8d56307
Merge remote-tracking branch main into fea-dry-run-protocol
achirkin Jun 17, 2026
a9bc0af
Follow up on merge commit
achirkin Jun 17, 2026
91107c3
Merge branch 'main' into fea-dry-run-protocol
achirkin Jun 23, 2026
fa24967
Merge branch 'main' into fea-dry-run-protocol
achirkin Jul 6, 2026
8e1855d
Fix style
achirkin Jul 6, 2026
584cac8
Merge branch 'main' into fea-dry-run-protocol
achirkin Jul 7, 2026
a0ee878
Revert changes to .clang-format
achirkin Jul 7, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion cpp/include/raft/core/bitset.cuh
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* SPDX-FileCopyrightText: Copyright (c) 2023-2026, NVIDIA CORPORATION.
* SPDX-FileCopyrightText: Copyright (c) 2023-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
* SPDX-License-Identifier: Apache-2.0
*/

Expand All @@ -10,6 +10,7 @@
#include <raft/core/device_container_policy.hpp>
#include <raft/core/device_mdarray.hpp>
#include <raft/core/operators.hpp>
#include <raft/core/resource/dry_run_flag.hpp>
#include <raft/core/resources.hpp>
#include <raft/linalg/map.cuh>
#include <raft/linalg/reduce.cuh>
Expand Down Expand Up @@ -166,6 +167,8 @@ void bitset_view<bitset_t, index_t>::repeat(const raft::resources& res,
index_t times,
bitset_t* output_device_ptr) const
{
// Only a copy and kernel run below this point.
if (resource::get_dry_run_flag(res)) { return; }
constexpr index_t bits_per_element = sizeof(bitset_t) * 8;

if (bitset_len_ % bits_per_element == 0) {
Expand Down
19 changes: 12 additions & 7 deletions cpp/include/raft/core/bitset.hpp
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* SPDX-FileCopyrightText: Copyright (c) 2024-2026, NVIDIA CORPORATION.
* SPDX-FileCopyrightText: Copyright (c) 2024-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
* SPDX-License-Identifier: Apache-2.0
*/

Expand All @@ -8,6 +8,7 @@
#include <raft/core/detail/macros.hpp>
#include <raft/core/device_container_policy.hpp>
#include <raft/core/device_mdarray.hpp>
#include <raft/core/resource/dry_run_flag.hpp>
#include <raft/core/resource/thrust_policy.hpp>
#include <raft/core/resources.hpp>
#include <raft/util/integer_utils.hpp>
Expand Down Expand Up @@ -133,9 +134,11 @@ struct bitset_view {
auto count_gpu_scalar = raft::make_device_scalar<index_t>(res, 0.0);
count(res, count_gpu_scalar.view());
index_t count_cpu = 0;
raft::update_host(
&count_cpu, count_gpu_scalar.data_handle(), 1, resource::get_cuda_stream(res));
resource::sync_stream(res);
if (!resource::get_dry_run_flag(res)) {
raft::update_host(
&count_cpu, count_gpu_scalar.data_handle(), 1, resource::get_cuda_stream(res));
resource::sync_stream(res);
}
return count_cpu;
}

Expand Down Expand Up @@ -408,9 +411,11 @@ struct bitset {
auto count_gpu_scalar = raft::make_device_scalar<index_t>(res, 0.0);
count(res, count_gpu_scalar.view());
index_t count_cpu = 0;
raft::update_host(
&count_cpu, count_gpu_scalar.data_handle(), 1, resource::get_cuda_stream(res));
resource::sync_stream(res);
if (!resource::get_dry_run_flag(res)) {
raft::update_host(
&count_cpu, count_gpu_scalar.data_handle(), 1, resource::get_cuda_stream(res));
resource::sync_stream(res);
}
return count_cpu;
}
/**
Expand Down
6 changes: 3 additions & 3 deletions cpp/include/raft/core/coo_matrix.hpp
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@

/*
* SPDX-FileCopyrightText: Copyright (c) 2023-2026, NVIDIA CORPORATION.
* SPDX-FileCopyrightText: Copyright (c) 2023-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
* SPDX-License-Identifier: Apache-2.0
*/
#pragma once
Expand Down Expand Up @@ -180,8 +180,8 @@ class coordinate_structure : public coordinate_structure_t<RowType, ColType, NZT
void initialize_sparsity(nnz_type nnz)
{
sparse_structure_type::initialize_sparsity(nnz);
c_rows_.resize(nnz);
c_cols_.resize(nnz);
c_rows_.reallocate(nnz);
c_cols_.reallocate(nnz);
}

protected:
Expand Down
6 changes: 3 additions & 3 deletions cpp/include/raft/core/csr_matrix.hpp
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* SPDX-FileCopyrightText: Copyright (c) 2023-2026, NVIDIA CORPORATION.
* SPDX-FileCopyrightText: Copyright (c) 2023-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
* SPDX-License-Identifier: Apache-2.0
*/
#pragma once
Expand Down Expand Up @@ -189,8 +189,8 @@ class compressed_structure
void initialize_sparsity(NZType nnz) override
{
sparse_structure_type::initialize_sparsity(nnz);
c_indptr_.resize(this->get_n_rows() + 1);
c_indices_.resize(nnz);
c_indptr_.reallocate(this->get_n_rows() + 1);
c_indices_.reallocate(nnz);
}

protected:
Expand Down
7 changes: 6 additions & 1 deletion cpp/include/raft/core/detail/copy.hpp
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* SPDX-FileCopyrightText: Copyright (c) 2023-2026, NVIDIA CORPORATION.
* SPDX-FileCopyrightText: Copyright (c) 2023-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
* SPDX-License-Identifier: Apache-2.0
*/

Expand All @@ -11,6 +11,7 @@
#include <raft/core/host_mdspan.hpp>
#include <raft/core/logger.hpp>
#include <raft/core/mdspan.hpp>
#include <raft/core/resource/dry_run_flag.hpp>
#include <raft/core/resource/stream_view.hpp>
#include <raft/core/resources.hpp>

Expand Down Expand Up @@ -399,6 +400,10 @@ mdspan_copyable_t<DstType, SrcType> copy(resources const& res, DstType&& dst, Sr
RAFT_EXPECTS(src.extent(i) == dst.extent(i), "Must copy between mdspans of the same shape");
}

// Dry-run guard: raft::copy is a pure data-movement utility with no
// allocations that callers would need tracked.
if (resource::get_dry_run_flag(res)) { return; }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I didn't realize we were adding this as a new resource. This would make it hard to use the dry-run for pre-initializing resources.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but that's fine! We can still push the initialized resources back to the original resources handle on destruction of the dry run resources

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NB: with #3052 , resources initialized in dry run mode will be automatically shared back with the input resources.


if constexpr (config::use_intermediate_src) {
#ifndef RAFT_DISABLE_CUDA
// Copy to intermediate source on device, then perform necessary
Expand Down
25 changes: 24 additions & 1 deletion cpp/include/raft/core/device_container_policy.hpp
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
/*
* SPDX-FileCopyrightText: Copyright (2019) Sandia Corporation
* SPDX-FileCopyrightText: Copyright (c) 2022-2026, NVIDIA CORPORATION.
* SPDX-FileCopyrightText: Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
* SPDX-License-Identifier: Apache-2.0 AND BSD-3-Clause
*/
/*
Expand Down Expand Up @@ -127,6 +127,29 @@ class device_uvector {

void resize(size_type size) { data_.resize(size, data_.stream()); }

/**
* @brief Resize the internal buffer without copying old data.
*
* Unlike resize(), this never copies old data.
* Thus, unlike in resize(), there's no point in time where the old and the new buffers are both
* alive, and the peak memory usage is lower.
*
* Unlike resize(), this deallocates the old buffer even if the new size is smaller.
* This ensures the memory is released promptly.
*/
void reallocate(size_type size)
{
if (size != data_.size()) {
auto stream = data_.stream();
auto mr = data_.memory_resource();
// Resize and shrink rmm::device_uvector: force deallocation without copying old data
data_.resize(0, data_.stream());
data_.shrink_to_fit(data_.stream());
// Assign a new value after the old one is deallocated
data_ = rmm::device_uvector<T>(size, stream, mr);
}
}

[[nodiscard]] auto data() noexcept -> pointer { return data_.data(); }
[[nodiscard]] auto data() const noexcept -> const_pointer { return data_.data(); }
};
Expand Down
5 changes: 3 additions & 2 deletions cpp/include/raft/core/device_mdarray.hpp
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* SPDX-FileCopyrightText: Copyright (c) 2022-2026, NVIDIA CORPORATION.
* SPDX-FileCopyrightText: Copyright (c) 2022-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
* SPDX-License-Identifier: Apache-2.0
*/

Expand All @@ -9,6 +9,7 @@
#include <raft/core/device_container_policy.hpp>
#include <raft/core/device_mdspan.hpp>
#include <raft/core/mdarray.hpp>
#include <raft/core/resource/dry_run_flag.hpp>
#include <raft/core/resources.hpp>

#include <rmm/resource_ref.hpp>
Expand Down Expand Up @@ -164,7 +165,7 @@ auto make_device_scalar(raft::resources const& handle, ElementType const& v)
using policy_t = typename device_scalar<ElementType, IndexType>::container_policy_type;
policy_t policy{};
auto scalar = device_scalar<ElementType, IndexType>{handle, extents, policy};
scalar(0) = v;
if (!resource::get_dry_run_flag(handle)) { scalar(0) = v; }
return scalar;
}

Expand Down
Loading
Loading