Skip to content

[HRX] Export the allocation in hrx_buffer_get_device_ptr - #393

Open
Geramy wants to merge 1 commit into
ROCm:mainfrom
Geramy:fix/buffer-get-device-ptr-gpu
Open

Geramy wants to merge 1 commit into
ROCm:mainfrom
Geramy:fix/buffer-get-device-ptr-gpu

Conversation

@Geramy

@Geramy Geramy commented Aug 21, 2026

Copy link
Copy Markdown

hrx_buffer_get_device_ptr only ever tried a host mapping, so it returned HRX_STATUS_UNAVAILABLE for any device-local buffer. The comment beside it already said what was missing:

// For local-task (CPU) devices, the device pointer is available via mapping.
// For real GPU devices, this would use iree_hal_buffer_export.
// For now, map the buffer to get a usable pointer.

The consequence is that nothing needing a device address is reachable through the C API:

  • a peer copy has no source address to name;
  • hsa_amd_agents_allow_access has no allocation to grant, so device pools stay DISALLOWED_BY_DEFAULT;
  • a caller wanting hsa_amd_memory_async_copy cannot describe either end, and falls back to the HAL's blit kernels.

On an 8x gfx1201 box that last one is the difference between a copy engine and the shader cores: HIP reaches ~48 GB/s H2D and ~56 GB/s D2H on the same link where a blit-kernel copy measures about 8.

This exports the buffer through the owning device's allocator with IREE_HAL_EXTERNAL_BUFFER_TYPE_DEVICE_ALLOCATION and returns the device pointer. The existing paths are untouched and still tried first, so an already-mapped buffer and a local-task device behave exactly as before, and a failed export falls through to the mapping attempt rather than becoming an error.

Tested on gfx1201 (Radeon AI PRO R9700, ROCm 7.13): device-local buffers now answer with an address, hsa_amd_memory_async_copy accepts both ends and completes, and the previous behaviour is unchanged for host-visible buffers.

The function only ever tried a host mapping, so it answered UNAVAILABLE for any
device-local buffer -- the comment beside it said as much, that a real GPU
device "would use iree_hal_buffer_export". Anything needing a device address
was therefore unreachable through the C API: a peer copy has no source to name,
hsa_amd_agents_allow_access has no allocation to grant, and a caller that wants
hsa_amd_memory_async_copy cannot describe either end, so transfers fall back to
the HAL's blit kernels.

It now exports the buffer through the owning device's allocator and returns the
device allocation pointer, falling back to the existing mapping paths so
host-visible and local-task buffers behave exactly as before.
@Geramy

Geramy commented Aug 21, 2026

Copy link
Copy Markdown
Author

@jimw567 Please take these PRs as suggestions made during creation of an HRX backed engine, if you would like them in hrx let me know what would need to follow to get it merged please.

@benvanik
benvanik requested a review from AWoloszyn September 1, 2026 22:01
@benvanik benvanik changed the title Export the allocation in hrx_buffer_get_device_ptr [HRX] Export the allocation in hrx_buffer_get_device_ptr Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant