Conversation
hrx_allocator_import_buffer always built a HOST_ALLOCATION external buffer, so a device address could not be imported at all -- the allocator rejects a device pointer described as host memory. That leaves no way to bring a peer's memory into this allocator, which is what a cross-device copy needs: a queue will not touch another device's buffer object, and handing it one faults the GPU. When the requested memory type is device-local and not host-visible, the import now describes a DEVICE_ALLOCATION instead. Host imports are unchanged. With this and the device-pointer export, a peer copy is expressible through the C API: get the source's device pointer, import it into the destination's allocator, copy on the destination's queue.
|
Withdrawing this — testing it on 8x gfx1201 showed it does not achieve what the description claims, and the reason suggests it is the wrong shape entirely. With the change applied, the import reaches Each And once that is addressed the right way — a logical device whose Leaving the observation here in case it is useful: there is currently no way through the C API to create a device spanning more than one GPU ( #393 is unaffected and stands on its own — it is what lets a caller reach |
hrx_allocator_import_bufferalways builds aHOST_ALLOCATIONexternal buffer:So a device address cannot be imported at all — the allocator rejects a device pointer described as host memory. That leaves no way to bring one device's memory into another's allocator, which is exactly what a cross-device copy needs:
hrx_queue_copy/hrx_stream_copy_bufferwill not accept another device'shrx_buffer_t, and passing one faults the GPU rather than returning an error.When the requested memory type is device-local and not host-visible, the import now describes an
IREE_HAL_EXTERNAL_BUFFER_TYPE_DEVICE_ALLOCATION. Host imports take the same path they always did.Paired with #393 (device-pointer export), a peer copy becomes expressible through the C API: get the source's device pointer, import it into the destination's allocator, copy on the destination's queue.
Built and exercised on 8x gfx1201 (Radeon AI PRO R9700, ROCm 7.13).