feat(compiler): pack dynamic pool allocs by lifetime (hip-pool-allocs) by fhanuman · Pull Request #570 · ROCm/hip-ep

fhanuman · 2026-07-27T21:00:25Z

Port of closed PR #469. Its precursor, #468, is already merged into main.

Problem

Dynamic allocs can currently share a pool slab only when they have the same dynamic dimensions. Two lifetime-disjoint allocs with different dynamic dimensions therefore consume the sum of their footprints even though they are never live together.

Fix

Default hip-pool-allocs to lifetime-only dynamic packing:

Treat every runtime-sized alloc as its own group.
Allow provably lifetime-disjoint groups to share a slab regardless of dynamic dimensions.
Size each slab to the maximum footprint of its members.
Round small, non-alignment-multiple allocations so subsequent slabs remain aligned.

The previous per-dynamic-dimension best-fit strategy remains available with lifetime-only=false.

Example IR

Two disjoint allocs sized by different dynamic dimensions (%d0, %d1):

Before (stacked — pool = A + B):

%f0 = arith.muli %d0, %c32768 : index      // footprint A
%f1 = arith.muli %d1, %c16384 : index      // footprint B
%sz = arith.addi %f0, %f1 : index          // pool = A + B
%pool = hip.get_pool(%ctx, %sz) : memref<?xi8>
%a = memref.view %pool[%c0] ... : memref<?x8192xf32>
%b = memref.view %pool[%f0] ... : memref<?x4096xf32>   // B after A

After (shared slab — pool = max(A, B)):

%f0 = arith.muli %d0, %c32768 : index
%f1 = arith.muli %d1, %c16384 : index
%sz = arith.maxui %f0, %f1 : index         // pool = max(A, B)
%pool = hip.get_pool(%ctx, %sz) : memref<?xi8>
%a = memref.view %pool[%c0] ... : memref<?x8192xf32>
%b = memref.view %pool[%c0] ... : memref<?x4096xf32>   // aliases A; never co-live

Test plan

Pre-commit hooks passed.
LIT: 349 passed, 2 unsupported.
OutputAllocator and StaticPlugins unit tests passed.

Made with Cursor

Default `hip-pool-allocs` to lifetime-only dynamic packing: each runtime-sized alloc is its own group, and a cross-group lifetime coloring lets any two lifetime-disjoint allocs share one pool slab regardless of their dynamic dimensions. A slab's cost is the MAX of its concurrently-live members rather than the SUM of every alloc routed to it. Only provably-disjoint groups ever share, so overlapping their address ranges is always safe (no mis-pairing). Small (non-alignment-multiple) allocs ride the same path; a slab that holds one rounds its width up so the next slab stays aligned. Keep the previous strategy as a fallback (`lifetime-only=false`): group by dynamic dimension and best-fit-pack the integer static factors within a group before the same cross-group coloring, sending non-multiple allocs to per-size byte-space bins. It is strictly <= the default's per-group footprint and never mis-pairs. Rework the debug-only fragmentation report to compare the emitted dynamic-pool coefficient against the max-load lower bound (single dynamic factor only), which is the meaningful signal under the new default. Tests: `hip-pool-allocs-lifetime-only.mlir` covers slab sharing across different dynamic dims, the overlap-must-not-share guardrail, and the small-alloc round-up; `hip-pool-allocs-dynamic-binning.mlir` now covers the fallback path; the fragmentation-report expectations track the new remark. Validated end-to-end on GPU with a byte-identical pool. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Made-with: Cursor

github-actions · 2026-07-27T21:01:16Z

Thanks for opening a PR!

This project follows LLVM's incremental-development and AI-tool-use
guidance. See CONTRIBUTING.md
for the project workflow.

Before requesting review, please check that:

The change is focused. Substantial work links the relevant issue
or design discussion.
The PR documents relevant test results and updates affected
documentation.
If AI tools provided substantial assistance, the description
explains what was assisted and how it was validated, and commit
trailers identify the tool. The contributor has reviewed and
understands the result.

Reviewers are assigned through
CODEOWNERS where ownership
is configured.

fhanuman · 2026-07-27T21:10:45Z

/run-test models=all

amd-yanjunz · 2026-07-27T21:12:23Z

ROCm OGA CI finished with result: SUCCESS.

PR: feat(compiler): pack dynamic pool allocs by lifetime (hip-pool-allocs) #570
Source comment: models=all
TEST_MODE: performance,accuracy
MODELZOO: LLM_ROCm_dynamic
TEST_MODELS: (all)
FORBID_MODELS: google-gemma-4-26B-A4B-it-FP16W4-qmoe,gemma-4-12B-it-fp16-int4-kquant,NVIDIA-Nemotron-3-Super-120B-A12B-BF16-onnx-fp16-int4-rtn-onnx
NODE: (OGA job default)
CI_OPTIONS: LLM_RUN_REPETITIONS=5 LLM_WARMUP_ITERATIONS=1 LLM_RUN_EPS=hip OGA_MODEL_BENCHMARK_MAX_LENGTH=0 LLM_TOKENS_MITRIX=128|128,2048|128
Jenkins build: http://xsjvitisaijenkins:8080/job/daily_hip_ep_oga/1513/
NTS report: http://xsjnts.xilinx.com/ROCm/hipdnn-detail?database=development&collection=daily_hip_ep_oga&id=6a67dec1f30a8908bb845757

github-actions · 2026-07-27T21:34:44Z

L2 Accuracy Results (EP vs CPU)

Model	Combined L2	Total Elems
conv_test_hybrid	4.8668E-07	64
GroupQueryAttention_seq256	25.2366	2621440
MatMulNBits_o_seq128	259.906	368640
QMoE_seq128	34.957	368640

Threshold: 0.01 | Run: 3432 - Commit: a9715dc

github-actions · 2026-07-27T21:34:47Z

MorphiZen EP Performance Results

Model	QPS	Session (s)	1st Infer (ms)	CPU%	Mem (MB)
full_model_seq128	7.50	6.12	369	3	1243
GroupQueryAttention_seq128	4363.41	1.66706	10	6	311
matmul_down_seq128	504.45	2.36	73	4	351

EPContext Export Performance

Model	QPS	Session (s)	1st Infer (ms)	CPU%	Mem (MB)
full_model_seq128	7.53	45.99	358	3	15590

EPContext Import Performance

Model	QPS	Session (s)	1st Infer (ms)	CPU%	Mem (MB)
full_model_seq128	7.51	9.59	358	3	15760

OGA Benchmark Results

Model	Warmup	Reps	Prompt Len	Gen Tokens	TTFT (ms)	TPS	Peak Mem (GB)	GPU Mem (GB)
gpt-oss-20b-webgpu-int4-rtn-block-32	1	5	128	128	171.7	79.7	1.33	13.54
Llama-3.1-8B-awq-g128-int4-asym-fp16-onnx-dml	1	5	128	128	233.3	42.0	1.22	6.43

OGA Wheel Smoke (Python benchmark_e2e.py)

Model	TTFT (ms)	TPS
Llama-3.1-8B-awq-g128-int4-asym-fp16-onnx-dml	195	40.8

Run: 3432 - Commit: a9715dc

fhanuman · 2026-07-30T14:36:24Z

/run-test models=all

amd-yanjunz · 2026-07-30T14:37:27Z

ROCm OGA CI finished with result: UNSTABLE.

PR: feat(compiler): pack dynamic pool allocs by lifetime (hip-pool-allocs) #570
Source comment: models=all
TEST_MODE: performance,accuracy
MODELZOO: LLM_ROCm_dynamic
TEST_MODELS: (all)
FORBID_MODELS: google-gemma-4-26B-A4B-it-FP16W4-qmoe,gemma-4-12B-it-fp16-int4-kquant,NVIDIA-Nemotron-3-Super-120B-A12B-BF16-onnx-fp16-int4-rtn-onnx
NODE: (OGA job default)
CI_OPTIONS: LLM_RUN_REPETITIONS=5 LLM_WARMUP_ITERATIONS=1 LLM_RUN_EPS=hip OGA_MODEL_BENCHMARK_MAX_LENGTH=0 LLM_TOKENS_MITRIX=128|128,2048|128
Jenkins build: http://xsjvitisaijenkins:8080/job/daily_hip_ep_oga/1589/
NTS report: http://xsjnts.xilinx.com/ROCm/hipdnn-detail?database=development&collection=daily_hip_ep_oga&id=6a6b779568560050cd0ec14d

fhanuman requested review from amd-mingw, edelaye, qianglin-amd and wcy123 as code owners July 27, 2026 21:00

fhanuman mentioned this pull request Jul 28, 2026

[RFC] HIP EP result shapes, dynamic pool packing, and host visibility #580

Open

Merge branch 'main' into perf/pool-alloc-lifetime-only-port

a9715dc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(compiler): pack dynamic pool allocs by lifetime (hip-pool-allocs) - #570

feat(compiler): pack dynamic pool allocs by lifetime (hip-pool-allocs)#570
fhanuman wants to merge 2 commits into
mainfrom
perf/pool-alloc-lifetime-only-port

fhanuman commented Jul 27, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

fhanuman commented Jul 27, 2026

Uh oh!

amd-yanjunz commented Jul 27, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 27, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 27, 2026 •

edited

Loading

Uh oh!

fhanuman commented Jul 30, 2026

Uh oh!

amd-yanjunz commented Jul 30, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

fhanuman commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Example IR

Test plan

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

fhanuman commented Jul 27, 2026

Uh oh!

amd-yanjunz commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

L2 Accuracy Results (EP vs CPU)

Uh oh!

github-actions Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MorphiZen EP Performance Results

EPContext Export Performance

EPContext Import Performance

OGA Benchmark Results

OGA Wheel Smoke (Python benchmark_e2e.py)

Uh oh!

fhanuman commented Jul 30, 2026

Uh oh!

amd-yanjunz commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fhanuman commented Jul 27, 2026 •

edited

Loading

amd-yanjunz commented Jul 27, 2026 •

edited

Loading

github-actions Bot commented Jul 27, 2026 •

edited

Loading

github-actions Bot commented Jul 27, 2026 •

edited

Loading

amd-yanjunz commented Jul 30, 2026 •

edited

Loading