Cherrypick: into release/2.13 from release/2.12 [ ifu 20260722 ]#3465
Open
pablo-garay wants to merge 115 commits into
Open
Cherrypick: into release/2.13 from release/2.12 [ ifu 20260722 ]#3465pablo-garay wants to merge 115 commits into
pablo-garay wants to merge 115 commits into
Conversation
…ytorch#180508) Enable fetch-tags in checkout-pytorch to fix release tag detection (pytorch#180504) ## Summary Release candidate builds triggered by tags (e.g. `v2.12.0-rc1`) are incorrectly producing dev nightly versions instead of RC versions. The root cause is that pytorch#172544 added `single-branch: true` to the `checkout-pytorch` action when switching from `actions/checkout@v4` to `pytorch/test-infra/.github/actions/checkout@main`. With `single-branch: true`, git restricts the fetch refspec and tags are no longer fetched — even with `fetch-depth: 0`. This causes `binary_populate_env.sh` to fail at: git describe --tags --match 'v[0-9].[0-9].[0-9]*' --exact fatal: No names found, cannot describe anything. and fall through to the dev version path, producing `2.12.0.dev20260415+cpu` instead of `2.12.0+cpu`. The fix adds `fetch-tags: true` to both checkout steps (primary and retry) in the `checkout-pytorch` action, ensuring tags are explicitly fetched regardless of `single-branch` mode. Failing build: https://github.com/pytorch/pytorch/actions/runs/24474783525/job/71524093912 ## Test plan - Verify that a workflow triggered by an RC tag (e.g. `v2.12.0-rc1`) correctly sets `PYTORCH_BUILD_VERSION=2.12.0+cpu` instead of `2.12.0.dev<date>+cpu` - Verify that nightly builds on the `nightly` branch are unaffected (they should still produce dev versions) Pull Request resolved: pytorch#180504 Approved by: https://github.com/seemethere (cherry picked from commit ba240c9) Co-authored-by: atalman <atalman@fb.com>
[ROCm] Fix inline_asm_elementwise for ROCm (pytorch#180257) Fixes pytorch#179925. Fixes pytorch#179927. Fixes pytorch#180010. Fixes pytorch#179965. Fixes pytorch#179982. Fixes pytorch#179946. The inline_asm_elementwise HOP had all assembly hardcoded as PTX and the test file was blocklisted on ROCm. This PR adds AMDGCN equivalents behind torch.version.hip guards so every test case works on both backends. The NVIDIA path is untouched throughout. Most PTX-to-AMDGCN translations are direct (mov.f32 to v_mov_b32, f/r constraints to v). The fp16/bf16 native cases required a different approach. AMDGCN has no h constraint for 16-bit registers, so we add in f32 and convert down via v_cvt_f16_f32 and v_cvt_pk_bf16_f32. Two Inductor codegen fixes handle AMDGCN's newline-separated instructions (PTX uses semicolons on one line). In utils.py, multiline format_node() output is split so each line gets the comment prefix. In triton.py, asm literals are double-escaped via repr() to survive two layers of Python string parsing. Both are HIP-guarded. Pull Request resolved: pytorch#180257 Approved by: https://github.com/pruthvistony, https://github.com/jeffdaily, https://github.com/pragupta (cherry picked from commit 9e52279) Co-authored-by: Chinmay Kuchinad <ChinmayDattanand.Kuchinad@amd.com>
…fo_property (pytorch#180687) [UT][ROCm][inductor] ROCm-specific XFAILS list for torchinductor_opinfo_property (pytorch#180505) Fixes pytorch#180032 Fixes pytorch#180033 Fixes pytorch#180034 Fixes pytorch#180035 Fixes pytorch#180036 Fixes pytorch#180037 Fixes pytorch#180038 Fixes pytorch#180039 Fixes pytorch#180046 Fixes pytorch#180047 Fixes pytorch#180048 Fixes pytorch#180049 Fixes pytorch#180051 Fixes pytorch#180053 Fixes pytorch#180054 Fixes pytorch#180059 Fixes pytorch#180060 Fixes pytorch#180062 Fixes pytorch#180063 Fixes pytorch#180069 Fixes pytorch#179962 These tests were actually passing on ROCm, but because of the shared (CUDA+ROCm) XFAILED list, those ROCm passes were being turned into failures. This PR adds a ROCm-specific XFAILS list which is a subset of the CUDA XFAILS list. Pull Request resolved: pytorch#180505 Approved by: https://github.com/jeffdaily (cherry picked from commit d129991) Co-authored-by: Nichols A. Romero <nick.romero@amd.com>
[ROCm] Update scaled_mm DeepSeek error message (pytorch#180384) Fixes pytorch#180074. Fixes pytorch#180075. Fixes pytorch#180076. Fixes pytorch#180077. Fixes pytorch#180078. Fixes pytorch#179954. Pull Request resolved: pytorch#180384 Approved by: https://github.com/jeffdaily (cherry picked from commit c83ef11) Co-authored-by: Jagadish Krishnamoorthy <jagadish.krishnamoorthy@amd.com>
…ytorch#180691) [ROCm] Enable ROCm swizzle check and update scaled_mm swizzle tests (pytorch#179782) On ROCm, require MX scaled_mm swizzle inputs to provide one value for both A and B and enforce that both are NO_SWIZZLE. Update test_passed_swizzle_arrays to use ROCm-specific expectations and add coverage for the explicit NO_SWIZZLE value check. For nvfp4, swizzle check is skipped but eventually fails with error NVFP4 scaling not supported on ROCM. miscellaneous fix: fix swizzle validation error messages to use correct singular/plural value wording. Fixes pytorch#180073 Pull Request resolved: pytorch#179782 Approved by: https://github.com/jeffdaily, https://github.com/drisspg (cherry picked from commit f07a136) Co-authored-by: Jagadish Krishnamoorthy <jagadish.krishnamoorthy@amd.com>
Increase Python docs build timeout to 45m (pytorch#180509) ## What Increases the timeout for Python documentation builds from 30 minutes to 45 minutes in both the standard and ARC runner matrix configurations. ## Why The Python docs build has been occasionally exceeding the 30-minute timeout, causing spurious CI failures. A 45-minute limit provides enough headroom for slower builds while still catching genuinely stuck jobs. # Changes - `.github/workflows/_docs.yml`: Updated timeout for Python docs from 30m to 45m in the standard runner matrix (line ~96) - `.github/workflows/_docs.yml`: Updated timeout for Python docs from 30m to 45m in the ARC runner matrix (line ~287) Pull Request resolved: pytorch#180509 Approved by: https://github.com/malfet, https://github.com/huydhn, https://github.com/atalman (cherry picked from commit d5268cd) Co-authored-by: Jean Schmidt <contato@jschmidt.me>
…ytorch#180815) [xpu][fix] Include lazy_triton_compile.h in XPU cpp_wrapper header (pytorch#180440) Motivation: Commit 385fd32 removed the inline #include <lazy_triton_compile.h> from cpp_wrapper codegen, assuming the precompiled header (cuda.h) already includes it. However, xpu.h does not include it, breaking ~90 XPU cpp_wrapper tests with compilation errors (undeclared LazyKernelCompileResult, _module_pending_kernels, etc.) in nightly CI. Solution: Add the missing include to xpu.h to match cuda.h. Test Plan: source .env && python test/inductor/test_torchinductor_codegen_dynamic_shapes.py -k test_multi_device # passes Pull Request resolved: pytorch#180440 Approved by: https://github.com/desertfire, https://github.com/jansel
…g graph capture (pytorch#180692) [ROCm] Resolve timeouts caused due to hipblasLT module creation during graph capture (pytorch#179892) Fixes pytorch#179943. Fixes pytorch#179945. Fixes pytorch#179947. After pytorch#179053 , ROCm hipBLASLt handle caching changed from per-device to per-(device, stream). That means first use on a capture stream can now trigger lazy hipblasLtCreate on that same stream; on ROCm this init path does capture-unsafe internal allocation/setup, which can fail with stream-capture errors (and sometimes hang) if it runs during capture. This PR fixes that by pre-initializing the hipBLASLt handle for the target capture stream immediately before capture_begin, so handle creation never occurs inside capture. Pull Request resolved: pytorch#179892 Approved by: https://github.com/jeffdaily (cherry picked from commit c2cc96e) Co-authored-by: Chinmay Kuchinad <ChinmayDattanand.Kuchinad@amd.com>
) [ROCm] Fix evaluate_platform_supports_fp8 false-positive (pytorch#180518) On ROCm, evaluate_platform_supports_fp8() did not return False when the device architecture did not match any supported FP8 GPU, causing the function to fall through and report support unconditionally. Add an explicit False return when no supported ROCm architecture is matched, ensuring correct FP8 capability detection on unsupported devices. Fixes pytorch#179960 Fixes pytorch#179949 Pull Request resolved: pytorch#180518 Approved by: https://github.com/jeffdaily (cherry picked from commit 702aa84) Co-authored-by: Jagadish Krishnamoorthy <jagadish.krishnamoorthy@amd.com>
…NCCL lower than 2.29.7 (pytorch#180693) Conditionally skipped NCCL suspend/get_memory_stats/resume tests for NCCL lower than 2.29.7 (pytorch#180547) Added conditional skips for test_suspend, test_get_memory_stats and test_resume for NCCL lower than 2.29.7 Fixes: pytorch#180009 Fixes: pytorch#179953 Fixes: pytorch#179950 Pull Request resolved: pytorch#180547 Approved by: https://github.com/pragupta, https://github.com/Skylion007 (cherry picked from commit 1b07af0) Co-authored-by: Albert Malewski <albert.malewski@amd.com>
…acing (pytorch#180964) [dynamo] Filter aliased intermediates in autograd.Function forward tracing (pytorch#180921) In-place op inside autograd.Function.forward combined with a side effect (list append) causes collect_intermediate_outputs to add a pre-mutation alias as an extra subgraph output. Because the alias shares the same TensorImpl as the real return value, set_gradient_edge overwrites output_nr, routing the backward gradient to the wrong slot and producing zero gradients. Use the filter_aliased_intermediates infra on autograd.Function HOP as well to avoid the issue. pytorch#180642 Authored with Claude. Pull Request resolved: pytorch#180921 Approved by: https://github.com/xmfan, https://github.com/ezyang, https://github.com/zou3519 (cherry picked from commit d4791a1) Co-authored-by: Animesh Jain <anijain@umich.edu>
[caching] Fix SymBool pickling issue with torch.cond. (pytorch#180385) Summary: From vllm we got an issue report vllm-project/vllm#39386 that aot compile has trouble serializing SymBool in the graph when torch.cond is used. ``` EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/aot_autograd.py", line 1093, in aot_module_simplified (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] compiled_fn = AOTAutogradCache.try_load( (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/autograd_cache.py", line 721, in try_load (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] raise e (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/autograd_cache.py", line 633, in try_load (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] cache_key, debug_lines = autograd_cache_key( (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] ^^^^^^^^^^^^^^^^^^^ (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] File "/app/rfc/2nd/vllm/vllm/compilation/backends.py", line 326, in autograd_cache_key (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] result = orig(*args, **kwargs) (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/autograd_cache.py", line 496, in autograd_cache_key (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] key = "a" + pickler.get_hash(details) (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/codecache.py", line 640, in get_hash (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] serialized_data = self.dumps(obj) (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] ^^^^^^^^^^^^^^^ (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/codecache.py", line 625, in dumps (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] self.dump(obj) (EngineCore pid=63310) ERROR 04-04 03:52:41 [core.py:1108] RuntimeError: <pybind11_builtins.pybind11_detail_function_record_v1_system_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1 object at 0x7f39e8116030> is not pickleable. ``` This PR tries to faithfully reproduce the same error and fix the issue for both caching (pickling) and inductor. Test Plan: CI Reviewers: Subscribers: Tasks: Tags: Pull Request resolved: pytorch#180385 Approved by: https://github.com/Lucaskabela, https://github.com/bobrenjc93 (cherry picked from commit 8bc4cad) Co-authored-by: zhxchen17 <zhxchen17@fb.com>
…orch#181070) Skip llms-full.txt during Sphinx build, generate in nightly push (pytorch#180918) - Set `llm_generate_full = "false"` in `conf.py` to skip llms-full.txt generation during the Sphinx build to avoid long build - Add a post-build step in `python_doc_push_script.sh` that generates llms-full.txt and markdown files only during nightly/release pushes (`WITH_PUSH=true`), after Sphinx has finished ## Motivation - The llms-full.txt generation converts all HTML pages to markdown and concatenates them into a single file. On a build this large (~2800 pages), whih may cause CI timeouts. Since llms-full.txt only needs to be updated once per nightly push (not on every PR), we can decouple it from Sphinx and run it as a separate step. Pull Request resolved: pytorch#180918 Approved by: https://github.com/atalman (cherry picked from commit bca0ce3) Co-authored-by: Svetlana Karslioglu <svekars@meta.com>
…180816) [xpu][fix] Enable bmm_outer_product Triton override for XPU (pytorch#180441) Motivation: The bmm_outer_product Triton override and its test (added in a021b85) were CUDA-only, causing test_bmm_outer_product failures on XPU nightly CI. Solution: Register the Triton bmm_outer_product implementation for the XPU dispatch key when torch.xpu is available. Generalize the test to use GPU_TYPE so it runs on both CUDA and XPU. Test Plan: source .env && python test/test_bmm_outer_product.py # all 9 tests pass Pull Request resolved: pytorch#180441 Approved by: https://github.com/guangyey, https://github.com/desertfire ghstack dependencies: pytorch#180440
…rch#180897) [ROCm] Run test_scaled_mm_deepseek_error_messages on mi350 arch (pytorch#180731) On ROCm, fp8 mx block size is supported on mi355 only. mi300 has support of fp8 only, hence skipping the test. Pull Request resolved: pytorch#180731 Approved by: https://github.com/jeffdaily (cherry picked from commit 92b5ad2) Co-authored-by: Jagadish Krishnamoorthy <jagadish.krishnamoorthy@amd.com>
…or_opinfo test (pytorch#180903) [ROCm][UT] Remove previously retained Triton 3.7 skip for torchinductor_opinfo test (pytorch#179856) Remove the intentionally retained `@skipIfRocm(msg=\"Fails with Triton 3.7 on MI200\")` decorator from `test_torchinductor_opinfo.py::TestInductorOpInfo.test_comprehensive`. The NEW FAILURE reported by CI is due to the shard timing out because the sharding logic is using an unrealistic time in [test_times.json](https://raw.githubusercontent.com/pytorch/test-infra/refs/heads/generated-stats/stats/test-times.json). Note that this is enabling several thousands UT. The test_torchinductor_opinfo is completing, but there are several restarts due to hitting the 30 minute mark. test_torchinductor_opinfo summary below (from a previous run): ``` MI300: - PASSED: 3549 - SKIPPED: 715 - XFAILED: 53 - FAILED/ERROR: 0 ``` ``` MI355: - PASSED: 3293 - SKIPPED: 751 - XFAILED: 74 - FAILED/ERROR: 0 ``` After rebase, the latest runs also ran test_torchinductor_opinfo successfully: [rocm-mi300](https://github.com/pytorch/pytorch/actions/runs/24598305023/job/71933767526) and [rocm-mi355](https://github.com/pytorch/pytorch/actions/runs/24598305307) Made with [Cursor](https://cursor.com) Related PRs: pytorch#174896 pytorch#178450 pytorch#178559 pytorch#179794 Pull Request resolved: pytorch#179856 Approved by: https://github.com/jeffdaily, https://github.com/atalman (cherry picked from commit 05e702f) Co-authored-by: Nichols A. Romero <nick.romero@amd.com>
pytorch#180916) [user-streams] Fix cudagraphs compatibility with current stream (pytorch#180497) Fixes pytorch#180396 Issue: When torch.compile(fn, mode="reduce-overhead") captures a CUDA graph, custom stream ops (event record/wait) resolve the "current stream" from the external object registry. But the registry was populated during the dynamo bytecode prologue with the trace-time default stream — not the cudagraph capture stream. This caused cudaErrorStreamCaptureUnsupported. Fix: Register the current stream at index 0 in the external object registry at trace time. The inductor wrapper emits set_external_object_by_index(0, torch.cuda.current_stream()) at runtime, so during cudagraph capture, index 0 resolves to the actual capture stream instead of the stale default stream. Real code changes (~30 lines across 4 files): 1. graph_bytecode_inputs.py (+9) — Added CURRENT_STREAM_INDEX = 0 constant and set_external_object_by_index() which updates an entry at runtime and keeps the object alive via the existing keep_alive list. 2. variables/streams.py (+~20) — SymbolicStreamState.__init__ now registers the current stream at index 0 when the registry is fresh. Simplified _get_stream_arg to just return user_object_index (no conditional logic). Added back cur_stream_id() which output_graph.py needs. 3. variables/builder.py (+4/-4) — When wrapping a stream with CurrentStreamSource, use CURRENT_STREAM_INDEX instead of allocating a new index. 4. codegen/wrapper.py (+7) — Emit set_external_object_by_index(0, torch.cuda.current_stream()) at the top of the wrapper so custom ops see the actual runtime stream (capture stream during cudagraph recording). Pull Request resolved: pytorch#180497 Approved by: https://github.com/Lucaskabela, https://github.com/eellison
…d via issue) (pytorch#180927) [ROCm] skip test_autoheuristic in-code (already disabled via issue)
[Release 2.12] Update torch-xpu-ops commit pin
…ytorch#181019) [dynamo] Fix cuda_stream pointer extraction for generic torch.Stream (pytorch#180623) Fixes pytorch#180179 ## [dynamo] Fix cuda_stream pointer extraction for generic torch.Stream ### Summary I noticed that `torch.compile` narrows `torch.cuda.Stream` down to generic `torch.Stream` objects during tracing. This is a problem because generic streams don't have the `.cuda_stream` property, so Dynamo falls back to returning the `stream_id` (a small int like `3`) instead of the actual hardware pointer. This leads to segfaults or "illegal memory access" errors when that integer is passed into a CUDA kernel expecting a real `cudaStream_t` address. ### Fix I updated the logic in `CudaStreamVariable.var_getattr` to use a `hasattr` check instead of a strict type check, which lets us handle narrowed or proxied stream objects. I also added a recovery path that pulls the hardware pointer from the current CUDA context if the attribute is missing, while ensuring the default stream (ID 0) still returns a literal 0 to maintain parity with Eager mode and avoid breaking existing behavior. ### Testing Added `test/dynamo/test_stream_pointer.py` to cover a few cases and verified that the compiled pointer matches the eager pointer for custom streams. Also verified whether the returned value is a real memory address (not just a truncated ID). It avoids issues from pytorch#173172 too. Pull Request resolved: pytorch#180623 Approved by: https://github.com/mlazos (cherry picked from commit 83ba944) Co-authored-by: unknown <sparkingaqua@gmail.com>
Disable llms-full.txt (pytorch#181131) Pull Request resolved: pytorch#181131 Approved by: https://github.com/atalman (cherry picked from commit e24247f) Co-authored-by: Svetlana Karslioglu <svekars@meta.com>
* [NCCL][Symmetric Memory] Fix groupName in IntraNodeComm (pytorch#180663) This PR fixes: ``` RuntimeError: Could not resolve the process group registered under the name ``` in `python test/distributed/test_c10d_nccl.py CommTest.test_intra_node_comm_all_reduce`. Just FYI: A quite similar error was attempted to be fixed in pytorch#177700, however the refactor pytorch#176506 partially fixed the original one by lazy pg registration, though missed the process group name. Pull Request resolved: pytorch#180663 Approved by: https://github.com/Skylion007 (cherry picked from commit 168339d) * Add parameterization to test_intra_node_comm_all_reduce Parameterized the test for intra-node communication to allow for custom group names. --------- Co-authored-by: Aidyn-A <aidyn.b.aitzhan@gmail.com> Co-authored-by: Aidyn-A <31858918+Aidyn-A@users.noreply.github.com>
[MPS] Fix sliced channels_last tensors handling (pytorch#180992) By calling `Tensor::suggest_memory_format(/*channels_last_strides_exact_match=*/true)`, that not only checks that slices look like channels last, but also that they are contiguously channels last Fixed incorrect channels last slicing handling in `conv2d`, `batch_norm` and `[adaptive_]avg_pool2d` Fixes pytorch#180984 Pull Request resolved: pytorch#180992 Approved by: https://github.com/Skylion007 (cherry picked from commit fc92844) Co-authored-by: Nikita Shulga <nshulga@meta.com>
Add TLS stack_bounds so we dont pay expensive reads (pytorch#181018) Before this PR ```Shell python /home/drisspg/meta/pytorch/agent_space/unwind_repro/bench_unwind.py platform : aarch64 / Linux torch : 2.12.0.dev20260414+cu130 git : 1b7d09a TORCH_SHOW_CPP : 0 cuda available : True iters : 20000 (warmup 500) direct 20000 iters 3.740s 187.02 us/call worker 20000 iters 0.018s 0.88 us/call alloc 20000 iters 9.568s 478.39 us/call ``` After: ```Shell python /home/drisspg/meta/pytorch/agent_space/unwind_repro/bench_unwind.py platform : aarch64 / Linux torch : 2.13.0a0+git446033e git : 446033e TORCH_SHOW_CPP : 0 cuda available : True iters : 20000 (warmup 500) direct 20000 iters 0.008s 0.39 us/call worker 20000 iters 0.007s 0.36 us/call alloc 20000 iters 0.082s 4.08 us/call ``` Script I worked on with claude as a proxy: ```Py """ Microbenchmark for the aarch64 C++ unwinder regression. Background: on aarch64, torch::unwind::unwind() now does real frame-pointer walking. Its get_stack_bounds() calls pthread_getattr_np() per invocation, which on the *main thread* parses /proc/self/maps every time. Non-main threads cache stack bounds in TLS at pthread_create, so they should be fast. Three modes: direct -- gather_traceback() in a tight loop on the main thread worker -- same loop on a spawned thread (expected: much faster on aarch64) alloc -- allocate many small CUDA tensors with _record_memory_history on, exercising the full CachingAllocator -> CapturedTraceback path Usage: python bench_unwind.py --mode direct --iters 20000 python bench_unwind.py --mode worker --iters 20000 python bench_unwind.py --mode alloc --iters 20000 python bench_unwind.py --mode all --iters 20000 """ import argparse import os import platform import threading import time import torch from torch._C._profiler import gather_traceback def bench_direct(iters: int) -> float: t0 = time.perf_counter() for _ in range(iters): gather_traceback(python=False, script=False, cpp=True) return time.perf_counter() - t0 def bench_worker(iters: int) -> float: elapsed: list[float] = [] def run(): t0 = time.perf_counter() for _ in range(iters): gather_traceback(python=False, script=False, cpp=True) elapsed.append(time.perf_counter() - t0) th = threading.Thread(target=run) th.start() th.join() return elapsed[0] def bench_alloc(iters: int) -> float: if not torch.cuda.is_available(): raise RuntimeError("alloc mode requires CUDA") torch.cuda.memory._record_memory_history(enabled="all", max_entries=iters * 2) try: torch.cuda.synchronize() t0 = time.perf_counter() for _ in range(iters): torch.empty(16, device="cuda") torch.cuda.synchronize() return time.perf_counter() - t0 finally: torch.cuda.memory._record_memory_history(enabled=None) def report(label: str, iters: int, seconds: float) -> None: us_per_call = seconds * 1e6 / iters print(f"{label:<8} {iters:>8} iters {seconds:8.3f}s {us_per_call:8.2f} us/call") def main() -> None: ap = argparse.ArgumentParser() ap.add_argument("--mode", choices=["direct", "worker", "alloc", "all"], default="all") ap.add_argument("--iters", type=int, default=20000) ap.add_argument("--warmup", type=int, default=500) args = ap.parse_args() print(f"platform : {platform.machine()} / {platform.system()}") print(f"torch : {torch.__version__}") print(f"git : {torch.version.git_version}") print(f"TORCH_SHOW_CPP : {os.environ.get('TORCH_SHOW_CPP_STACKTRACES', '<unset>')}") print(f"cuda available : {torch.cuda.is_available()}") print(f"iters : {args.iters} (warmup {args.warmup})") print() for _ in range(args.warmup): gather_traceback(python=False, script=False, cpp=True) if args.mode in ("direct", "all"): report("direct", args.iters, bench_direct(args.iters)) if args.mode in ("worker", "all"): report("worker", args.iters, bench_worker(args.iters)) if args.mode in ("alloc", "all"): if torch.cuda.is_available(): report("alloc", args.iters, bench_alloc(args.iters)) else: print("alloc skipped (no CUDA)") if __name__ == "__main__": main() ``` Pull Request resolved: pytorch#181018 Approved by: https://github.com/ezyang (cherry picked from commit ba15482) Co-authored-by: drisspg <drisspguessous@gmail.com>
…#181170) [Bugfix] Fix the MetaProxy error caused by skipping dispatch (pytorch#181135) Fixes pytorch#180906 ### Summary * torch/fx/passes/_tensorify_python_scalars.py — Wraps the pass body with torch._C._EnableTorchFunction(). The tensorify_python_scalars pass uses MetaProxy objects which rely on __torch_function__ dispatch. PR pytorch#177088 introduced a DisableTorchFunctionSubclass wrapper around compiled graphs to prevent double dispatch at runtime, but this context can also be active during AOTAutograd compilation where tensorify_python_scalars runs. _EnableTorchFunction re-enables dispatch so MetaProxy ops are intercepted correctly. * test/dynamo/test_subclasses.py — regression test passing on this PR ### Test plan ```bash python -m pytest "test/dynamo/test_subclasses.py::SubclassTests::test_tensorify_under_disabled_torch_function" -xvs ``` ``` test/dynamo/test_subclasses.py::SubclassTests::test_tensorify_under_disabled_torch_function PASSED [0.0199s] ``` Authored with Claude opus 4.7 Pull Request resolved: pytorch#181135 Approved by: https://github.com/atalman (cherry picked from commit 67450d0) Co-authored-by: Lucas Kabela <lucaskabela@meta.com>
Split onehot checks for CPU and accelerators (pytorch#179831) Fixes intel/torch-xpu-ops#3284 Currently OneHot has boundary checks for CPU and these checks are skipped for `CUDA`, `MPS`, `XLA` and `PrivateUser1`. As `XPU` is not included on the list it causes D2H calls that affect performance. Instead of adding `XPU` to the list I think it's better to flip the condition to check boundaries only for `CPU` and allow accelerators to skip this validation. Pull Request resolved: pytorch#179831 Approved by: https://github.com/guangyey, https://github.com/albanD (cherry picked from commit 0467d16) Co-authored-by: Sławomir Siwek <slawomir.siwek@intel.com>
…ytorch#181287) [inductor] makes cuda 13.0 cross compliation works (pytorch#179229) (pytorch#179229) Summary: CUDA 13.0+ ships a hybrid cudart.lib that contains MSVC-compiled static objects referencing GS security symbols (__security_cookie, __security_check_cookie, etc.) which MinGW cannot resolve, causing undefined reference errors during Windows cross-compilation. This diff adds a _ensure_mingw_cudart_import_lib() function to cpp_builder.py that automatically generates a pure MinGW-compatible import library (libcudart.a) from the CUDA runtime DLL using gendef + x86_64-w64-mingw32-dlltool. If WINDOWS_CUDA_HOME is not set or libcudart.a already exists, no action is taken. Locates cudart64_*.dll in the CUDA bin directory, generates a .def file via gendef, and creates libcudart.a via dlltool. Falls back gracefully to the original cudart.lib if gendef/dlltool are unavailable or fail to support lower version like 12.6. Test Plan: ci Differential Revision: D99398736 Pull Request resolved: pytorch#179229 Approved by: https://github.com/malfet, https://github.com/seemethere (cherry picked from commit c42a861) Co-authored-by: Songhao Jia <gasoonjia@meta.com>
[Bugfix] Fix triton crash in mixed device case for bmm (pytorch#181251) ## Summary Partial fix of pytorch#180909 PyTorch's bmm_outer_product Triton kernel override crashes with ValueError: Pointer argument cannot be accessed from Triton (cpu tensor?) when a non-CUDA tensor reaches the CUDA dispatch key handler. This happens in practice when transformers models have non-persistent CPU buffers (e.g. inv_freq in rotary embeddings) that get matmul'd with GPU tensors — the @ operator dispatches through the CUDA key, but the Triton kernel can't access the CPU pointer. bmm_outer_product_impl guards on shape, Triton availability, and CoW status, but never checks that both inputs are actually on CUDA before dispatching to the Triton kernel. The fallback path (fallback_kernel.call_boxed) already exists and would handle this correctly (Runtime Error with descriptive failure), but the missing device guard means it's never reached so we add guards to check this ## Tests - test_mixed_device_outer_product_fallback: CPU + CUDA outer-product inputs now raise a clean RuntimeError (device mismatch) instead of a Triton pointer crash. - test_cpu_outer_product_fallback: Pure-CPU outer-product bmm produces correct results via the fallback path. Pull Request resolved: pytorch#181251 Approved by: https://github.com/atalman (cherry picked from commit 3143279) Co-authored-by: Lucas Kabela <lucaskabela@meta.com>
…#181349) [XPU][dynamo] Save XPU autocast state in dynamo global state (pytorch#180309) save_global_state() only saves CUDA and CPU autocast state. When a graph break occurs inside autocast('xpu'), the resume function is traced with default XPU state (enabled=False, dtype=float16) instead of the active state, causing dtype mismatch. This adds XPU autocast enabled and dtype to save_global_state() and _supported_global_states. Pull Request resolved: pytorch#180309 Approved by: https://github.com/jianyizh, https://github.com/guangyey, https://github.com/mlazos (cherry picked from commit 81dfcde) Co-authored-by: Deng, Weishi <weishi.deng@intel.com> Co-authored-by: Yu, Guangye <guangye.yu@intel.com>
…pool_ctx_multithread (#3372) `test_cuda.py::TestMemPool::test_mempool_ctx_multithread` assumes the device allocator snapshot is empty after empty_cache(), but earlier tests in the same process can leave inactive segments behind. Call `torch._C._cuda_clearCublasWorkspaces()` at the start of the test (before empty_cache()) so prior tests do not pollute the initial segment count. This matches the pattern already used in other tests Reproducer (fails without the fix): ```bash python test_cuda.py TestCuda.test_matmul_device_mismatch TestMemPool.test_mempool_ctx_multithread ``` test_matmul_device_mismatch leaves reserved segments. test_mempool_ctx_multithread then fails with `Expected 0 but got N` on the opening assertion. The test passes in isolation. Fixes pytorch#153460 Pull Request resolved: pytorch#187991 Approved by: https://github.com/jeffdaily (cherry picked from commit 77fa1dc)
…rchs (#3316) Add gfx1100, gfx1101, gfx1151 to hipBLASLt preferred archs
Backport upstream PR pytorch#178264 to branch [ROCm:release/2.12](https://github.com/ROCm/pytorch/tree/release/2.12) so torch.cuda.Event(external=True) works inside HIP graphs. cc: @mgehre-amd
…ts (pytorch#188580) Cherry-pick of #3384 Co-authored-by: zjliu-amd <zhuojin.liu@amd.com>
https://amd-hub.atlassian.net/browse/ROCM-24803?focusedCommentId=500051 Interim unblock: pins torchaudio to the stable release/2.11 (2.11.0), which already builds and publishes on Windows, replacing the floating pytorch/audio main (2.11.0a0) pin. The torchaudio 2.11.0.1 ROCm GPU port is tracked separately in #3382 and ROCm/audio#16.
## Summary Updates the apex pinned commit in `related_commits` for the ROCm apex `release/1.12.0` branch, from `7e4aca37812b99e9c3e302d7d1d2e37ad6377f8c` to `ad1376f20dc0ac107040b6077c96aed52bd14298`. This bump brings in a cherry-pick of ROCm/apex PR #334 (ROCm/apex#334), which fixes non-contiguous inputs in the fused RoPE aiter backend by enforcing last-dim contiguity at the apex/aiter boundary (fixing the "rope_2d_bwd_impl requires all stride_d to be 1" crash on transposed inputs and broadcast/expanded gradients). This is purely a dependency pin update to pull in the fix, not a PyTorch code change. ## Test plan - [ ] Metadata-only change to the `related_commits` pin file; CI validates apex builds against the new pin. Authored with the assistance of Claude (Anthropic AI assistant).
…behavior change Cherry-pick of #3394 Co-authored-by: zjliu-amd <zhuojin.liu@amd.com>
…y repeated call… (#3403) This is a cherry-pick of pytorch#188673 Removing a repeated call of hipMemMap which is causing the errors in SymmetricMemory.
…#183348) Pull Request resolved: pytorch#183348 Approved by: https://github.com/jeffdaily
…ch#178310) Change to turn CK build on by default. Various filters were added to reduce the number of kernels generated by default Pull Request resolved: pytorch#178310 Approved by: https://github.com/jeffdaily, https://github.com/pruthvistony Co-authored-by: Chinmay Kuchinad <ChinmayDattanand.Kuchinad@amd.com>
## Summary - Backports upstream PyTorch PR pytorch#181112 to unify Inductor warp-size plumbing and make HIP wave32 devices avoid the Wave64 num_warps halving path. - Backports upstream PyTorch PR pytorch#183014 to make missing CUDA/HIP warp-size metadata explicit and preserve non-GPU fallback behavior. - Resolves release/2.12 conflicts by preserving ROCm release branch structure while applying the upstream warp-size behavior. ## Test plan - `python -m py_compile` on all changed Python files - `git diff --check origin/release/2.12...HEAD` - IDE lints: no errors - Not run: focused runtime unit tests, because the checkout is not built/generated enough for local `torch.version` --------- Co-authored-by: Bin Bao <binbao@meta.com>
…AGMA is not available with ROCM (pytorch#178229)" (#3412) Reverts #3340, 2.12 doesn't need this change
pytorch#178310)" This reverts commit c06c8ac.
…ytorch#183348)" This reverts commit cb841d0.
Reformat ck_sdpa cmake logic to ONLY build for supported arch's. Also add SFINAE for potentially incompatible CK versions --------- Co-authored-by: Chinmay Kuchinad <ChinmayDattanand.Kuchinad@amd.com>
## Summary ROCm PyTorch 2.11 is already on ROCm Triton 3.7.1. Refer to: #3359 Plus these two additional fixes: 60ae578 - [release/3.7.1] [Revert] [Coalesce] Fix the default order to be row major (pytorch#5707) pytorch#7143 (pytorch#8920) (#949) 0263a6a - [Release 3.7.x][AMD] Backport descriptor load pipelining predicate fix First commit is required to resolve some vLLM performance Jiras. Work done by @jayfurmanek https://amd-hub.atlassian.net/browse/ROCM-27315 https://amd-hub.atlassian.net/browse/ROCM-27318 https://amd-hub.atlassian.net/browse/ROCM-27316 Second commit is an MI450 + TDM only change. ## Test plan - [x] build and editable-install `/home/niromero/docker_workspace/triton` from `release/internal/3.7.x` - [x] verify `import triton` reports version `3.7.1` - [ ] run PyTorch CI Made with [Cursor](https://cursor.com) --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com>
#3421) ## Summary Validation theRock Run: https://github.com/ROCm/TheRock/actions/runs/29045360944 Cherry-pick of upstream [pytorch#188597](pytorch#188597) onto `release/2.12` to add initial ROCm support for **gfx1250 (CDNA5)**. Co-authored-by: @glen-amd ### What this PR enables - Gates gfx1250-specific behavior on ROCm 7.14+ (the support floor), applied consistently across CUDABlas.cpp, ScaledBlas.cpp, CUDAHooks.cpp, and the test helpers. - Adds gfx1250 to the hipBLASLt preferred/supported arch lists and the hipSparseLt support check (ROCm 7.14+). - Extends the scaled GEMM / MX-format paths (block-wise Float8_e8m0fnu, mxfp8, mxfp4) to gfx1250. - Sets the gfx1250 shared-memory limit to 320 KB (vs 160 KB on gfx950), matching on the suffix-stripped arch name. - Adds CDNA2/CDNA3/CDNA5 "or later" arch helpers and updates FP8 / MX GEMM test gating in common_cuda.py. - Builds Composable Kernel (CK) GEMM as a separate ck_gemm library with gfx1250 filtered out of HIP_ARCHITECTURES, so a multi-arch build that includes gfx1250 keeps CK GEMM for the other archs instead of failing to compile for gfx1250 (composable_kernel has no gfx1250 support yet). The filter is removed once CK supports gfx1250. - Extends the gfx942 nontemporal vectorized-load path in MemoryAccess.cuh to gfx1250. - Refactors duplicated arch-check logic in CUDABlas.cpp, ScaledBlas.cpp, and cuSPARSELtOps.cpp per review feedback. - Writes the CublasHandlePool workspace archs in full (gfx942/gfx950/gfx1250). Enabled via the main merge: - Flash attention and memory-efficient attention on gfx1250 via AOTriton 0.12.1b (from pytorch#188242), which this branch merges in. What is not enabled yet: - CK SDPA on gfx1250: composable_kernel has no gfx1250 support; the CK SDPA target auto-filters to gfx942/gfx950. - CK GEMM on gfx1250: filtered out (see above) until composable_kernel supports it. - FP8 grouped GEMM on gfx1250: MSLK builds only gfx942/gfx950. - int4 mm on gfx1250: the tinygemm MFMA kernel needs a WMMA port; the ops return a clear "not supported yet" error and the int4 unit tests skip gfx1250. CI note: - gfx1250 is not added to the ROCm 7.2 (rocm-n) docker image, whose HIP compiler cannot target gfx1250. gfx1250 CI belongs on the nightly ROCm image and is added in a follow-up. Co-authored-by: @glen-amd ## References - Upstream PR: pytorch#188597 - Related AOTriton bump: pytorch#188242 --------- Co-authored-by: Prachi Gupta <prachi.gupta@amd.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Jeff Daily <jeff.daily@amd.com>
Validation theRock Run: https://github.com/ROCm/TheRock/actions/runs/29045354548 [ROCm] Bump AOTriton to 0.13b Notable new features compared with 0.12.1b: * Debug printing can be enabled with env var `AOTRITON_DEBUG_LEVEL`, without re-compiling + We use the same logging level as of `AMD_LOG_LEVEL` - CAVEAT: AOTriton itself does not read `AMD_LOG_LEVEL` + Set `AOTRITON_DEBUG_LEVEL=5` to print everything * **BREAKING** `.aks2` files are packed into **uncompressed** zip files + `chcp 65001` is not required to build on Win32 any more. * TheRock 7.15 support Bug Fixes: Add `check_requires_grad_and_nested` back to `can_use_mem_efficient_attention`, which was removed accidentally. *Other changes are intentionally left out: they consist mainly of internal refactoring that is not perceivable to pytorch users.* [ROCm] Enable gfx1250 SDPA with AOTriton 0.12.1b (pytorch#188612) Notable New Features: * Tech Preview support of gfx1250 hardware + For broader testing, `TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1` is NOT NEEDED for gfx1250 at the moment. + This support tier is tracked by new function `aotriton::isArchTechPreview()` [ROCm] Bump AOTriton to 0.12b (pytorch#184288) Notable new features: * **BREAKING** Varlen LSE tensor shape changes to (H, Total_seqlen) * Support head_dim != head_dim_v * Support `use_deterministic_algorithims` * Support seqused_k in test/test_varlen_attention.py * gfx1100 and gfx1151 promoted out of experimental * Partial FAv3 support on gfx950 Bug Fixes: * GQA kernel failed to read bias tensor with the right offset. Known Issues * gfx950's Triton kernel has problem handling hdim=16's fwd, in addition to hdim=48/80's bwd. * Disables gfx90a's CK SDPA support due to GPU Segfault. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…totune fallback (#3409) Cherry-pick of pytorch#180892 (30e2dee) minus the mix_order_reduction_allow_multi_stages default flip. The mix_order_reduction_allow_multi_stages flip is an unrelated mix-order-reduction perf feature that the pytorch#17189 conv OOM fix doesn't need — the num_stages=1 fallback delivers that on its own. It was dropped because it's tuned for NVIDIA's ~228 KB shared memory and on gfx942's 64 KB LDS would mostly cause double-compile churn and new overflow risk with no proven benefit, keeping this a conservative, fix-only backport. When every Triton config for a choice fails to build with an OutOfResources/OutOfMemoryError and pipelining is active (num_stages > 1), CachingAutotuner now recompiles that config with num_stages=1 and retries instead of raising. On gfx942 (64KB LDS) this recovers the BLOCK_K=64/128 triton_convolution2d configs that otherwise overflow shared memory, eliminating the "No valid triton configs ... out of resource" autotuning errors seen in ROCm/frameworks-internal#17189 and keeping those kernels available as real choices. Excludes the mix_order_reduction_allow_multi_stages opt-in -> default-on change from the original PR to keep this a fix-only backport. --------- Co-authored-by: shunting314 <shunting@meta.com>
This test needs to be more robust because a previous test's behaviour can impact this test's behaviour.
A specific case where we ran into this situation is on gfx1101, where the test fails with
```
test/test_cuda.py::TestCuda::test_set_per_process_memory_fraction FAILED
[0.1163s]
Traceback (most recent call last):
File "/var/lib/jenkins/pytorch/test/test_cuda.py", line 471, in test_set_per_process_memory_fraction
tmp_tensor = torch.empty(application, dtype=torch.int8, device="cuda")
RuntimeError: Trying to create tensor with negative dimension -5681285432: [-5681285432]
```
This error is coming from an integer overflow when another unit test,
test/test_cuda.py::TestCuda::test_randint_generation_for_large_numel
creates a tensor with a huge numel, which overflows into a higher
torch.cuda.max_memory_reserved() when you call
test/test_cuda.py::TestCuda::test_set_per_process_memory_fraction
afterward. To avoid this we introduced torch.cuda.empty_cache() to clean up CUDA states.
Pull Request resolved: pytorch#181605
Approved by: https://github.com/liangel-02, https://github.com/jeffdaily
(cherry picked from commit 1a4c7f4)
…st_cuda.py (#1… (#3429) …81992) test_garbage_collect_expandable left expandable_segments:True in the runtime allocator config because parseArgs doesn't auto-reset that sticky knob. Later tests like test_allocate_in_thread_to_pool then saw a runtime/EXPANDABLE_SEGMENTS mismatch, making pytorch#158764 flaky. The toggling tests now restore the suite's baseline (EXPANDABLE_SEGMENTS) in their finally; test_cuda_expandable_segments.py sets the env var before importing test_cuda; and TestCudaAllocator gets a tearDown that fails fast if any future test leaks the knob. Fixes pytorch#158764 Fixes pytorch#180263 Authored with Claude. Pull Request resolved: pytorch#181992 Approved by: https://github.com/albanD (cherry picked from commit 6dd5f10) Co-authored-by: Jeff Daily <jeff.daily@amd.com>
ROCm/TheRock#5866 Updating pins for python 3.14 because the Rock CI can't build binary dependencies from source, therefore needs to bump several dependencies to a minimal version that ships python 3.14 binaries. Example workflow run, dependency installed successfully on python 3.14 and 3.13: https://github.com/ROCm/TheRock/actions/runs/29517645214 https://github.com/ROCm/TheRock/actions/runs/29517795234
…'HIPCUB_CCCL_VERSION' (#3441) Cherry-pick of pytorch#188072 (commit c8e2473) onto `release/2.12`. ## Summary - Introduce dependency on libhipcxx - Guard `FpLimits` for `c10::BFloat16` backport to pre CCCL 3.x - Extend `CUB_VERSION` derivation for hipCUB using `HIPCUB_CCCL_VERSION` ## Notes This PR has merge conflicts that need to be resolved manually (conflict markers committed as requested). ## Test plan - [x] Build with ROCm - We were able to run the core PyTorch tests, and verify that they are passing here: https://github.com/ROCm/TheRock/actions/runs/29543027958 - [x] Verify hipCUB/CCCL compatibility glue compiles Authored with assistance from Cursor Made with [Cursor](https://cursor.com) --------- Co-authored-by: Nara Prasetya <nara@streamhpc.com> Co-authored-by: Stanley Tsang <stanley.tsang@amd.com>
Solves ROCm/TheRock#6723 In the rock core UT workflow, fbscribelogger's transitive dependency thriftpy2 doesn't ship a python 3.14 prebuilt wheel and causing installation to fail because there isn't any compiler available in the environment. This wasn't observed in previous dependency bump PR runs because those were run on the `test_pytorch_wheels_full` workflow, where a compiler is available, but some headers and libraries are missing. Presumably this dependency doesn't need any of those missing components and therefore able to be built. Example core UT workflow run: https://github.com/ROCm/TheRock/actions/runs/29853620514 (installation succeeded)
## Summary - Advance the ROCm Triton pin to the tip of `ROCm/triton` `release/internal/3.8.x`. - Update `.ci/docker/triton_version.txt` from `3.7.1` to `3.8.0`. Triton pin: `285c85383660b2e4aac38d4947d23aa61dab3ce2` Triton branch: `ROCm/triton@release/internal/3.8.x` ## Test plan - [x] `git diff --check -- .ci/docker/ci_commit_pins/triton.txt .ci/docker/triton_version.txt .ci/docker/common/install_triton.sh` - [x] Verified Triton checkout reports `__version__ = '3.8.0'` - [ ] Run PyTorch CI Follows the structure of #3360. Made with [Cursor](https://cursor.com) --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…3_IFU_20260722 # Conflicts: # .ci/docker/ci_commit_pins/triton.txt # .ci/docker/requirements-ci.txt # .ci/docker/requirements-docs.txt # .ci/docker/triton_version.txt # .ci/manywheel/build_cuda.sh # .ci/pytorch/build.sh # .ci/pytorch/common_utils.sh # .github/ci_commit_pins/xla.txt # .github/scripts/filter_test_configs.py # .github/templates/common.yml.j2 # .github/templates/linux_binary_build_workflow.yml.j2 # .github/templates/macos_binary_build_workflow.yml.j2 # .github/templates/windows_binary_build_workflow.yml.j2 # .github/workflows/_binary-build-flash-attention-wheel-linux.yml # .github/workflows/_binary-build-flash-attention-wheel-windows.yml # .github/workflows/_binary-build-linux.yml # .github/workflows/_binary-test-linux.yml # .github/workflows/_binary-upload.yml # .github/workflows/_docs.yml # .github/workflows/_lint.yml # .github/workflows/_linux-build.yml # .github/workflows/_linux-test-stable-fa3.yml # .github/workflows/_linux-test.yml # .github/workflows/_mac-build.yml # .github/workflows/_mac-test.yml # .github/workflows/_rocm-test.yml # .github/workflows/_vllm-benchmark.yml # .github/workflows/_win-build.yml # .github/workflows/_win-test.yml # .github/workflows/_xpu-test.yml # .github/workflows/b200-distributed.yml # .github/workflows/b200-symm-mem.yml # .github/workflows/build-almalinux-images.yml # .github/workflows/build-manywheel-images-s390x.yml # .github/workflows/build-manywheel-images.yml # .github/workflows/build-triton-wheel.yml # .github/workflows/build-vllm-wheel.yml # .github/workflows/claude-autorevert-advisor.yml # .github/workflows/claude-code.yml # .github/workflows/claude-issue-triage-run.yml # .github/workflows/close-nonexistent-disable-issues.yml # .github/workflows/create_release.yml # .github/workflows/docker-builds.yml # .github/workflows/docker-cache-rocm.yml # .github/workflows/docker-release.yml # .github/workflows/dtensor.yml # .github/workflows/dynamo-unittest.yml # .github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml # .github/workflows/generated-linux-binary-manywheel-nightly.yml # .github/workflows/generated-linux-s390x-binary-manywheel-nightly.yml # .github/workflows/generated-macos-arm64-binary-wheel-nightly.yml # .github/workflows/generated-windows-arm64-binary-libtorch-debug-nightly.yml # .github/workflows/generated-windows-arm64-binary-wheel-nightly.yml # .github/workflows/generated-windows-binary-libtorch-debug-nightly.yml # .github/workflows/generated-windows-binary-wheel-nightly.yml # .github/workflows/h100-cutlass-backend.yml # .github/workflows/h100-distributed.yml # .github/workflows/h100-symm-mem.yml # .github/workflows/inductor-micro-benchmark.yml # .github/workflows/inductor-nightly.yml # .github/workflows/inductor-pallas.yml # .github/workflows/inductor-perf-compare.yml # .github/workflows/inductor-perf-test-b200.yml # .github/workflows/inductor-perf-test-nightly-aarch64.yml # .github/workflows/inductor-perf-test-nightly-h100.yml # .github/workflows/inductor-perf-test-nightly-rocm-mi300.yml # .github/workflows/inductor-perf-test-nightly-rocm-mi355.yml # .github/workflows/inductor-perf-test-nightly-x86-zen.yml # .github/workflows/inductor-perf-test-nightly-x86.yml # .github/workflows/inductor-perf-test-nightly-xpu.yml # .github/workflows/inductor-perf-test-nightly.yml # .github/workflows/inductor-periodic.yml # .github/workflows/inductor-rocm-mi200.yml # .github/workflows/inductor-rocm-mi300.yml # .github/workflows/inductor-rocm-mi355.yml # .github/workflows/inductor-unittest.yml # .github/workflows/inductor.yml # .github/workflows/lint-bc.yml # .github/workflows/lint.yml # .github/workflows/llm_td_retrieval.yml # .github/workflows/nightly-s3-uploads.yml # .github/workflows/nightly.yml # .github/workflows/nitpicker.yml # .github/workflows/operator_microbenchmark.yml # .github/workflows/periodic-rocm-mi200.yml # .github/workflows/periodic-rocm-mi300.yml # .github/workflows/periodic-rocm-mi355.yml # .github/workflows/periodic.yml # .github/workflows/pull.yml # .github/workflows/quantization-periodic.yml # .github/workflows/rocm-mi200.yml # .github/workflows/rocm-mi300.yml # .github/workflows/rocm-mi355.yml # .github/workflows/rocm-navi31.yml # .github/workflows/rocm-nightly.yml # .github/workflows/slow-rocm-mi200.yml # .github/workflows/slow.yml # .github/workflows/target-determination-indexer.yml # .github/workflows/target_determination.yml # .github/workflows/test-b200.yml # .github/workflows/test-check-binary.yml # .github/workflows/test-h100.yml # .github/workflows/tools-unit-tests.yml # .github/workflows/torchbench.yml # .github/workflows/trunk-rocm-sandbox.yml # .github/workflows/trunk.yml # .github/workflows/unstable.yml # .github/workflows/update-viablestrict.yml # .github/workflows/update_pytorch_labels.yml # .github/workflows/upload-test-stats-while-running.yml # .github/workflows/upload-test-stats.yml # .github/workflows/upload-torch-dynamo-perf-stats.yml # .github/workflows/upload_test_stats_intermediate.yml # .github/workflows/vllm-benchmark.yml # .github/workflows/weekly.yml # .github/workflows/xpu.yml # aten/src/ATen/CMakeLists.txt # aten/src/ATen/cuda/detail/CUDAHooks.cpp # aten/src/ATen/native/ConvUtils.h # aten/src/ATen/native/Onehot.cpp # aten/src/ATen/native/cuda/MemoryAccess.cuh # aten/src/ATen/native/miopen/LossCTC_miopen.cpp # aten/src/ATen/native/mps/kernels/Attention.metal # aten/src/ATen/native/mps/operations/Attention.mm # aten/src/ATen/native/mps/operations/Convolution.mm # aten/src/ATen/native/transformers/cuda/sdp_utils.cpp # aten/src/ATen/native/transformers/hip/flash_attn/ck/CMakeLists.txt # aten/src/ATen/native/transformers/hip/flash_attn/ck/fav_v3/CMakeLists.txt # cmake/External/aotriton.cmake # related_commits # requirements-build.txt # requirements.txt # scripts/release/apply-release-changes.sh # test/distributed/test_c10d_nccl.py # test/dynamo/test_autograd_function.py # test/dynamo/test_compile.py # test/functorch/test_leaf_function.py # test/inductor/test_compiled_autograd.py # test/inductor/test_max_autotune.py # test/inductor/test_max_autotune_blackwell.py # test/inductor/test_torchinductor_opinfo_properties.py # test/inductor/test_triton_heuristics.py # test/inductor/test_user_streams.py # test/test_bmm_outer_product.py # test/test_cuda.py # test/test_cuda_expandable_segments.py # test/test_fake_tensor.py # test/test_linalg.py # test/test_nn.py # test/test_opaque_obj_v2.py # test/test_public_bindings.py # test/test_scatter_gather_ops.py # test/test_testing.py # third_party/xpu.txt # tools/stats/import_test_stats.py # torch/_dynamo/eval_frame.py # torch/_dynamo/variables/streams.py # torch/_inductor/codegen/cuda/device_op_overrides.py # torch/_inductor/codegen/triton.py # torch/_inductor/codegen/triton_combo_kernel.py # torch/_inductor/runtime/hints.py # torch/_inductor/runtime/triton_heuristics.py # torch/_inductor/scheduler.py # torch/_inductor/stream_utils.py # torch/_inductor/utils.py # torch/_native/ops/__init__.py # torch/_native/ops/bmm_outer_product/triton_impl.py # torch/csrc/inductor/aoti_torch/generated/c_shim_mps.h # torch/csrc/inductor/static_launcher/cuda.cpp # torch/csrc/shim_common.cpp # torch/csrc/stable/c/shim.h # torch/csrc/stable/c/shim_function_versions.txt # torch/csrc/stable/library.h # torch/cuda/__init__.py # torch/distributed/fsdp/_fully_shard/_fsdp_param_group.py # torch/headeronly/build.bzl # torch/testing/_internal/common_cuda.py # torchgen/aoti/fallback_ops.py # version.txt
pablo-garay
requested review from
jataylo,
jeffdaily,
jithunnair-amd and
pruthvistony
as code owners
July 22, 2026 20:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rocm_base: a6ab526
IFU: merges release/2.12 into release/2.13.
Recreated from the prior IFU merge commit (5d4b554); targeted explicitly at ROCm/pytorch release/2.13.