RuntimeError: CUDA error: an illegal memory access was encountered

[albertz]

RETURNN starting up, version 1.20260227.073346+git.9e93588a, date/time 2026-03-03-11-05-14 (UTC+0100), pid 79310, cwd /rwthfs/rz/cluster/hpcwork/p0023
565/mwk22690/setups/2025-11-10-start/work/i6_core/returnn/training/ReturnnTrainingJob.OuK06vxsrJJq/work, Python /home/az668407/work/py-envs/py3.12-tor
ch2.7/bin/python3
RETURNN command line options: ['/rwthfs/rz/cluster/home/mwk22690/setups/2025-11-10-start/work/i6_core/returnn/training/ReturnnTrainingJob.OuK06vxsrJJq
/output/returnn.config']
Hostname: n23g0010.hpc.itc.rwth-aachen.de
...
PyTorch: 2.7.1+cu126 (e2d141dbde55c2a4370fac5165b0561b6af4798b) (<site-package> in /home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-pac
kages/torch)
CUDA_VISIBLE_DEVICES=0
MKL_EXAMPLES=/cvmfs/software.hpc.rwth.de/Linux/RH9/x86_64/intel/sapphirerapids/software/imkl/2024.2.0/mkl/2024.2/share/doc/mkl/examples
CUDA_PATH=/cvmfs/software.hpc.rwth.de/Linux/RH9/x86_64/intel/sapphirerapids/software/CUDA/12.6.3
CUDA_ROOT=/cvmfs/software.hpc.rwth.de/Linux/RH9/x86_64/intel/sapphirerapids/software/CUDA/12.6.3
OMP_NUM_THREADS=24
CUDA_HOME=/cvmfs/software.hpc.rwth.de/Linux/RH9/x86_64/intel/sapphirerapids/software/CUDA/12.6.3
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
MKL_NUM_THREADS=24
CUDA_VISIBLE_DEVICES is set to '0'.
Num NVML devices: 1
Available CUDA devices:
  1/1: cuda:0
       name: NVIDIA H100
       total_memory: 93.1GB
       free_memory: 92.6GB (99%)
       capability: 9.0
       device_index: 0
       uuid: c67fb24c-9d87-f003-79a7-5aed55c54f76
       nvml_device_index: 0
RETURNN global startup callback.
/w0/tmp disk usage: total 695.8GB, used 54.0GB, free 641.9GB
Total freed space: 0B

...
ep 18 train, step 52, no_collapse_ctc 1.580, no_collapse_ctc_err 0.363, ctc_4 1.580, ctc_err_4 0.363, ctc_10 1.339, ctc_err_10 0.287, ctc_16 1.176, ctc_err_16 0.252, ce 0.866, fer 0.160, num_seqs 28, max_size:time 266552, max_size:out-spatial 69, mem_usage:cuda 52.4GB, 0.305 sec/step, elapsed 0:00:28, exp. remaining 2:57:08, complete 0.27%
ep 18 train, step 53, txt_ctc_10 1.032, txt_ctc_err_10 0.157, txt_ctc_16 0.944, txt_ctc_err_16 0.149, txt_ce 0.916, txt_fer 0.135, grad_norm:p2 11.455, num_seqs 200, max_size:time 0, max_size:out-spatial 32, mem_usage:cuda 52.4GB, 0.280 sec/step, elapsed 0:00:29, exp. remaining 2:58:40, complete 0.27%
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

...
  File "/rwthfs/rz/cluster/home/mwk22690/setups/2025-11-10-start/tools/returnn/returnn/torch/frontend/_backend.py", line 755, in TorchBackend.ctc_loss
    line: loss_raw = torch.nn.functional.ctc_loss(
              log_probs=log_probs,
              targets=targets_raw,
              input_lengths=input_lengths,
              target_lengths=targets_lengths,
              blank=blank_index,
              zero_infinity=True,
              reduction="none",
          )
    locals:
      log_probs = <local> <torch.Tensor: repr-error RuntimeError: CUDA error: an illegal memory access was encountered
                          CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
                          For debugging consider passing CUDA_LAUNCH_BLOCKING=1
                          Compile with `TORCH_USE_CUDA_D...
      targets = <local> Tensor{'text', [B,T|'out-spatial'[B]], dtype='int32', sparse_dim=Dim{F'vocab'(10240)}}
      targets_raw = <local> <torch.Tensor: repr-error RuntimeError: CUDA error: an illegal memory access was encountered
                            CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
                            For debugging consider passing CUDA_LAUNCH_BLOCKING=1
                            Compile with `TORCH_USE_CUDA_D...
      input_lengths = <local> tensor[28] i32 x∈[228, 280] μ=254.179 σ=23.777
      targets_lengths = <local> tensor[28] i32 x∈[35, 66] μ=50.429 σ=9.183
      blank_index = <local> 10240
  File "/home/az668407/work/py-envs/py3.12-torch2.7/lib/python3.12/site-packages/torch/nn/functional.py", line 3079, in ctc_loss
    line: return torch.ctc_loss(
              log_probs,
              targets,
              input_lengths,
              target_lengths,
              blank,
              _Reduction.get_enum(reduction),
              zero_infinity,
          )
    locals:
      torch.ctc_loss = <global> <built-in method ctc_loss of type object at 0x14b283689fa0>
      log_probs = <local> <torch.Tensor: repr-error RuntimeError: CUDA error: an illegal memory access was encountered
                          CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
                          For debugging consider passing CUDA_LAUNCH_BLOCKING=1
                          Compile with `TORCH_USE_CUDA_D...
      targets = <local> <torch.Tensor: repr-error RuntimeError: CUDA error: an illegal memory access was encountered
                        CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
                        For debugging consider passing CUDA_LAUNCH_BLOCKING=1
                        Compile with `TORCH_USE_CUDA_D...
      input_lengths = <local> tensor[28] i32 x∈[228, 280] μ=254.179 σ=23.777
      target_lengths = <local> tensor[28] i32 x∈[35, 66] μ=50.429 σ=9.183
      blank = <local> 10240
      reduction = <local> 'none'
      zero_infinity = <local> True
RuntimeError: CUDA error: an illegal memory access was encountered

[albertz]

Just reporting this here in case we get this again more often.

It's not totally clear whether ctc_loss is really the cause here.

It might just be some hardware / cluster hiccup.

[albertz]

Note, I think it could be related to #1811, as this is a very similar setup.

[albertz]

Ok, same setup, another crash:

RETURNN starting up, version 1.20260301.203315+git.e59eada9, date/time 2026-03-03-18-06-28 (UTC+0100), pid 33487, cwd /rwthfs/rz/cluster/hpc
work/p0023565/mwk22690/setups/2025-11-10-start/work/i6_core/returnn/training/ReturnnTrainingJob.hcdWrWBixZQX/work, Python /home/az668407/wor
k/py-envs/py3.12-torch2.7/bin/python3
RETURNN command line options: ['/rwthfs/rz/cluster/home/mwk22690/setups/2025-11-10-start/work/i6_core/returnn/training/ReturnnTrainingJob.hc
dWrWBixZQX/output/returnn.config']
Hostname: r23g0004.hpc.itc.rwth-aachen.de
...
Num NVML devices: 1
Available CUDA devices:
  1/1: cuda:0
       name: NVIDIA H100
       total_memory: 93.1GB
       free_memory: 92.6GB (99%)
       capability: 9.0
       device_index: 0
       uuid: c399d568-7c1d-bc30-ab84-97a14adc8197
       nvml_device_index: 0
...
ep 14 train, step 4, no_collapse_ctc 2.220, no_collapse_ctc_err 0.465, pseudo_speech_enc_mse 0.845, ctc_4 2.220, ctc_err_4 0.465, ctc_10 1.9
05, ctc_err_10 0.381, ctc_16 1.749, ctc_err_16 0.364, ce 1.139, fer 0.229, num_seqs 28, max_size:time 95305, max_size:out-spatial 22, mem_us
age:cuda 28.2GB, 0.212 sec/step, elapsed 0:00:10, exp. remaining 10:36:09, complete 0.03%
ep 14 train, step 5, txt_ctc_10 1.342, txt_ctc_err_10 0.211, txt_ctc_16 1.211, txt_ctc_err_16 0.195, txt_ce 1.189, txt_fer 0.181, grad_norm:
p2 14.029, num_seqs 200, max_size:time 0, max_size:out-spatial 68, mem_usage:cuda 44.5GB, 0.524 sec/step, elapsed 0:00:11, exp. remaining 10
:43:36, complete 0.03%
ep 14 train, step 6, no_collapse_ctc 1.817, no_collapse_ctc_err 0.407, pseudo_speech_enc_mse 0.834, ctc_4 1.817, ctc_err_4 0.407, ctc_10 1.5
05, ctc_err_10 0.317, ctc_16 1.408, ctc_err_16 0.302, ce 0.992, fer 0.203, num_seqs 32, max_size:time 105600, max_size:out-spatial 33, mem_u
sage:cuda 44.5GB, 0.257 sec/step, elapsed 0:00:11, exp. remaining 8:07:19, complete 0.04%
ep 14 train, step 7, txt_ctc_10 1.348, txt_ctc_err_10 0.207, txt_ctc_16 1.207, txt_ctc_err_16 0.192, txt_ce 1.203, txt_fer 0.184, grad_norm:
p2 14.110, num_seqs 200, max_size:time 0, max_size:out-spatial 75, mem_usage:cuda 50.1GB, 0.576 sec/step, elapsed 0:00:12, exp. remaining 8:
17:43, complete 0.04%
ep 14 train, step 8, no_collapse_ctc 2.221, no_collapse_ctc_err 0.464, pseudo_speech_enc_mse 0.844, ctc_4 2.221, ctc_err_4 0.464, ctc_10 1.8
89, ctc_err_10 0.384, ctc_16 1.726, ctc_err_16 0.355, ce 1.137, fer 0.218, num_seqs 28, max_size:time 130065, max_size:out-spatial 36, mem_u
sage:cuda 50.1GB, 0.229 sec/step, elapsed 0:00:12, exp. remaining 6:56:59, complete 0.05%
ep 14 train, step 9, txt_ctc_10 1.257, txt_ctc_err_10 0.197, txt_ctc_16 1.126, txt_ctc_err_16 0.183, txt_ce 1.104, txt_fer 0.165, grad_norm:
p2 22.255, num_seqs 200, max_size:time 0, max_size:out-spatial 41, mem_usage:cuda 50.1GB, 0.340 sec/step, elapsed 0:00:12, exp. remaining 7:
00:12, complete 0.05%
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

...
  File "/rwthfs/rz/cluster/home/mwk22690/setups/2025-11-10-start/tools/returnn/returnn/torch/frontend/_backend.py", line 545, in TorchBacken
d.pad
    line: left + middle.get_size_tensor(device=out.device),
    locals:
      left = <local> 0
      middle = <local> Dim{'out-spatial'[B]}
      out = <local> Tensor{'text', [B,T|'(out-spatial)+1'[B]], dtype='int32', sparse_dim=Dim{F'vocab'(10240)}}
      out.device = <local> 'cuda:0', len = 6
  File "/rwthfs/rz/cluster/home/mwk22690/setups/2025-11-10-start/tools/returnn/returnn/tensor/_dim_extra.py", line 1941, in _DimMixin.get_size_tensor
    line: return self.get_dyn_size_ext_for_device(device)
    locals:
      self = <local> Dim{'out-spatial'[B]}
      device = <local> 'cuda:0', len = 6
  File "/rwthfs/rz/cluster/home/mwk22690/setups/2025-11-10-start/tools/returnn/returnn/tensor/_dim_extra.py", line 837, in _DimMixin.get_dyn_size_ext_for_device
    line: self._extra.cache_dyn_size_ext_dev[device] = rf.copy_to_device(self.dyn_size_ext, device=device)
    locals:
      self = <local> Dim{'out-spatial'[B]}
      self._extra = <local> <returnn.tensor._dim_extra._DimExtra object at 0x14d2bd8aeab0>
      self._extra.cache_dyn_size_ext_dev = <local> {}
      device = <local> 'cuda:0', len = 6
      self.dyn_size_ext = <local> Tensor{'out-spatial', [B], dtype='int32'}
  File "/rwthfs/rz/cluster/home/mwk22690/setups/2025-11-10-start/tools/returnn/returnn/frontend/device.py", line 40, in copy_to_device
    line: return x._raw_backend.copy_to_device(x, device)
    locals:
      x = <local> Tensor{'out-spatial', [B], dtype='int32'}
      x._raw_backend = <local> <class returnn.torch.frontend._backend.TorchBackend>
      x._raw_backend.copy_to_device = <local> <function returnn.torch.frontend._backend.TorchBackend.copy_to_device>
      device = <local> 'cuda:0', len = 6
  File "/rwthfs/rz/cluster/home/mwk22690/setups/2025-11-10-start/tools/returnn/returnn/torch/frontend/_backend.py", line 169, in TorchBackend.copy_to_device
    line: x.raw_tensor = x.raw_tensor.to(device)
    locals:
      x = <local> Tensor{'out-spatial', [B], dtype='int32'}
      x.raw_tensor = <local> tensor[28] i32 x∈[17, 36] μ=27.143 σ=5.421
      device = <local> 'cuda:0', len = 6
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

This time, not in ctc_loss.

But, these errors are async, so hard to tell from this log.