Crash when running DeepVariant on GPU

[gergo-hollo]

Hey there! I've been trying to use DeepVariant fast pipeline based on the case study, but I have problems running it. Thanks for the time taking to help!

Issue
I've followed the NVIDIA CUDA Installation Guide for Linux and the NVIDIA Container Toolkit Installation Guide before running the pipeline.

Running nvidia-smi in the container works properly:

singularity exec --oci --nv ./singularity/deepvariant-gpu.oci.sif nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 590.48.01              Driver Version: 590.48.01      CUDA Version: 13.1     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5060 Ti     On  |   00000000:01:00.0 Off |                  N/A |
|  0%   29C    P8              3W /  180W |       2MiB /  16311MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

However, when I try running it, this is the error I get after a few seconds:

File "tmp/Bazel.runfiles_zp5s9wsq/runfiles/com_google_deepvariant/deepvariant/call_variants.py", line 1116, in <module>
      app.run(main)
    File "tmp/Bazel.runfiles_zp5s9wsq/runfiles/absl_py/absl/app.py", line 312, in run
      _run_main(main, args)
    File "tmp/Bazel.runfiles_zp5s9wsq/runfiles/absl_py/absl/app.py", line 258, in _run_main
      sys.exit(main(argv))
    File "tmp/Bazel.runfiles_zp5s9wsq/runfiles/com_google_deepvariant/deepvariant/call_variants.py", line 1092, in main
      call_variants(
    File "tmp/Bazel.runfiles_zp5s9wsq/runfiles/com_google_deepvariant/deepvariant/call_variants.py", line 960, in call_variants
      for distributed_inputs in dist_dataset:
    File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/distribute/input_lib.py", line 264, in __next__
      return self.get_next()
    File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/distribute/input_lib.py", line 349, in get_next
      num_replicas_with_values > 0, _value_or_dummy, _eof, strict=True)
    File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/ops/tensor_math_operator_overrides.py", line 150, in wrapper
      return fn(x, y, *args, **kwargs)
    File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 4278, in greater
      _ops.raise_from_not_ok_status(e, name)
    File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/ops.py", line 5983, in raise_from_not_ok_status
      raise core._status_to_exception(e) from None  # pylint: disable=protected-access
  tensorflow.python.framework.errors_impl.InternalError: {{function_node __wrapped__Greater_device_/job:localhost/replica:0/task:0/device:GPU:0}} 'cuLaunchKernel(function, gridX, gridY, gridZ, blockX, blockY, blockZ, 0, reinterpret_cast<CUstream>(stream), params, nullptr)' failed with 'CUDA_ERROR_INVALID_HANDLE' [Op:Greater] name: 
  I0000 00:00:1770812529.947133      40 fast_pipeline.cc:211] postprocess_variants_bin: "/opt/deepvariant/bin/postprocess_variants"
  I0000 00:00:1770812529.947147      40 fast_pipeline.cc:212] Spawning postprocess_variants process
  I0000 00:00:1770812529.948255      40 fast_pipeline.cc:216] postprocess_variants process stared
  2026-02-11 12:22:10.497911: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
  2026-02-11 12:22:10.520597: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
  To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
  2026-02-11 12:22:10.899907: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
  2026-02-11 12:22:11.593359: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:282] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
  2026-02-11 12:22:11.593377: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:134] retrieving CUDA diagnostic information for host: pop-os
  2026-02-11 12:22:11.593381: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:141] hostname: pop-os
  2026-02-11 12:22:11.593441: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:165] libcuda reported version is: INVALID_ARGUMENT: expected %d.%d, %d.%d.%d, or %d.%d.%d.%d form for driver version; got "1"
  2026-02-11 12:22:11.593453: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:169] kernel reported version is: NOT_FOUND: could not find kernel module information in driver version file contents: "NVRM version: NVIDIA UNIX Open Kernel Module for x86_64  590.48.01  Release Build  (dvs-builder@U22-I3-AE18-23-3)  Mon Dec  8 13:05:00 UTC 2025
  GCC version:  gcc version 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04) 
  "
  Traceback (most recent call last):
    File "tmp/Bazel.runfiles_3kht2prg/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 2348, in <module>
      app.run(main)
    File "tmp/Bazel.runfiles_3kht2prg/runfiles/absl_py/absl/app.py", line 312, in run
      _run_main(main, args)
    File "tmp/Bazel.runfiles_3kht2prg/runfiles/absl_py/absl/app.py", line 258, in _run_main
      sys.exit(main(argv))
    File "tmp/Bazel.runfiles_3kht2prg/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 2259, in main
      cvo_paths = get_cvo_paths(_INFILE.value)
    File "tmp/Bazel.runfiles_3kht2prg/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1633, in get_cvo_paths
      raise ValueError(
  ValueError: ('Found multiple file patterns in input filename space: ', './output/HG002.cvo.tfrecord.gz')

Furthermore, interestingly when I run the pipeline using run_deepvariant with the same GPU image, DeepVariant make_examples runs for 7m 30s on 100% CPU. Then starting with call_variants, VRAM goes 100% but with 0% usage. After 5 minutes I get the following error message:

     raise Empty
  _queue.Empty
  Process ForkProcess-15:
  Traceback (most recent call last):
    File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
      self.run()
    File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
      self._target(*self._args, **self._kwargs)
    File "tmp/Bazel.runfiles_ey3g62wy/runfiles/com_google_deepvariant/deepvariant/call_variants.py", line 561, in post_processing
      item = output_queue.get(timeout=300)
    File "/usr/lib/python3.10/multiprocessing/queues.py", line 114, in get
      raise Empty
  _queue.Empty
  Process ForkProcess-13:
  Traceback (most recent call last):
    File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
      self.run()
    File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
      self._target(*self._args, **self._kwargs)
    File "tmp/Bazel.runfiles_ey3g62wy/runfiles/com_google_deepvariant/deepvariant/call_variants.py", line 561, in post_processing
      item = output_queue.get(timeout=300)
    File "/usr/lib/python3.10/multiprocessing/queues.py", line 114, in get
      raise Empty
  _queue.Empty
  Process ForkProcess-14:
  Traceback (most recent call last):
    File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
      self.run()
    File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
      self._target(*self._args, **self._kwargs)
    File "tmp/Bazel.runfiles_ey3g62wy/runfiles/com_google_deepvariant/deepvariant/call_variants.py", line 561, in post_processing
      item = output_queue.get(timeout=300)
    File "/usr/lib/python3.10/multiprocessing/queues.py", line 114, in get
      raise Empty
  _queue.Empty
  Process ForkProcess-16:
  Traceback (most recent call last):
    File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
      self.run()
    File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
      self._target(*self._args, **self._kwargs)
    File "tmp/Bazel.runfiles_ey3g62wy/runfiles/com_google_deepvariant/deepvariant/call_variants.py", line 561, in post_processing
      item = output_queue.get(timeout=300)
    File "/usr/lib/python3.10/multiprocessing/queues.py", line 114, in get
      raise Empty
  _queue.Empty

Parabricks DeepVariant runs without problems.

Setup

• System:
  • OS: PopOS 24.04
  • CPU: AMD Ryzen 9 7900 12-Core (24 threads) AVX512
  • GPU: NVIDIA GeForce RTX 5060 Ti 16GB (driver 590.48.01)
  • CUDA: 13.1
• DeepVariant version: r1.10.0-beta
• Installation method: Singularity image built from docker://google/deepvariant:1.10.0-beta-gpu
• Type of data: WES HG002 from SRR2962669, mapped to GCA_000001405.15_GRCh38_no_alt_analysis_set with Parabricks fq2bam

Steps to reproduce:
Command in Nextflow 25.10.3 (with containerOptions: '--nv'):

    def regions_arg = target_bed == [] ? "" : "--regions ${target_bed}"
    def haploid_arg = params.gender == "male" ? "--haploid_contigs=chrX,chrY" : ""
    def par_regions_arg = params.gender == "male" ? "--par_regions_bed=PAR.bed" : ""
    def model = params.mode.toLowerCase() // Model is 'wes' here
    """
    mkdir -p config
    FILE=config/make_examples.ini

    # Create config 1.
    cat <<EOM >\$FILE
--examples=/tmp/examples.tfrecords@14.gz
--gvcf=/tmp/examples.gvcf.tfrecord@14.gz
--mode=calling
--reads=${bam_input}
--ref=${fasta}
--alt_aligned_pileup=diff_channels
--max_reads_per_partition=600
--min_mapping_quality=1
--parse_sam_aux_fields
--partition_size=25000
--phase_reads
--pileup_image_width=147
--norealign_reads
--sort_by_haplotypes
--track_ref_reads
--vsc_min_fraction_indels=0.12
--trim_reads_for_pileup
--trained_small_model_path=/opt/smallmodels/wgs
--small_model_snp_gq_threshold=25
--small_model_indel_gq_threshold=30
--small_model_vaf_context_window_size=51
--output_phase_info
--checkpoint=/opt/models/${model}
${regions_arg}
EOM

    FILE=config/call_variants.ini

    cat <<EOM >\$FILE
--outfile=./output/${meta.id}.cvo.tfrecord.gz
--checkpoint=/opt/models/${model}
--batch_size=1024
--writer_threads=1
EOM

    FILE=config/postprocess_variants.ini

    cat <<EOM >\$FILE
--ref=${fasta}
--infile=./output/${meta.id}.cvo.tfrecord.gz
--nonvariant_site_tfrecord_path=/tmp/examples.gvcf.tfrecord@14.gz
--outfile=./${meta.id}.vcf.gz
--gvcf_outfile=./${meta.id}.g.vcf.gz
--small_model_cvo_records=/tmp/examples_call_variant_outputs.tfrecords@14.gz
--cpus=${task.cpus}
${haploid_arg}
${par_regions_arg}
EOM

    # Create PAR.bed if male
    if [ "${params.gender}" = "male" ]; then
        cat > PAR.bed <<EOF
chrX	10000	44821
chrX	94821	133871
chrX	222346	226276
chrX	226351	1949345
chrX	2132994	2137388
chrX	2137488	2781479
chrX	155701383	156030895
chrY	10000	44821
chrY	94821	133871
chrY	222346	226276
chrY	226351	1949345
chrY	2132994	2137388
chrY	2137488	2781479
chrY	56887902	57217415
EOF
    fi

    # Bazel is using TMPDIR and writing files there,
    # therefore redirecting that to a writable folder (current workdir).
    export TMPDIR=\$PWD/tmp
    mkdir tmp

    /opt/deepvariant/bin/fast_pipeline \
    --make_example_flags ./config/make_examples.ini \
    --call_variants_flags ./config/call_variants.ini \
    --postprocess_variants_flags ./config/postprocess_variants.ini \
    --shm_prefix dv \
    --num_shards ${task.cpus} \
    --buffer_size 10485760 

run_pipeline command:

    def regions_arg = target_bed == [] ? "" : "--regions ${target_bed}"
    def haploid_arg = params.gender == "male" ? "--haploid_contigs chrX,chrY" : ""
    def par_regions_arg = params.gender == "male" ? "--par_regions_bed PAR.bed" : ""
    """
    # Create PAR.bed if male
    if [ "${params.gender}" = "male" ]; then
        cat > PAR.bed <<EOF
chrX	10000	44821
chrX	94821	133871
chrX	222346	226276
chrX	226351	1949345
chrX	2132994	2137388
chrX	2137488	2781479
chrX	155701383	156030895
chrY	10000	44821
chrY	94821	133871
chrY	222346	226276
chrY	226351	1949345
chrY	2132994	2137388
chrY	2137488	2781479
chrY	56887902	57217415
EOF
    fi

    # Bazel is using TMPDIR and writing files there,
    # therefore redirecting that to a writable folder (current workdir).
    export TMPDIR=\$PWD/tmp
    mkdir tmp

    /opt/deepvariant/bin/run_deepvariant \
    --model_type ${params.mode} \
    --ref ${fasta} \
    --reads ${bam_input} \
    ${regions_arg} \
    --output_vcf ./${meta.id}.output.vcf.gz \
    --output_gvcf ./${meta.id}.output.g.vcf.gz \
    --num_shards ${task.cpus} \
    ${haploid_arg} \
    ${par_regions_arg}

[kishwarshafin]

@gergo-hollo

This pipeline requires you use a clean directory everytime you run it. So once you run under "/tmp/", then if you try again it will see a file pattern exists and it will throw the "multiple file pattern" error.

Can you please use a clean directory, that is not "/tmp/" and try it again?

[gergo-hollo]

Thank you for the quick reply @kishwarshafin!

I am running it with Nextflow within a Singularity image, therefore /tmp should be clean.
Nevertheless, I also tried running it using a different directory path, but I still get the same error message.

[akolesnikov]

Hi @gergo-hollo,

It seems these are two distinct issues. I'll need some time to reproduce the Fast pipeline problem, but I did spot a potential discrepancy:

• make_examples: Hardcoded to 14 shards.
• fast_pipeline: Uses the task.cpus variable.

Could you verify if task.cpus is set to 14 in your environment?

[gergo-hollo]

Hi @akolesnikov,

'task.cpus' was set to 24 in my original run. I tried running it now after setting it to 14, but it led to the same error message.

[akolesnikov]

@gergo-hollo,

I tried running our case study for Fast Pipeline and it worked normally. To further investigate the issue may I ask you to try version 1.9.0?

[gergo-hollo]

Thank you!

I realized the mistake I made was a typo in fast_pipeline: I wrote --regions ${target_bed} instead of --regions=${target_bed}, which led to the error message above.

After fixing this, I tried running the fast pipeline with version 1.9.0. However, after a short burst (~1 min) of CPU usage, DeepVariant stalls and nothing happens:

• 0% GPU,
• 0% CPU, and
• 0% I/O usage

for at least 25m on a WES sample. I see the same result with version 1.10.0-beta using both 24 threads and 14 threads.

Version 1.9.0 run_deepvariant works properly and uses my GPU as expected, but 1.10.0-beta stalls indefinitely after make_examples finishes.

What is particularly interesting is that in the log for 1.10.0-beta there is the line:
2026-02-11 12:22:11.593359: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:282] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected.
However, this line is absent in 1.9.0 run_deepvariant, where the GPU is recognized correctly. In contrast, when running the fast pipeline in 1.9.0, I see this error as well (see log below).

This is the exact command I use for fast pipeline:

Fast Pipeline command

#!/bin/bash -ue
OUTDIR=$PWD/output
mkdir -p $OUTDIR

mkdir -p config
FILE=config/make_examples.ini

# Create config 1.
cat <<EOM >$FILE
--examples=/tmp/examples.tfrecords@14.gz
--gvcf=/tmp/examples.gvcf.tfrecord@14.gz
--mode=calling
--reads=HG004.bam
--ref=GCA_000001405.15_GRCh38_no_alt_analysis_set_noY.fa.gz
--alt_aligned_pileup=diff_channels
--max_reads_per_partition=600
--min_mapping_quality=1
--parse_sam_aux_fields
--partition_size=25000
--phase_reads
--pileup_image_width=147
--norealign_reads
--sort_by_haplotypes
--track_ref_reads
--vsc_min_fraction_indels=0.12
--trim_reads_for_pileup
--call_small_model_examples
--trained_small_model_path=/opt/smallmodels/wgs
--small_model_snp_gq_threshold=25
--small_model_indel_gq_threshold=30
--small_model_vaf_context_window_size=51
--output_phase_info
--checkpoint=/opt/models/wes
--regions=S04380110_Regions.clean.bed
EOM

FILE=config/call_variants.ini

cat <<EOM >$FILE
--outfile=$OUTDIR/HG004.cvo.tfrecord.gz
--checkpoint=/opt/models/wes
--batch_size=1024
--writer_threads=1
EOM

FILE=config/postprocess_variants.ini

cat <<EOM >$FILE
--ref=GCA_000001405.15_GRCh38_no_alt_analysis_set_noY.fa.gz
--infile=$OUTDIR/HG004.cvo.tfrecord.gz
--nonvariant_site_tfrecord_path=/tmp/examples.gvcf.tfrecord@14.gz
--outfile=$OUTDIR/HG004.vcf.gz
--gvcf_outfile=$OUTDIR/HG004.g.vcf.gz
--small_model_cvo_records=/tmp/examples_call_variant_outputs.tfrecords@14.gz
--cpus=24


EOM

# Needed env variable in 1.9.0
export DV_BIN_PATH=/opt/deepvariant/bin

# Bazel is using TMPDIR and writing files there,
# therefore redirecting that to a writable folder (current workdir).
export TMPDIR=$PWD/tmp
mkdir tmp

/opt/deepvariant/bin/fast_pipeline \
    --make_example_flags ./config/make_examples.ini \
    --call_variants_flags ./config/call_variants.ini \
    --postprocess_variants_flags ./config/postprocess_variants.ini \
    --shm_prefix dv \
    --num_shards 24 \
    --buffer_size 10485760

This is the end of the log file before stalling indefinitely in fast pipeline version 1.9.0:

Log Fast Pipeline 1.9.0

2026-02-16 09:10:22.513239: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:268] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2026-02-16 09:10:22.513293: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:168] retrieving CUDA diagnostic information for host: pop-os
2026-02-16 09:10:22.513318: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:175] hostname: pop-os
2026-02-16 09:10:22.513420: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:199] libcuda reported version is: INVALID_ARGUMENT: expected %d.%d, %d.%d.%d, or %d.%d.%d.%d form for driver version; got "1"
2026-02-16 09:10:22.513468: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:203] kernel reported version is: NOT_FOUND: could not find kernel module information in driver version file contents: "NVRM version: NVIDIA UNIX Open Kernel Module for x86_64  590.48.01  Release Build  (dvs-builder@U22-I3-AE18-23-3)  Mon Dec  8 13:05:00 UTC 2025
GCC version:  gcc version 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04) 
"
I0216 09:10:22.513682 128083474571712 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:22.513932 127356011794880 make_examples_core.py:348] Task 3/14: Preparing inputs
I0216 09:10:22.515115 126110666641856 dv_utils.py:332] From /opt/models/wes/example_info.json: Shape of input examples: [100, 221, 7], Channels of input examples: [1, 2, 3, 4, 5, 6, 19].
W0216 09:10:22.515343 126110666641856 make_examples_options.py:1128] --parse_sam_aux_fields is deprecated. 
    HP is parsed if --phase_reads=False and --sort_by_haplotypes, --reverse_haplotypes, or --hp_tag_for_assembly_polishing are set.
    HP is parsed if --phase_reads=False and the 'haplotype' channel is requested.
    HP is parsed if --output_phasing_error_stats is set.
    OQ is parsed if --use_original_quality_scores is set.
    MM, ML, MN are parsed if the "base_methylation" channel is requested or
    if --enable_methylation_calling or if --enable_methylation_aware_phasing
    is set.

I0216 09:10:22.515559 126110666641856 make_examples_options.py:1145] Parsing AUX Fields: []
I0216 09:10:22.515736 127977190990272 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:22.516689 137130836279744 make_examples_core.py:348] Task 9/14: Common contigs are ['chr1', 'chr2', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chr8', 'chr9', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr19', 'chr20', 'chr21', 'chr22', 'chrX', 'chrY', 'chrM']
I0216 09:10:22.516886 128083474571712 dv_utils.py:332] From /opt/models/wes/example_info.json: Shape of input examples: [100, 221, 7], Channels of input examples: [1, 2, 3, 4, 5, 6, 19].
I0216 09:10:22.517033 137781917340096 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
W0216 09:10:22.517105 128083474571712 make_examples_options.py:1128] --parse_sam_aux_fields is deprecated. 
    HP is parsed if --phase_reads=False and --sort_by_haplotypes, --reverse_haplotypes, or --hp_tag_for_assembly_polishing are set.
    HP is parsed if --phase_reads=False and the 'haplotype' channel is requested.
    HP is parsed if --output_phasing_error_stats is set.
    OQ is parsed if --use_original_quality_scores is set.
    MM, ML, MN are parsed if the "base_methylation" channel is requested or
    if --enable_methylation_calling or if --enable_methylation_aware_phasing
    is set.

I0216 09:10:22.517323 128083474571712 make_examples_options.py:1145] Parsing AUX Fields: []
I0216 09:10:22.518405 126110666641856 make_examples_core.py:348] Task 13/14: Preparing inputs
I0216 09:10:22.518648 127977190990272 dv_utils.py:332] From /opt/models/wes/example_info.json: Shape of input examples: [100, 221, 7], Channels of input examples: [1, 2, 3, 4, 5, 6, 19].
W0216 09:10:22.518902 127977190990272 make_examples_options.py:1128] --parse_sam_aux_fields is deprecated. 
    HP is parsed if --phase_reads=False and --sort_by_haplotypes, --reverse_haplotypes, or --hp_tag_for_assembly_polishing are set.
    HP is parsed if --phase_reads=False and the 'haplotype' channel is requested.
    HP is parsed if --output_phasing_error_stats is set.
    OQ is parsed if --use_original_quality_scores is set.
    MM, ML, MN are parsed if the "base_methylation" channel is requested or
    if --enable_methylation_calling or if --enable_methylation_aware_phasing
    is set.

I0216 09:10:22.519154 127977190990272 make_examples_options.py:1145] Parsing AUX Fields: []
I0216 09:10:22.520131 128083474571712 make_examples_core.py:348] Task 11/14: Preparing inputs
I0216 09:10:22.520110 137781917340096 dv_utils.py:332] From /opt/models/wes/example_info.json: Shape of input examples: [100, 221, 7], Channels of input examples: [1, 2, 3, 4, 5, 6, 19].
Traceback (most recent call last):
  File "/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_ayrlo04d/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 240, in <module>
    app.run(main)
  File "/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_ayrlo04d/runfiles/absl_py/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_ayrlo04d/runfiles/absl_py/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_ayrlo04d/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 226, in main
    options = default_options(add_flags=True, flags_obj=FLAGS)
  File "/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_ayrlo04d/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 167, in default_options
    options = make_examples_options.shared_flags_to_options(
  File "/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_ayrlo04d/runfiles/com_google_deepvariant/deepvariant/make_examples_options.py", line 1108, in shared_flags_to_options
    ) = sharded_file_utils.resolve_filespecs(
  File "/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_ayrlo04d/runfiles/com_google_deepvariant/third_party/nucleus/io/sharded_file_utils.py", line 242, in resolve_filespecs
    raise ValueError('Invalid shard={} value with master={} sharding'.format(
ValueError: Invalid shard=16 value with master=/tmp/examples.tfrecords@14.gz sharding
I0216 09:10:22.521162 127356011794880 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:22.521308 127568477659584 make_examples_core.py:348] Task 10/14: Common contigs are ['chr1', 'chr2', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chr8', 'chr9', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr19', 'chr20', 'chr21', 'chr22', 'chrX', 'chrY', 'chrM']
I0216 09:10:22.521527 137130836279744 genomics_reader.py:223] Reading S04380110_Regions.clean.bed with NativeBedReader
I0216 09:10:22.521960 127977190990272 make_examples_core.py:348] Task 6/14: Preparing inputs
I0216 09:10:22.524420 131906621968832 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:22.526366 127568477659584 genomics_reader.py:223] Reading S04380110_Regions.clean.bed with NativeBedReader
I0216 09:10:22.526809 126110666641856 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:22.527110 130125045510592 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:22.527880 131906621968832 dv_utils.py:332] From /opt/models/wes/example_info.json: Shape of input examples: [100, 221, 7], Channels of input examples: [1, 2, 3, 4, 5, 6, 19].
Traceback (most recent call last):
  File "/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_iz1hnn8p/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 240, in <module>
    app.run(main)
  File "/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_iz1hnn8p/runfiles/absl_py/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_iz1hnn8p/runfiles/absl_py/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_iz1hnn8p/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 226, in main
    options = default_options(add_flags=True, flags_obj=FLAGS)
  File "/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_iz1hnn8p/runfiles/com_google_deepvariant/deepvariant/make_examples.py", line 167, in default_options
    options = make_examples_options.shared_flags_to_options(
  File "/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_iz1hnn8p/runfiles/com_google_deepvariant/deepvariant/make_examples_options.py", line 1108, in shared_flags_to_options
    ) = sharded_file_utils.resolve_filespecs(
  File "/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_iz1hnn8p/runfiles/com_google_deepvariant/third_party/nucleus/io/sharded_file_utils.py", line 242, in resolve_filespecs
    raise ValueError('Invalid shard={} value with master={} sharding'.format(
ValueError: Invalid shard=18 value with master=/tmp/examples.tfrecords@14.gz sharding
I0216 09:10:22.529242 128083474571712 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:22.529472 127977190990272 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:22.526575 136409907081664 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:22.530851 127356011794880 make_examples_core.py:348] Task 3/14: Common contigs are ['chr1', 'chr2', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chr8', 'chr9', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr19', 'chr20', 'chr21', 'chr22', 'chrX', 'chrY', 'chrM']
I0216 09:10:22.533781 130125045510592 dv_utils.py:332] From /opt/models/wes/example_info.json: Shape of input examples: [100, 221, 7], Channels of input examples: [1, 2, 3, 4, 5, 6, 19].
W0216 09:10:22.534007 130125045510592 make_examples_options.py:1128] --parse_sam_aux_fields is deprecated. 
    HP is parsed if --phase_reads=False and --sort_by_haplotypes, --reverse_haplotypes, or --hp_tag_for_assembly_polishing are set.
    HP is parsed if --phase_reads=False and the 'haplotype' channel is requested.
    HP is parsed if --output_phasing_error_stats is set.
    OQ is parsed if --use_original_quality_scores is set.
    MM, ML, MN are parsed if the "base_methylation" channel is requested or
    if --enable_methylation_calling or if --enable_methylation_aware_phasing
    is set.

I0216 09:10:22.534233 130125045510592 make_examples_options.py:1145] Parsing AUX Fields: []
I0216 09:10:22.535789 136409907081664 dv_utils.py:332] From /opt/models/wes/example_info.json: Shape of input examples: [100, 221, 7], Channels of input examples: [1, 2, 3, 4, 5, 6, 19].
I0216 09:10:22.535927 127356011794880 genomics_reader.py:223] Reading S04380110_Regions.clean.bed with NativeBedReader
W0216 09:10:22.536019 136409907081664 make_examples_options.py:1128] --parse_sam_aux_fields is deprecated. 
    HP is parsed if --phase_reads=False and --sort_by_haplotypes, --reverse_haplotypes, or --hp_tag_for_assembly_polishing are set.
    HP is parsed if --phase_reads=False and the 'haplotype' channel is requested.
    HP is parsed if --output_phasing_error_stats is set.
    OQ is parsed if --use_original_quality_scores is set.
    MM, ML, MN are parsed if the "base_methylation" channel is requested or
    if --enable_methylation_calling or if --enable_methylation_aware_phasing
    is set.

I0216 09:10:22.536246 136409907081664 make_examples_options.py:1145] Parsing AUX Fields: []
I0216 09:10:22.539258 130125045510592 make_examples_core.py:348] Task 0/14: Preparing inputs
I0216 09:10:22.539255 126110666641856 make_examples_core.py:348] Task 13/14: Common contigs are ['chr1', 'chr2', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chr8', 'chr9', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr19', 'chr20', 'chr21', 'chr22', 'chrX', 'chrY', 'chrM']
I0216 09:10:22.539536 127977190990272 make_examples_core.py:348] Task 6/14: Common contigs are ['chr1', 'chr2', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chr8', 'chr9', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr19', 'chr20', 'chr21', 'chr22', 'chrX', 'chrY', 'chrM']
I0216 09:10:22.541921 128083474571712 make_examples_core.py:348] Task 11/14: Common contigs are ['chr1', 'chr2', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chr8', 'chr9', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr19', 'chr20', 'chr21', 'chr22', 'chrX', 'chrY', 'chrM']
I0216 09:10:22.542235 136409907081664 make_examples_core.py:348] Task 8/14: Preparing inputs
I0216 09:10:22.544601 127977190990272 genomics_reader.py:223] Reading S04380110_Regions.clean.bed with NativeBedReader
I0216 09:10:22.545145 126110666641856 genomics_reader.py:223] Reading S04380110_Regions.clean.bed with NativeBedReader
I0216 09:10:22.548109 128083474571712 genomics_reader.py:223] Reading S04380110_Regions.clean.bed with NativeBedReader
I0216 09:10:22.557734 136409907081664 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:22.557976 130125045510592 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:22.582129 136409907081664 make_examples_core.py:348] Task 8/14: Common contigs are ['chr1', 'chr2', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chr8', 'chr9', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr19', 'chr20', 'chr21', 'chr22', 'chrX', 'chrY', 'chrM']
I0216 09:10:22.582504 130125045510592 make_examples_core.py:348] Task 0/14: Common contigs are ['chr1', 'chr2', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chr8', 'chr9', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr19', 'chr20', 'chr21', 'chr22', 'chrX', 'chrY', 'chrM']
I0216 09:10:22.594803 136409907081664 genomics_reader.py:223] Reading S04380110_Regions.clean.bed with NativeBedReader
I0216 09:10:22.594930 130125045510592 genomics_reader.py:223] Reading S04380110_Regions.clean.bed with NativeBedReader
2026-02-16 09:10:22.598768: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2026-02-16 09:10:22.650118: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2026-02-16 09:10:22.651810: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2026-02-16 09:10:22.657730: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2052] TensorFlow was not built with CUDA kernel binaries compatible with compute capability 12.0. CUDA kernels will be jit-compiled from PTX, which could take 30 minutes or longer.
I0216 09:10:22.657950 134183555793344 call_variants.py:965] call_variants: env = {'NV_LIBCUBLAS_VERSION': '11.11.3.6-1', 'NVIDIA_VISIBLE_DEVICES': 'all', 'NV_NVML_DEV_VERSION': '11.8.86-1', 'NV_CUDNN_PACKAGE_NAME': 'libcudnn8', 'NV_LIBNCCL_DEV_PACKAGE': 'libnccl-dev=2.15.5-1+cuda11.8', 'NV_LIBNCCL_DEV_PACKAGE_VERSION': '2.15.5-1', 'PYTHON_VERSION': '3.10', 'SINGULARITY_NAME': '/bioinformatics/varis_pipeline/././singularity/deepvariant-gpu-1.9.0.oci.sif', 'NVIDIA_REQUIRE_CUDA': 'cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471', 'NV_LIBCUBLAS_DEV_PACKAGE': 'libcublas-dev-11-8=11.11.3.6-1', 'NV_NVTX_VERSION': '11.8.86-1', 'NV_CUDA_CUDART_DEV_VERSION': '11.8.89-1', 'NV_LIBCUSPARSE_VERSION': '11.7.5.86-1', 'NV_LIBNPP_VERSION': '11.8.0.86-1', 'NXF_TASK_WORKDIR': '/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661', 'DV_GPU_BUILD': '1', 'NCCL_VERSION': '2.15.5-1', 'PWD': '/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661', 'NV_CUDNN_PACKAGE': 'libcudnn8=8.9.6.50-1+cuda11.8', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NV_NVPROF_DEV_PACKAGE': 'cuda-nvprof-11-8=11.8.87-1', 'NV_LIBNPP_PACKAGE': 'libnpp-11-8=11.8.0.86-1', 'TF_ENABLE_ONEDNN_OPTS': '1', 'NV_LIBNCCL_DEV_PACKAGE_NAME': 'libnccl-dev', '_': '/usr/bin/python3', 'NV_LIBCUBLAS_DEV_VERSION': '11.11.3.6-1', 'NVIDIA_PRODUCT_NAME': 'CUDA', 'NV_LIBCUBLAS_DEV_PACKAGE_NAME': 'libcublas-dev-11-8', 'NV_CUDA_CUDART_VERSION': '11.8.89-1', 'HOME': '/home/garyraven', 'CUDA_VERSION': '11.8.0', 'SINGULARITY_CONTAINER': '/usr/local/var/singularity/mnt/session/oci-bundle135313511', 'NV_LIBCUBLAS_PACKAGE': 'libcublas-11-8=11.11.3.6-1', 'NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE': 'cuda-nsight-compute-11-8=11.8.0-1', 'TMPDIR': '/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp', 'DV_BIN_PATH': '/opt/deepvariant/bin', 'NV_LIBNPP_DEV_PACKAGE': 'libnpp-dev-11-8=11.8.0.86-1', 'NV_LIBCUBLAS_PACKAGE_NAME': 'libcublas-11-8', 'NV_LIBNPP_DEV_VERSION': '11.8.0.86-1', 'NV_LIBCUSPARSE_DEV_VERSION': '11.7.5.86-1', 'LIBRARY_PATH': '/usr/local/cuda/lib64/stubs', 'NV_CUDNN_VERSION': '8.9.6.50', 'SHLVL': '2', 'NV_CUDA_LIB_VERSION': '11.8.0-1', 'NVARCH': 'x86_64', 'NV_CUDNN_PACKAGE_DEV': 'libcudnn8-dev=8.9.6.50-1+cuda11.8', 'NXF_DEBUG': '0', 'NV_CUDA_COMPAT_PACKAGE': 'cuda-compat-11-8', 'NV_LIBNCCL_PACKAGE': 'libnccl2=2.15.5-1+cuda11.8', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs', 'NV_CUDA_NSIGHT_COMPUTE_VERSION': '11.8.0-1', 'NV_NVPROF_VERSION': '11.8.87-1', 'PATH': '/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/conda/bin:/opt/conda/envs/bio/bin:/opt/deepvariant/bin', 'NV_LIBNCCL_PACKAGE_NAME': 'libnccl2', 'VERSION': '1.9.0', 'NV_LIBNCCL_PACKAGE_VERSION': '2.15.5-1', 'OLDPWD': '/opt/deepvariant', 'LC_CTYPE': 'C.UTF-8', 'PYTHONPATH': '/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_9a3iyf_c/runfiles:/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_9a3iyf_c/runfiles/com_google_protobuf/python:/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_9a3iyf_c/runfiles/absl_py:/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_9a3iyf_c/runfiles/bazel_tools:/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_9a3iyf_c/runfiles/com_google_deepvariant:/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_9a3iyf_c/runfiles/com_google_protobuf:/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_9a3iyf_c/runfiles/six_archive', 'RUNFILES_DIR': '/bioinformatics/varis_pipeline/work/d4/f1f5c13afe1e7bb16c1aef06403661/tmp/Bazel.runfiles_9a3iyf_c/runfiles', 'TPU_ML_PLATFORM': 'Tensorflow', 'TF2_BEHAVIOR': '1', 'TF_CPP_MIN_LOG_LEVEL': '1'}
I0216 09:10:22.685591 134183555793344 call_variants.py:813] Total 1 writing processes started.
I0216 09:10:22.686966 134183555793344 call_variants.py:824] Use saved model: True
2026-02-16 09:10:22.930303: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2026-02-16 09:10:22.932504: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2026-02-16 09:10:22.934536: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2026-02-16 09:10:22.936353: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2052] TensorFlow was not built with CUDA kernel binaries compatible with compute capability 12.0. CUDA kernels will be jit-compiled from PTX, which could take 30 minutes or longer.
2026-02-16 09:10:23.037396: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2026-02-16 09:10:23.038367: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2026-02-16 09:10:23.039297: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2026-02-16 09:10:23.040166: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13913 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 5060 Ti, pci bus id: 0000:01:00.0, compute capability: 12.0
I0216 09:10:27.998553 134183555793344 dv_utils.py:332] From /opt/models/wes/example_info.json: Shape of input examples: [100, 221, 7], Channels of input examples: [1, 2, 3, 4, 5, 6, 19].
I0216 09:10:27.998684 134183555793344 call_variants.py:858] example_shape: [100, 221, 7]
2026-02-16 09:10:28.291162: I deepvariant/stream_examples_kernel.cc:208] Shard 0 is completed
2026-02-16 09:10:28.291266: I deepvariant/stream_examples_kernel.cc:208] Shard 1 is completed
2026-02-16 09:10:28.291315: I deepvariant/stream_examples_kernel.cc:208] Shard 2 is completed
2026-02-16 09:10:28.291329: I deepvariant/stream_examples_kernel.cc:208] Shard 3 is completed
2026-02-16 09:10:28.291368: I deepvariant/stream_examples_kernel.cc:208] Shard 4 is completed
2026-02-16 09:10:28.291412: I deepvariant/stream_examples_kernel.cc:208] Shard 5 is completed
2026-02-16 09:10:28.291462: I deepvariant/stream_examples_kernel.cc:208] Shard 6 is completed
2026-02-16 09:10:28.291511: I deepvariant/stream_examples_kernel.cc:208] Shard 7 is completed
I0216 09:10:28.293783 134183555793344 call_variants.py:592] Wrote 0 examples over 0 batches. Closing writer.
I0216 09:10:28.322436 134183555793344 call_variants.py:1014] Complete: call_variants.
I0216 09:10:43.216613 126744590062016 make_examples_core.py:348] Task 4/14: Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1771233043.216769     213 sam_reader.cc:1001] Setting HTS_OPT_BLOCK_SIZE to 134217728
I0216 09:10:43.259020 126744590062016 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:43.460994 126744590062016 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:43.462895 126744590062016 make_examples_core.py:348] Task 4/14: Writing gvcf records to /tmp/examples.gvcf.tfrecord-00004-of-00014.gz
I0000 00:00:1771233043.464422     213 stream_examples.cc:80] Creating buffer_empty mutex
I0000 00:00:1771233043.464443     213 stream_examples.cc:84] Creating items_available mutex
I0216 09:10:43.716357 128334425899456 make_examples_core.py:348] Task 12/14: Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1771233043.716499     216 sam_reader.cc:1001] Setting HTS_OPT_BLOCK_SIZE to 134217728
I0216 09:10:43.761287 128334425899456 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:43.835329 130125045510592 make_examples_core.py:348] Task 0/14: Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1771233043.835469     193 sam_reader.cc:1001] Setting HTS_OPT_BLOCK_SIZE to 134217728
I0216 09:10:43.865818 140419873022400 make_examples_core.py:348] Task 5/14: Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1771233043.865967     214 sam_reader.cc:1001] Setting HTS_OPT_BLOCK_SIZE to 134217728
I0216 09:10:43.878655 130125045510592 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:43.909608 140419873022400 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:43.928729 128334425899456 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:43.930081 128334425899456 make_examples_core.py:348] Task 12/14: Writing gvcf records to /tmp/examples.gvcf.tfrecord-00012-of-00014.gz
I0000 00:00:1771233043.931001     216 stream_examples.cc:80] Creating buffer_empty mutex
I0000 00:00:1771233043.931021     216 stream_examples.cc:84] Creating items_available mutex
I0216 09:10:43.939594 126110666641856 make_examples_core.py:348] Task 13/14: Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1771233043.939729     199 sam_reader.cc:1001] Setting HTS_OPT_BLOCK_SIZE to 134217728
I0216 09:10:43.983625 126110666641856 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:43.992811 133326641848768 make_examples_core.py:348] Task 2/14: Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1771233043.992953     217 sam_reader.cc:1001] Setting HTS_OPT_BLOCK_SIZE to 134217728
I0216 09:10:44.036770 133326641848768 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:44.050823 130125045510592 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:44.052325 130125045510592 make_examples_core.py:348] Task 0/14: Writing gvcf records to /tmp/examples.gvcf.tfrecord-00000-of-00014.gz
I0000 00:00:1771233044.053300     193 stream_examples.cc:80] Creating buffer_empty mutex
I0000 00:00:1771233044.053318     193 stream_examples.cc:84] Creating items_available mutex
I0216 09:10:44.055336 135566842806720 make_examples_core.py:348] Task 1/14: Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1771233044.055468     200 sam_reader.cc:1001] Setting HTS_OPT_BLOCK_SIZE to 134217728
I0216 09:10:44.084236 140419873022400 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:44.085796 140419873022400 make_examples_core.py:348] Task 5/14: Writing gvcf records to /tmp/examples.gvcf.tfrecord-00005-of-00014.gz
I0000 00:00:1771233044.086828     214 stream_examples.cc:80] Creating buffer_empty mutex
I0000 00:00:1771233044.086848     214 stream_examples.cc:84] Creating items_available mutex
I0216 09:10:44.098564 135566842806720 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:44.153213 126110666641856 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:44.154591 126110666641856 make_examples_core.py:348] Task 13/14: Writing gvcf records to /tmp/examples.gvcf.tfrecord-00013-of-00014.gz
I0000 00:00:1771233044.155490     199 stream_examples.cc:80] Creating buffer_empty mutex
I0000 00:00:1771233044.155507     199 stream_examples.cc:84] Creating items_available mutex
I0216 09:10:44.192765 133326641848768 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:44.194107 133326641848768 make_examples_core.py:348] Task 2/14: Writing gvcf records to /tmp/examples.gvcf.tfrecord-00002-of-00014.gz
I0000 00:00:1771233044.195112     217 stream_examples.cc:80] Creating buffer_empty mutex
I0000 00:00:1771233044.195130     217 stream_examples.cc:84] Creating items_available mutex
I0216 09:10:44.262294 135566842806720 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:44.263603 135566842806720 make_examples_core.py:348] Task 1/14: Writing gvcf records to /tmp/examples.gvcf.tfrecord-00001-of-00014.gz
I0000 00:00:1771233044.264606     200 stream_examples.cc:80] Creating buffer_empty mutex
I0000 00:00:1771233044.264624     200 stream_examples.cc:84] Creating items_available mutex
I0216 09:10:44.301330 127977190990272 make_examples_core.py:348] Task 6/14: Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1771233044.301474     198 sam_reader.cc:1001] Setting HTS_OPT_BLOCK_SIZE to 134217728
I0216 09:10:44.342330 127977190990272 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:44.425639 137567338668480 make_examples_core.py:348] Task 7/14: Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1771233044.425776     196 sam_reader.cc:1001] Setting HTS_OPT_BLOCK_SIZE to 134217728
I0216 09:10:44.466176 137567338668480 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:44.491388 127977190990272 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:44.492692 127977190990272 make_examples_core.py:348] Task 6/14: Writing gvcf records to /tmp/examples.gvcf.tfrecord-00006-of-00014.gz
I0000 00:00:1771233044.493576     198 stream_examples.cc:80] Creating buffer_empty mutex
I0000 00:00:1771233044.493592     198 stream_examples.cc:84] Creating items_available mutex
I0216 09:10:44.617978 137567338668480 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:44.619592 137567338668480 make_examples_core.py:348] Task 7/14: Writing gvcf records to /tmp/examples.gvcf.tfrecord-00007-of-00014.gz
I0000 00:00:1771233044.620602     196 stream_examples.cc:80] Creating buffer_empty mutex
I0000 00:00:1771233044.620623     196 stream_examples.cc:84] Creating items_available mutex
I0216 09:10:45.149549 127356011794880 make_examples_core.py:348] Task 3/14: Starting from v0.9.0, --use_ref_for_cram is default to true. If you are using CRAM input, note that we will decode CRAM using the reference you passed in with --ref
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1771233045.149689     211 sam_reader.cc:1001] Setting HTS_OPT_BLOCK_SIZE to 134217728
I0216 09:10:45.188975 127356011794880 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:45.333370 127356011794880 genomics_reader.py:223] Reading HG004.bam with NativeSamReader
I0216 09:10:45.334646 127356011794880 make_examples_core.py:348] Task 3/14: Writing gvcf records to /tmp/examples.gvcf.tfrecord-00003-of-00014.gz
I0000 00:00:1771233045.335521     211 stream_examples.cc:80] Creating buffer_empty mutex
I0000 00:00:1771233045.335536     211 stream_examples.cc:84] Creating items_available mutex