Skip to content

Commit d8c1ef4

Browse files
authored
Fix ADC parameter calculation overflow errors (#553)
As discussed in #515, an overflow can occur when calculating ADC parameters from limited-precision physical signal data. The root cause of this overflow is precision loss. When computing min/max values during parameter calculation, insufficient floating-point precision at boundaries (e.g. `int32` boundaries) leads to gain/baseline values that cause digital samples to overflow by exactly 1 value. The fix ensures `float64` precision is used throughout ADC parameter calculation and conversion: - In `calc_adc_params()`: Convert physical signals to `float64` before computing min/max - In `adc_inplace()`: Convert to `float64` before performing gain/baseline arithmetic Users can now safely provide lower-precision physical data (e.g., `float32`) without validation errors. This PR includes a test for the overflow which uses the data supplied by @Ivorforce in #515. The test checks that: - float32 input data can be written with `fmt=32` without overflow - digital values stay within the valid range - the physical signal round-trips accurately through write/read
2 parents 1310968 + d8073da commit d8c1ef4

4 files changed

Lines changed: 809 additions & 7 deletions

File tree

‎changelog.d/adc-float32-overflow.rst‎

Whitespace-only changes.

0 commit comments

Comments
 (0)