PT-2536 - Correct EMA calculation for WeightedAvgRate internal averages

[marcelohpf]

Average number of rows and average time taken to process rows are convergent to the average by adding the 1-weight to the calculation.

Previous calculation was divergent and a not correct implementation of ema (Exponential Moving Average);

The result of update() is not affected due to the proportional calculation of number-rows/time-taken

You can verify the behavior of current calculation with :

use WeightedAvgRate;

my $weight = 0.8;
my $w = WeightedAvgRate->new( target_t => 10, weight => $weight );
my @fake_n = ( 10, 5, 3, 1, 10, 3, 5, 10, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20 );

my ( $avg, $avg_ewma, $count, $sum ) = ( 0, 0, 0, 0 );
for my $n (@fake_n) {
    $sum   += $n;
    $count += 1;
    $avg = $sum / $count;
    $w->update( $n, 1 );
    $avg_ewma = $avg_ewma * $weight + $n * (1-$weight);
    printf( "%2.0f | %5.2f | %5.2f | %5.2f\n",
        $n, $avg, $w->{avg_n}, $avg, $avg_ewma );
}

1;

Root cause

The current calculation of average number of rows (avg_n) and average time (avg_t) seems to implement an smoothing algorithm to add some lag into the average convergence to protect estimation form sudden changes in a single datapoint, but also offer a faster convergence to that value on the long dataset (100+ data-points).

However, the current calculation is divergent serie because it does not apply a reduction factor to new data point every iteration.

As consequence, the avg_n and avg_t will increase forever and never converge to the "most common representative value";
Current formula is (avg * accumulative-factor) + new-value;
When it should be (avg * accumulative-factor) + (new-value * smooth-factor);

• The contributed code is licensed under GPL v2.0
• Contributor Licence Agreement (CLA) is signed
• util/update-modules has been ran
• Documentation updated
• Test suite update

[it-percona-cla]

All committers have signed the CLA.

[svetasmirnova]

Also tracked in Jira at https://perconadev.atlassian.net/browse/PT-2536