difference in number of validated tests by frame rate family

[jpiesing]

Looking in dpctf-tests / generated, there's a big difference in the number of validated tests for the different frame rate families.

114 "Validated" variants​

• 60 for audio​
• 28 for 25/50 Hz video​
• 14 for fractional frame rates​
• 12 for 30/60 Hz video​

For example, we have the following for 30/60 Hz with local hosting.

    "/cfhd_15_30_60-local/mse-appendwindow__t1.html",
    "/cfhd_15_30_60-local/out-of-order-loading__t1.html",
    "/cfhd_15_30_60-local/playback-of-encrypted-content-https__t1-cenc.html",
    "/cfhd_15_30_60-local/random-access-to-fragment__t1.html",
    "/cfhd_15_30_60-local/sequential-track-playback__t1.html",
    "/cfhd_15_30_60-local/sequential-track-playback__t3.html",

But for 25/50 Hz with local hosting we have these.

    "/cfhd_12.5_25_50-local/buffer-underrun-and-recovery__t2.html",
    "/cfhd_12.5_25_50-local/low-latency-initialization__t2.html",
    "/cfhd_12.5_25_50-local/low-latency-short-buffer-playback__t2.html",
    "/cfhd_12.5_25_50-local/mse-appendwindow__t1.html",
    "/cfhd_12.5_25_50-local/out-of-order-loading__t1.html",
    "/cfhd_12.5_25_50-local/playback-of-encrypted-content-https__t1-cenc.html",
    "/cfhd_12.5_25_50-local/random-access-to-fragment__t1.html",
    "/cfhd_12.5_25_50-local/regular-playback-of-chunked-content-non-aligned-append__chunked.html",
    "/cfhd_12.5_25_50-local/regular-playback-of-chunked-content__chunked.html",
    "/cfhd_12.5_25_50-local/restricted-splicing-of-encrypted-content-https__splice_main-cenc_splice_ad-cenc.html",
    "/cfhd_12.5_25_50-local/sequential-playback-of-encrypted-and-non-encrypted-baseline-content-https__splice_main-cenc_splice_ad.html",
    "/cfhd_12.5_25_50-local/playback-over-wave-baseline-splice-constraints__splice_main_splice_ad.html",
    "/cfhd_12.5_25_50-local/sequential-track-playback__t1.html",
    "/cfhd_12.5_25_50-local/sequential-track-playback__t3.html",

We need to discuss this, at least the following;

• Is the goal parity in numbers of tests between frame rate families (or at least very close to it)?
• Is parity between 30/60 and fractional frame rates needed or can we can we prioritize one & have reduced number of tests for the other?
• How much we want to infer about tests in one frame rate family from results for a different frame rate family?
• We are primarily running the 25/50 Hz tests, how much running of the 30/60 Hz tests do we require for a release?

The number of tests that are generated is much closer between the frame rates. There are 825 .html files under /generated/ as follows.
263 for 25/50 Hz video,
240 for 30/60Hz video,
238 for fractional frame rates and
84 for audio