You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@yanj-github Your comment in the other issue mentioned "The test is on default timeout at 10 minutes it is fine for most of test but not long duration.". Does that mean you think this is just a timeout issue and not a bug in the test?
The test timeout after 100 seconds (sorry I measured time wrong not 10 mins) as it set in creating test.
From following screen shot test created with 100 s timeout.
I guess test creation need to allow more timeout for this specific test. @FritzHeiden ?
This is indeed a problem. For now I will solve this problem by disabling the timeout for long duration test all together. Do you have any other suggestions?
Thanks @FritzHeiden I am not sure what is the impact on having no timeout for a test? I guess user need to stop it manually if something is wrong? I don't think I can think of any alternative solutions. @jpiesing do you have any comments on this please?
Thanks @FritzHeiden I am not sure what is the impact on having no timeout for a test? I guess user need to stop it manually if something is wrong? I don't think I can think of any alternative solutions. @jpiesing do you have any comments on this please?
@yanj-github Removing the timeout would make this test semi-automatic. It runs fine when there is no issue, however, if something happens that prevents the video from playing all the way through you would have to manually resume the session. Alternatively we could adjust the timeout according to the test duration. If the video is 2.5h long we may have a timeout of 3h maybe. wdyt?
@FritzHeiden when the test is around 100s in time, the test status on result.html shows completed and API results reports timedout. Can it be improved please?
Thanks @yanj-github I can confirm this issue. To use a different timeout on the server the test would need to be tagged as manual test, so the manual timeout can be used. To tag the test as manual the file name needs to include manual, so this may require some changes in the OF aswell
Thanks @yanj-github I can confirm this issue. To use a different timeout on the server the test would need to be tagged as manual test, so the manual timeout can be used. To tag the test as manual the file name needs to include manual, so this may require some changes in the OF aswell
Thanks, OF gets file name from tests.json from test code to map to the file name so wont require the change.
Thanks @yanj-github I can confirm this issue. To use a different timeout on the server the test would need to be tagged as manual test, so the manual timeout can be used. To tag the test as manual the file name needs to include manual, so this may require some changes in the OF aswell
Thanks, OF gets file name from tests.json from test code to map to the file name so wont require the change.
Thanks @yanj-github we will then rename the long time playback test to include manual in the filename, and change the timeout of manual tests to a time above 2h (like 2h + 5min or so). wdyt?
@FritzHeiden and @louaybassbouss the audio content is missing for long duration test, OF terminating because required audio configuration is missing. I don't think we have right audio stream yet. If it is an easy update can we have a placeholder short audio until right audio stream is available please? Maybe we can just use same audio as 9.2 test.
@FritzHeiden and @louaybassbouss the audio content is missing for long duration test, OF terminating because required audio configuration is missing. I don't think we have right audio stream yet. If it is an easy update can we have a placeholder short audio until right audio stream is available please? Maybe we can just use same audio as 9.2 test.
What do you mean by short audio. The duration of the audio should fit with the duration of the video. In other words, we need long duration audio as well.
@FritzHeiden and @louaybassbouss the audio content is missing for long duration test, OF terminating because required audio configuration is missing. I don't think we have right audio stream yet. If it is an easy update can we have a placeholder short audio until right audio stream is available please? Maybe we can just use same audio as 9.2 test.
What do you mean by short audio. The duration of the audio should fit with the duration of the video. In other words, we need long duration audio as well.
Hi @louaybassbouss I meant to have a temporally place holder audio until we have right audio stream available. So that OF can do the video testing. At the moment missing audio OF will terminating with error. If it is not an easy change to TR, or cause confusion, no worries I will comment out audio testing for long duration check so it can do video testing only. Thanks.
Pulling the dpctf-deploy repository before rebuilding is required for this to work
If it is not an easy change to TR, or cause confusion, no worries I will comment out audio testing for long duration check so it can do video testing only. Thanks.
I'd rather have this temporary fix in the OF as I don't know if having a shorter audio track in the player may cause unexpected behavior
Pulling the dpctf-deploy repository before rebuilding is required for this to work
If it is not an easy change to TR, or cause confusion, no worries I will comment out audio testing for long duration check so it can do video testing only. Thanks.
I'd rather have this temporary fix in the OF as I don't know if having a shorter audio track in the player may cause unexpected behavior
Thanks @FritzHeiden understand now. In this case it would be easier to turn off the audio observation for this test on OF end. I will make changes to OF temporally comment it out. So video only observation can be run.
I'd like to share video observation result here. The playback pauses a lot and it shows waiting when it does. This makes duration fail and current time match. This maybe a device specific issue waiting to get more test result on different devices.
[OF] Every video frame S[k,s] shall be rendered and the video frames shall be rendered in increasing presentation time order.
PASS: Total of missing frame count is 0.
[OF] Video start-up delay: The start-up delay should be sufficiently low.
PASS: Maximum permitted startup delay is 1000ms. The presentation start up delay is 2.5833ms.
[OF] Video: The playback duration shall match the duration of the CMAF Track
FAIL: Playback duration 15907504.87ms does not match expected duration 7200000.0ms +/- tolerance of 50ms. Detected duration is different by 8707504.87ms.
[OF] Video: The presented sample shall match the one reported by the currentTime value within the tolerance.
FAIL: Total failure count is 103.
Has anyone else tried this and experienced the same issue?
I have also raise issue here: cta-wave/device-playback-task-force#133 Do we really need to test each individual frame or pick small samples would be good enough?
@yanj-github can you check your local /dpctf-deploy/config.json what you have for manual timeout? Long duration tests are considered as manual tests where we apply a large timeout (large enough to play the tests from the start to the end). Can you send more details or screenshot of the last video frame you see? Did you tested in Desktop Browser or on TV?
Activity
jpiesing commentedon Feb 13, 2024
@yanj-github Your comment in the other issue mentioned "The test is on default timeout at 10 minutes it is fine for most of test but not long duration.". Does that mean you think this is just a timeout issue and not a bug in the test?
yanj-github commentedon Feb 13, 2024
The test timeout after 100 seconds (sorry I measured time wrong not 10 mins) as it set in creating test.
From following screen shot test created with 100 s timeout.
I guess test creation need to allow more timeout for this specific test. @FritzHeiden ?
FritzHeiden commentedon Feb 14, 2024
This is indeed a problem. For now I will solve this problem by disabling the timeout for long duration test all together. Do you have any other suggestions?
yanj-github commentedon Feb 16, 2024
Thanks @FritzHeiden I am not sure what is the impact on having no timeout for a test? I guess user need to stop it manually if something is wrong? I don't think I can think of any alternative solutions.
@jpiesing do you have any comments on this please?
FritzHeiden commentedon Mar 6, 2024
@yanj-github Removing the timeout would make this test semi-automatic. It runs fine when there is no issue, however, if something happens that prevents the video from playing all the way through you would have to manually resume the session. Alternatively we could adjust the timeout according to the test duration. If the video is 2.5h long we may have a timeout of 3h maybe. wdyt?
yanj-github commentedon Mar 13, 2024
I think having timeout is better.
FritzHeiden commentedon Apr 9, 2024
I merged some PRs to set the timeout for long duration tests to 3 hours using the test-config.json: #165 #166 cta-wave/dpctf-test-runner#50
This requires a rebuild of the image with
--reload-testsand--reload-runnerflags.yanj-github commentedon May 13, 2024
@FritzHeiden when the test is around 100s in time, the test status on result.html shows completed and API results reports timedout. Can it be improved please?
FritzHeiden commentedon May 13, 2024
Thanks @yanj-github I can confirm this issue. To use a different timeout on the server the test would need to be tagged as manual test, so the manual timeout can be used. To tag the test as manual the file name needs to include
manual, so this may require some changes in the OF aswellyanj-github commentedon May 13, 2024
Thanks, OF gets file name from tests.json from test code to map to the file name so wont require the change.
louaybassbouss commentedon May 14, 2024
Thanks @yanj-github we will then rename the long time playback test to include manual in the filename, and change the timeout of manual tests to a time above 2h (like 2h + 5min or so). wdyt?
jpiesing commentedon May 14, 2024
2024:05-14: Issue can be closed when the test is renamed.
yanj-github commentedon May 14, 2024
Thanks Jon, can it stay open after the fix on TR until I run it with OF please?
yanj-github commentedon May 15, 2024
@FritzHeiden and @louaybassbouss the audio content is missing for long duration test, OF terminating because required audio configuration is missing. I don't think we have right audio stream yet. If it is an easy update can we have a placeholder short audio until right audio stream is available please? Maybe we can just use same audio as 9.2 test.
louaybassbouss commentedon May 15, 2024
What do you mean by short audio. The duration of the audio should fit with the duration of the video. In other words, we need long duration audio as well.
jpiesing commentedon May 16, 2024
The 'tLD' content is at https://dash.akamaized.net/WAVE/vectors/cfhd_sets/12.5_25_50/tLD/2023-09-01/stream.mpd. I have downloaded the MPD and confirm it only contains video and no audio.
I think the relevant issue in the test content repo is cta-wave/Test-Content#39 (which was closed) which in turn refers to cta-wave/mezzanine#46.
That issue is closed with a comment from Nicholas Frame that the LD mezzanine has been updated to include the looped PN audio.
I think therefore that tLD needs re-encoding to have the audio & I've re-opened the issue.
yanj-github commentedon May 16, 2024
Hi @louaybassbouss I meant to have a temporally place holder audio until we have right audio stream available. So that OF can do the video testing. At the moment missing audio OF will terminating with error. If it is not an easy change to TR, or cause confusion, no worries I will comment out audio testing for long duration check so it can do video testing only. Thanks.
FritzHeiden commentedon May 16, 2024
Renamed the test #181 and updated internal manual timeout to a high value cta-wave/dpctf-deploy#92
Pulling the dpctf-deploy repository before rebuilding is required for this to work
I'd rather have this temporary fix in the OF as I don't know if having a shorter audio track in the player may cause unexpected behavior
yanj-github commentedon May 16, 2024
Thanks @FritzHeiden understand now. In this case it would be easier to turn off the audio observation for this test on OF end. I will make changes to OF temporally comment it out. So video only observation can be run.
yanj-github commentedon May 23, 2024
I'd like to share video observation result here. The playback pauses a lot and it shows waiting when it does. This makes duration fail and current time match. This maybe a device specific issue waiting to get more test result on different devices.
PASS: Total of missing frame count is 0.
PASS: Maximum permitted startup delay is 1000ms. The presentation start up delay is 2.5833ms.
FAIL: Playback duration 15907504.87ms does not match expected duration 7200000.0ms +/- tolerance of 50ms. Detected duration is different by 8707504.87ms.
FAIL: Total failure count is 103.
yanj-github commentedon Jan 29, 2025
Has anyone else tried this and experienced the same issue?
I have also raise issue here: cta-wave/device-playback-task-force#133 Do we really need to test each individual frame or pick small samples would be good enough?
louaybassbouss commentedon Jan 29, 2025
@yanj-github can you check your local /dpctf-deploy/config.json what you have for manual timeout? Long duration tests are considered as manual tests where we apply a large timeout (large enough to play the tests from the start to the end). Can you send more details or screenshot of the last video frame you see? Did you tested in Desktop Browser or on TV?
FritzHeiden commentedon Jul 21, 2026
Long duration tests are working fine now.
@yanj-github do you still encounter your issue?
yanj-github commentedon Jul 21, 2026
The LD test now working fine. Closing issue.