Archived issue #0032870
Tests - bugs/caf/bug31918_1 is unstable
Description
Test case bugs/caf/bug31918_1 is unstable and fails from time to time with error like this:
Test case has two problems:
- Compared values look too small (a couple of milliseconds) to be measured reliably (2ms vs. 4ms).
- `time` Tcl command is used for measurements instead of `dchrono`, which means that test compares **elapsed time** - an unreliable measure within multiple concurrent tests execution.
http://occt-tests/IR-2022-03-11-OCCT/Windows-64-VC14/bugs/caf/bug31918_1.html
Quater of document open time 2738.95 mcs Error : loading of quater of the document content too slow relatively to the whole document load ... Four quaters of document open time 4780.25 mcs Quater of document without arrays open time 2521 mcs Half of document arrays open time 2352 mcs
Test case has two problems:
set quater_time [lindex [time {
Open ${docname} D2 -read0:2
Close D2
} 20] 0]
puts "Quater of document open time $quater_time mcs"
# Check that open of quater of the document is at least twice faster than open of whole.
if { [expr $quater_time * 2] > $whole_time } {
puts "Error : loading of quater of the document content too slow relatively to the whole document load"
}
- Compared values look too small (a couple of milliseconds) to be measured reliably (2ms vs. 4ms).
- `time` Tcl command is used for measurements instead of `dchrono`, which means that test compares **elapsed time** - an unreliable measure within multiple concurrent tests execution.
http://occt-tests/IR-2022-03-11-OCCT/Windows-64-VC14/bugs/caf/bug31918_1.html
Steps to reproduce
bugs caf bug31918_1
Public activity
11 archived notes
Participants are labeled by their role within this record.
Dear KGV,
As you can see, the second argument of calling "time" is 20, it takes average of time of opening documents among 20. So, technically it is not "2ms vs. 4ms" but "40ms vs. 80ms".
What is your proposal how to improve this test? What do you think, is it worth to increase the test execution time, how much?
I have an idea to make measurement of each value several times and take the minimal one (the one, less affected by parallel threads). From your experience of writing tests, would it be helpful in this case?
As you can see, the second argument of calling "time" is 20, it takes average of time of opening documents among 20. So, technically it is not "2ms vs. 4ms" but "40ms vs. 80ms".
What is your proposal how to improve this test? What do you think, is it worth to increase the test execution time, how much?
I have an idea to make measurement of each value several times and take the minimal one (the one, less affected by parallel threads). From your experience of writing tests, would it be helpful in this case?
First of all what bothers me is that `time` measures elapsed time, not CPU time, which makes it very dependent on execution environment.
Using CPU time from `dchrono` should make test more reliable.
As for duration - 2ms (and even 20ms) is a pretty small number. Increasing a duration in 10x times might make sense.
Branch [archived branch] has been created by Participant.
[revision removed]
Detailed log of new commits:
Author: mpv
Date: Mon Mar 14 18:03:59 2022 +0300
0032870: Tests - bugs/caf/bug31918_1 is unstable
Improved the test stability: created 10 iterations of measurements and take the minimum to
minimize the affect of other processes and threads; increased the compare range of
the quarter and the whole of the document load.
[revision removed]
Detailed log of new commits:
Author: mpv
Date: Mon Mar 14 18:03:59 2022 +0300
0032870: Tests - bugs/caf/bug31918_1 is unstable
Improved the test stability: created 10 iterations of measurements and take the minimum to
minimize the affect of other processes and threads; increased the compare range of
the quarter and the whole of the document load.
It seems the test case is improved and become work more stable.
Combination -
OCCT branch : [archived branch]
master SHA - [revision removed]
[revision removed]
Products branch : [archived branch] SHA - [revision removed]
was compiled on Linux, MacOS and Windows platforms and tested in optimize mode.
Number of compiler warnings:
No new/fixed warnings
Regressions/Differences/Improvements:
No regressions/differences
CPU differences:
Debian80-64:
OCCT
Total CPU difference: 18487.460000000385 / 18395.730000000258 [+0.50%]
Products
Total CPU difference: 11709.300000000132 / 11708.150000000112 [+0.01%]
Windows-64-VC14:
OCCT
Total CPU difference: 20412.546875 / 20350.609375 [+0.30%]
Products
Total CPU difference: 13181.25 / 13154.5625 [+0.20%]
Image differences :
No differences that require special attention
Memory differences :
No differences that require special attention
OCCT branch : [archived branch]
master SHA - [revision removed]
[revision removed]
Products branch : [archived branch] SHA - [revision removed]
was compiled on Linux, MacOS and Windows platforms and tested in optimize mode.
Number of compiler warnings:
No new/fixed warnings
Regressions/Differences/Improvements:
No regressions/differences
CPU differences:
Debian80-64:
OCCT
Total CPU difference: 18487.460000000385 / 18395.730000000258 [+0.50%]
Products
Total CPU difference: 11709.300000000132 / 11708.150000000112 [+0.01%]
Windows-64-VC14:
OCCT
Total CPU difference: 20412.546875 / 20350.609375 [+0.30%]
Products
Total CPU difference: 13181.25 / 13154.5625 [+0.20%]
Image differences :
No differences that require special attention
Memory differences :
No differences that require special attention
Branch [archived branch] has been deleted by Author.
[revision removed]
[revision removed]
Test case is still unstable on Jenkins.
Branch [archived branch] has been created by Participant.
[revision removed]
Detailed log of new commits:
Author: mpv
Date: Wed Apr 6 11:08:20 2022 +0300
0032870: Tests - bugs/caf/bug31918_1 is unstable
# Make the compared values more stable.
[revision removed]
Detailed log of new commits:
Author: mpv
Date: Wed Apr 6 11:08:20 2022 +0300
0032870: Tests - bugs/caf/bug31918_1 is unstable
# Make the compared values more stable.
Additionally fixed in CR32870_1. Updated comparison coeffs to make test one more stable.
http://jenkins-test-occt.nnov.opencascade.com/job/CR32870_1-master-MPV-OCCT-Debian80-64-opt-test-compare/2/
http://jenkins-test-occt.nnov.opencascade.com//view/CR32870_1-master-MPV/view/COMPARE/job/CR32870_1-master-MPV-OCCT-Windows-64-VC14-opt-test-compare/
http://jenkins-test-occt.nnov.opencascade.com/job/CR32870_1-master-MPV-Products-Windows-64-VC14-opt-test-compare/2/
http://jenkins-test-occt.nnov.opencascade.com/job/CR32870_1-master-MPV-Products-Debian80-64-opt-test-compare/1/
http://jenkins-test-occt.nnov.opencascade.com/job/CR32870_1-master-MPV-OCCT-Debian80-64-opt-test-compare/2/
http://jenkins-test-occt.nnov.opencascade.com//view/CR32870_1-master-MPV/view/COMPARE/job/CR32870_1-master-MPV-OCCT-Windows-64-VC14-opt-test-compare/
http://jenkins-test-occt.nnov.opencascade.com/job/CR32870_1-master-MPV-Products-Windows-64-VC14-opt-test-compare/2/
http://jenkins-test-occt.nnov.opencascade.com/job/CR32870_1-master-MPV-Products-Debian80-64-opt-test-compare/1/
Combination -
OCCT branch : [archived branch]
master SHA - [revision removed]
[revision removed]
Products branch : [archived branch] SHA - [revision removed]
was compiled on Linux, MacOS and Windows platforms and tested in optimize mode.
Number of compiler warnings:
No new/fixed warnings
Regressions/Differences/Improvements:
No regressions/differences
CPU differences:
Debian80-64:
OCCT
Total CPU difference: 18452.87000000039 / 18490.94000000032 [-0.21%]
Products
Total CPU difference: 11761.310000000152 / 11719.920000000124 [+0.35%]
Windows-64-VC14:
OCCT
Total CPU difference: 20539.828125 / 20552.71875 [-0.06%]
Products
Total CPU difference: 13248.21875 / 13242.609375 [+0.04%]
Image differences :
No differences that require special attention
Memory differences :
No differences that require special attention
OCCT branch : [archived branch]
master SHA - [revision removed]
[revision removed]
Products branch : [archived branch] SHA - [revision removed]
was compiled on Linux, MacOS and Windows platforms and tested in optimize mode.
Number of compiler warnings:
No new/fixed warnings
Regressions/Differences/Improvements:
No regressions/differences
CPU differences:
Debian80-64:
OCCT
Total CPU difference: 18452.87000000039 / 18490.94000000032 [-0.21%]
Products
Total CPU difference: 11761.310000000152 / 11719.920000000124 [+0.35%]
Windows-64-VC14:
OCCT
Total CPU difference: 20539.828125 / 20552.71875 [-0.06%]
Products
Total CPU difference: 13248.21875 / 13242.609375 [+0.04%]
Image differences :
No differences that require special attention
Memory differences :
No differences that require special attention
Branch [archived branch] has been deleted by Participant.
[revision removed]
[revision removed]
Related records