Archived issue #0032095
Tests - drop comparison in unstable perf/ncollection group
Description
Test case perf/ncollection/A1 regularly fails due to instabilities in test system.
By it's nature test case was first created to compare performance of similar STL and OCCT collections in common operations and then collected numbers have been put into test in attempt to detect possible performance degradation in OCCT collections.
Obviously, as there are many different STL implementations (at least one per compiler), these implementation evolve and debug/release/optimization flags dramatically affect performance, it is practically impossible to embed meaningful numbers into test case itself. Several attempts have been already tried to adjust numbers and extend comparison tolerance, but test grid remains unstable on Jenkins.
Therefore, it is proposed to not report test case failures and just report deviations as warnings.
By it's nature test case was first created to compare performance of similar STL and OCCT collections in common operations and then collected numbers have been put into test in attempt to detect possible performance degradation in OCCT collections.
Obviously, as there are many different STL implementations (at least one per compiler), these implementation evolve and debug/release/optimization flags dramatically affect performance, it is practically impossible to embed meaningful numbers into test case itself. Several attempts have been already tried to adjust numbers and extend comparison tolerance, but test grid remains unstable on Jenkins.
Therefore, it is proposed to not report test case failures and just report deviations as warnings.
Steps to reproduce
perf/ncollection/A1
Public activity
7 archived notes
Participants are labeled by their role within this record.
Branch [archived branch] has been created by Author.
[revision removed]
Detailed log of new commits:
Author: Author
Date: Fri Jan 29 10:24:36 2021 +0300
0032095: Tests - drop comparison in unstable perf/ncollection group
Mark performance deviations compared to STL collections as warnings instead of errors.
[revision removed]
Detailed log of new commits:
Author: Author
Date: Fri Jan 29 10:24:36 2021 +0300
0032095: Tests - drop comparison in unstable perf/ncollection group
Mark performance deviations compared to STL collections as warnings instead of errors.
Branch [archived branch] has been updated forcibly by Author.
[revision removed]
[revision removed]
Branch [archived branch] has been updated forcibly by Author.
[revision removed]
[revision removed]
Proposed patch is ready for review (putting these failures to IGNORE rule)
- OCCT branch: [archived branch]
- OCCT branch: [archived branch]
No remarks, please integrate
Combination -
OCCT branch : [archived branch]
master SHA - [revision removed]
[revision removed]
Products branch : [archived branch] SHA - [revision removed]
was compiled on Linux, MacOS and Windows platforms and tested in optimize mode.
Number of compiler warnings:
No new/fixed warnings
Regressions/Differences/Improvements:
No regressions/differences
CPU differences:
Debian80-64:
OCCT
Total CPU difference: 17805.430000000237 / 17802.420000000013 [+0.02%]
Products
Total CPU difference: 11513.250000000124 / 11523.070000000105 [-0.09%]
Windows-64-VC14:
OCCT
Total CPU difference: 19375.296875 / 19372.265625 [+0.02%]
Products
Total CPU difference: 12857.265625 / 12882.921875 [-0.20%]
Image differences :
No differences that require special attention
Memory differences :
No differences that require special attention
OCCT branch : [archived branch]
master SHA - [revision removed]
[revision removed]
Products branch : [archived branch] SHA - [revision removed]
was compiled on Linux, MacOS and Windows platforms and tested in optimize mode.
Number of compiler warnings:
No new/fixed warnings
Regressions/Differences/Improvements:
No regressions/differences
CPU differences:
Debian80-64:
OCCT
Total CPU difference: 17805.430000000237 / 17802.420000000013 [+0.02%]
Products
Total CPU difference: 11513.250000000124 / 11523.070000000105 [-0.09%]
Windows-64-VC14:
OCCT
Total CPU difference: 19375.296875 / 19372.265625 [+0.02%]
Products
Total CPU difference: 12857.265625 / 12882.921875 [-0.20%]
Image differences :
No differences that require special attention
Memory differences :
No differences that require special attention
Branch [archived branch] has been deleted by Participant.
[revision removed]
[revision removed]
Related records