Archived issue #0026515
Exponential memory usage problems with BOPDS_IndexedMapOfPaveBlock and NCollection_IncAllocator
Description
In BopAlgo_PaveFiller, MakeBlocks(), NCollection_IncAllocator is used to allocate memory for the collections. When largish sets of tools (>150 SHAPES) are used in a boolean operation the memory consumption of this class rises exponentially, over 2GB of memory is consumed when the collections have the following numbers
aLSE=3607
aLBV=0
aMVOnIn=12
aMF=3739
aMVStick=0
aMVEF=0
aMI=18
aMPBOnIn=15
aMPBAdd=9799
aLPB=1
aMSCPB=2974
aMVI=4832
aDMExEdges=222
aMVTol=0
aDMI=222
aDMBV=0
These are not really large numbers and definitely should not consume 2GB of RAM
Removing the use of the allocator from the collection constructors completely resolves the problem, in fact there is no significant memory use at all in Task manager or processes viewer. Indicating the issue is in the allocation class.
I have tried to find the cause, it appears to be BOPDS_IndexedMapOfPaveBlock and BOPCol_MapOfInteger that causes excessive memory allocation.
Strangely, removing the use of the allocator not only improves memory performance but also speed in my environment (.NET, mix mode)
aLSE=3607
aLBV=0
aMVOnIn=12
aMF=3739
aMVStick=0
aMVEF=0
aMI=18
aMPBOnIn=15
aMPBAdd=9799
aLPB=1
aMSCPB=2974
aMVI=4832
aDMExEdges=222
aMVTol=0
aDMI=222
aDMBV=0
These are not really large numbers and definitely should not consume 2GB of RAM
Removing the use of the allocator from the collection constructors completely resolves the problem, in fact there is no significant memory use at all in Task manager or processes viewer. Indicating the issue is in the allocation class.
I have tried to find the cause, it appears to be BOPDS_IndexedMapOfPaveBlock and BOPCol_MapOfInteger that causes excessive memory allocation.
Strangely, removing the use of the allocator not only improves memory performance but also speed in my environment (.NET, mix mode)
Steps to reproduce
Cut more than 150 shapes from any shape. In fact the main class that clauses the problem BOPDS_IndexedMapOfPaveBlock only has 15 members in my example so the problem hopefully is simple, but I am afraid i couldn't grasp it.
Public activity
21 archived notes
Participants are labeled by their role within this record.
The aim of using IncAllocator is to reduce the number of times of calling malloc/free. It never returns memory back to system till its destruction or till someone calls its Reset method. Therefore it should be used with care. If a container is constructed with IncAllocator then it is supposed one of two conditions to be true:
1. Life time of this container is small.
2. If life time is large then the container will accommodate new objects without removing existing objects, because allocator's Free method does not return memory to system.
Probably, for container-fields of BOP classes that live long it is needed to refuse passing IncAllocator to constructor.
1. Life time of this container is small.
2. If life time is large then the container will accommodate new objects without removing existing objects, because allocator's Free method does not return memory to system.
Probably, for container-fields of BOP classes that live long it is needed to refuse passing IncAllocator to constructor.
Dear Author,
Could you, please attach input data and provide the draw script in "steps to reproduce" field? We need it to reproduce your problem and insert it in non-regression test base.
Thanks in advance,
Mikhail
Could you, please attach input data and provide the draw script in "steps to reproduce" field? We need it to reproduce your problem and insert it in non-regression test base.
Thanks in advance,
Mikhail
I will try it was a complex operation that didn't really set well in a brep file.
I will see if I can convert it to a draw script but it may take me a while.
PS I am pretty sure the function is only caused by the two functions I pinpointed BOPDS_IndexedMapOfPaveBlock and BOPCol_MapOfInteger is there something different about these from other maps?
I will see if I can convert it to a draw script but it may take me a while.
PS I am pretty sure the function is only caused by the two functions I pinpointed BOPDS_IndexedMapOfPaveBlock and BOPCol_MapOfInteger is there something different about these from other maps?
I did not understand the question, but the problem is here not with maps, but with overall algorithm behavior and mutual scope of containers and IncAllocator used for them. We have already another bug, which fix should help in your case. However, it would be fine to add to non-regression test base your case too, if possible.
Dear Ilya, please take the patch from the branch to related issue #26567, and merge it in master.
Branch [archived branch] has been created by Participant.
[revision removed]
Detailed log of new commits:
Author: pkv
Date: Tue Aug 18 10:06:11 2015 +0300
0026567: Exception in Boolean intersection command
Correcting the usage of NCollection_IncAllocator
Conflicts:
src/BOPAlgo/BOPAlgo_BOP.cxx
src/BOPAlgo/BOPAlgo_Builder.cxx
src/BOPAlgo/BOPAlgo_BuilderSolid.cxx
src/BOPAlgo/BOPAlgo_Builder_2.cxx
src/BOPAlgo/BOPAlgo_MakerVolume.cxx
src/BOPAlgo/BOPAlgo_PaveFiller.cdl
src/BOPAlgo/BOPAlgo_PaveFiller_1.cxx
src/BOPAlgo/BOPAlgo_PaveFiller_10.cxx
src/BOPAlgo/BOPAlgo_PaveFiller_3.cxx
src/BOPAlgo/BOPAlgo_PaveFiller_4.cxx
src/BOPAlgo/BOPAlgo_PaveFiller_5.cxx
src/BOPAlgo/BOPAlgo_PaveFiller_6.cxx
src/BOPAlgo/BOPAlgo_PaveFiller_7.cxx
src/BOPAlgo/BOPAlgo_PaveFiller_8.cxx
src/BOPAlgo/BOPAlgo_Section.cxx
src/BOPDS/BOPDS_DS.cxx
src/BOPDS/BOPDS_Iterator.cxx
src/BOPDS/BOPDS_IteratorSI.cxx
src/BOPDS/BOPDS_SubIterator.cxx
src/BOPTest/BOPTest_BOPCommands.cxx
src/BOPTest/BOPTest_PartitionCommands.cxx
src/IntTools/IntTools_Context.cxx
[revision removed]
Detailed log of new commits:
Author: pkv
Date: Tue Aug 18 10:06:11 2015 +0300
0026567: Exception in Boolean intersection command
Correcting the usage of NCollection_IncAllocator
Conflicts:
src/BOPAlgo/BOPAlgo_BOP.cxx
src/BOPAlgo/BOPAlgo_Builder.cxx
src/BOPAlgo/BOPAlgo_BuilderSolid.cxx
src/BOPAlgo/BOPAlgo_Builder_2.cxx
src/BOPAlgo/BOPAlgo_MakerVolume.cxx
src/BOPAlgo/BOPAlgo_PaveFiller.cdl
src/BOPAlgo/BOPAlgo_PaveFiller_1.cxx
src/BOPAlgo/BOPAlgo_PaveFiller_10.cxx
src/BOPAlgo/BOPAlgo_PaveFiller_3.cxx
src/BOPAlgo/BOPAlgo_PaveFiller_4.cxx
src/BOPAlgo/BOPAlgo_PaveFiller_5.cxx
src/BOPAlgo/BOPAlgo_PaveFiller_6.cxx
src/BOPAlgo/BOPAlgo_PaveFiller_7.cxx
src/BOPAlgo/BOPAlgo_PaveFiller_8.cxx
src/BOPAlgo/BOPAlgo_Section.cxx
src/BOPDS/BOPDS_DS.cxx
src/BOPDS/BOPDS_Iterator.cxx
src/BOPDS/BOPDS_IteratorSI.cxx
src/BOPDS/BOPDS_SubIterator.cxx
src/BOPTest/BOPTest_BOPCommands.cxx
src/BOPTest/BOPTest_PartitionCommands.cxx
src/IntTools/IntTools_Context.cxx
Branch from issue ##26567 have been merged with the current master
Some changes were not transfered. Please check all changes manually.
Branch [archived branch] has been updated forcibly by Participant.
[revision removed]
[revision removed]
The changes in the files
src\BOPAlgo\BOPAlgo_PaveFiller_7.cxx
src\BOPAlgo\BOPAlgo_Section.cxx
src\BOPTest\BOPTest_PartitionCommands.cxx
were not transfered. Please make the changes in them also.
src\BOPAlgo\BOPAlgo_PaveFiller_7.cxx
src\BOPAlgo\BOPAlgo_Section.cxx
src\BOPTest\BOPTest_PartitionCommands.cxx
were not transfered. Please make the changes in them also.
Branch [archived branch] has been updated by Participant.
[revision removed]
Detailed log of new commits:
Author: isn
Date: Fri Sep 11 11:29:57 2015 +0300
additional corrections
[revision removed]
Detailed log of new commits:
Author: isn
Date: Fri Sep 11 11:29:57 2015 +0300
additional corrections
Reviewed.
Branch [archived branch] has been updated forcibly by Participant.
[revision removed]
[revision removed]
Branch [archived branch] has been rebased on the current master
Dear Commenter 1,
Branch CR26515 from occt git-repository (and master from products git-repository) was compiled on Linux and Windows platforms and tested.
[revision removed]
Number of compiler warnings:
occt component:
Linux: 15 (15 on master)
Windows: 0 (0 on master)
products component:
Linux: 39 (39 on master)
Windows: 0 (0 on master)
Regressions/Differences:
http://occt-tests/CR26515-master-occt-64/Debian70-64/summary.html
http://occt-tests/CR26515-master-occt-64/Windows-64-VC10/summary.html
bugs fclasses bug7287_1
Testing on Linux:
Total MEMORY difference: 90349201 / 92721808 [-2.56%]
Total CPU difference: 17980.68999999927 / 17686.84999999906 [+1.66%]
Testing on Windows:
Total MEMORY difference: 56152860 / 57753560 [-2.77%]
Total CPU difference: 16694.18181329908 / 16847.78039789915 [-0.91%]
Branch CR26515 from occt git-repository (and master from products git-repository) was compiled on Linux and Windows platforms and tested.
[revision removed]
Number of compiler warnings:
occt component:
Linux: 15 (15 on master)
Windows: 0 (0 on master)
products component:
Linux: 39 (39 on master)
Windows: 0 (0 on master)
Regressions/Differences:
http://occt-tests/CR26515-master-occt-64/Debian70-64/summary.html
http://occt-tests/CR26515-master-occt-64/Windows-64-VC10/summary.html
bugs fclasses bug7287_1
Testing on Linux:
Total MEMORY difference: 90349201 / 92721808 [-2.56%]
Total CPU difference: 17980.68999999927 / 17686.84999999906 [+1.66%]
Testing on Windows:
Total MEMORY difference: 56152860 / 57753560 [-2.77%]
Total CPU difference: 16694.18181329908 / 16847.78039789915 [-0.91%]
Dear Alexey, please remove TODO of the improved test case bugs fclasses bug7287_1.
Branch [archived branch] has been updated by Participant.
[revision removed]
Detailed log of new commits:
Author: abv
Date: Tue Sep 15 13:54:13 2015 +0300
Adaptation of test case bugs fclasses bug7287_1 (limit for detection of memory leak reduced due to leak reduction)
[revision removed]
Detailed log of new commits:
Author: abv
Date: Tue Sep 15 13:54:13 2015 +0300
Adaptation of test case bugs fclasses bug7287_1 (limit for detection of memory leak reduced due to leak reduction)
I have pushed correction of test case bugs fclasses bug7287_1 to branch CR26515, please check it (only this test!).
Besides, while integrating this fix, note that title of its commit messages is that of #26567; this is correct.
Besides, while integrating this fix, note that title of its commit messages is that of #26567; this is correct.
Branch [archived branch] has been updated by Participant.
[revision removed]
Detailed log of new commits:
Author: apv
Date: Tue Sep 15 17:42:07 2015 +0300
Adjusting test-case bugs fclasses bug7287_1
[revision removed]
Detailed log of new commits:
Author: apv
Date: Tue Sep 15 17:42:07 2015 +0300
Adjusting test-case bugs fclasses bug7287_1
Result of testing case bugs fclasses bug7287_1:
http://occt-tests/CR26515-master-occt-64/Debian70-64/bugs/fclasses/bug7287_1.html
http://occt-tests/CR26515-master-occt-64/Windows-64-VC10/bugs/fclasses/bug7287_1.html
http://occt-tests/CR26515-master-occt-64/Debian70-64/bugs/fclasses/bug7287_1.html
http://occt-tests/CR26515-master-occt-64/Windows-64-VC10/bugs/fclasses/bug7287_1.html
Branch [archived branch] has been deleted by Participant.
[revision removed]
[revision removed]