Archived issue #0026338
STL export (especially binary) needs a lot of time if selected export path is not local
Description
If the output path of an STL export refers to a network location instead of a file on a local disk, the export can be quite slow.
The reason is that in RWStl::WriteAscii the file is accessed once for every triangle. In RWStl::WriteBinary it is even worse. The file is accessed 13 times for every triangle. If a model contains many triangles this can be quite noticeable and block the export for a while. In a test, for 6000 triangles the binary export needed 25 seconds, whereas the ASCII export needed only a few seconds. Surely, it depends on the network.
Wouldn't it be better to use some buffer or "virtual" file in memory to do the export and then to write it to the real file location only once. Then the export should be quite quick. In case of extraordinary big files this could lead to memory problems but I cannot imagine someone exporting an STL file of several giga bytes.
The ASCII export already uses a buffer so that there is only one file access per triangle. At least it would be good that binary export does the same. But I personally would prefer one file access per export.
What do you think about it?
The reason is that in RWStl::WriteAscii the file is accessed once for every triangle. In RWStl::WriteBinary it is even worse. The file is accessed 13 times for every triangle. If a model contains many triangles this can be quite noticeable and block the export for a while. In a test, for 6000 triangles the binary export needed 25 seconds, whereas the ASCII export needed only a few seconds. Surely, it depends on the network.
Wouldn't it be better to use some buffer or "virtual" file in memory to do the export and then to write it to the real file location only once. Then the export should be quite quick. In case of extraordinary big files this could lead to memory problems but I cannot imagine someone exporting an STL file of several giga bytes.
The ASCII export already uses a buffer so that there is only one file access per triangle. At least it would be good that binary export does the same. But I personally would prefer one file access per export.
What do you think about it?
Steps to reproduce
Copy the appended brep file to a network location. Start Draw and change the directory to the network location.
Then run the following commands to see the difference between ASCII and binary export. The file contains ~ 6000 triangles. In case the difference is not noticable you can increase it via incmesh command.
restore cylinder.brep c
whatis c
trinfo c
writestl p c1.stl 0
writestl p c2.stl 1
Then run the following commands to see the difference between ASCII and binary export. The file contains ~ 6000 triangles. In case the difference is not noticable you can increase it via incmesh command.
restore cylinder.brep c
whatis c
trinfo c
writestl p c1.stl 0
writestl p c2.stl 1
Public activity
13 archived notes
Participants are labeled by their role within this record.
Branch [archived branch] has been created by Participant.
[revision removed]
Detailed log of new commits:
Author: abv
Date: Thu Feb 25 07:30:18 2016 +0300
0026338: STL export (especially binary) needs a lot of time if selected export path is not local
Method StlAPI_Writer::Write() is reimplemented to write triangulation directly, without conversion to StlMesh_Mesh.
New DRAW command "tessellate" is added to generate rapidly triangulation of prescribed size (on surface).
Command "tricheck" is protected to deal correctly with triangulation without UV data.
New tests added: perf de bug26338_1 and _2
[revision removed]
Detailed log of new commits:
Author: abv
Date: Thu Feb 25 07:30:18 2016 +0300
0026338: STL export (especially binary) needs a lot of time if selected export path is not local
Method StlAPI_Writer::Write() is reimplemented to write triangulation directly, without conversion to StlMesh_Mesh.
New DRAW command "tessellate" is added to generate rapidly triangulation of prescribed size (on surface).
Command "tricheck" is protected to deal correctly with triangulation without UV data.
New tests added: perf de bug26338_1 and _2
Fix pushed to CR26338, please review.
Note that the problem was more than just inefficiency of the file access (OSD_File implementation). The whole procedure was organized very inefficiently: instead of writing triangulation from shape directly, it converted it to StlMesh data structures which are poorly designed and extremely slow for big meshes. This component should be completely redesigned. Meanwhile I have re-implemented StlAPI_Writer class to write shape triangulation directly to STL.
The result is significant boost of performance in all modes.
CPU time of writing big mesh (2M triangles) on my workstation (Intel i5-3450):
| master | CR26338
----------------------------------------------------------
binary STL, local drive | 17 s | 0.26 s
Ascii STL, local drive | 33 s | 14.8 s (mostly spent by printf())
binary STL, network drive | -- | 11 s
Ascii STL, network drive | -- | 75 s
(here I could not wait till master finished writing to network drive, and killed process after waiting for 1.5 hours)
Elapsed time of writing mesh with 2K triangles to network drive:
| master | CR26338
------------------------------------------------------
binary STL | 97 s | 0.11 s
Ascii STL | 7 s | 0.23 s
Note that the problem was more than just inefficiency of the file access (OSD_File implementation). The whole procedure was organized very inefficiently: instead of writing triangulation from shape directly, it converted it to StlMesh data structures which are poorly designed and extremely slow for big meshes. This component should be completely redesigned. Meanwhile I have re-implemented StlAPI_Writer class to write shape triangulation directly to STL.
The result is significant boost of performance in all modes.
CPU time of writing big mesh (2M triangles) on my workstation (Intel i5-3450):
| master | CR26338
----------------------------------------------------------
binary STL, local drive | 17 s | 0.26 s
Ascii STL, local drive | 33 s | 14.8 s (mostly spent by printf())
binary STL, network drive | -- | 11 s
Ascii STL, network drive | -- | 75 s
(here I could not wait till master finished writing to network drive, and killed process after waiting for 1.5 hours)
Elapsed time of writing mesh with 2K triangles to network drive:
| master | CR26338
------------------------------------------------------
binary STL | 97 s | 0.11 s
Ascii STL | 7 s | 0.23 s
Branch [archived branch] has been updated forcibly by Participant.
[revision removed]
[revision removed]
Branch [archived branch] has been updated forcibly by Participant.
[revision removed]
[revision removed]
Branch [archived branch] has been created by Participant.
[revision removed]
Detailed log of new commits:
Author: abv
Date: Thu Feb 25 07:30:18 2016 +0300
0026338: STL export (especially binary) needs a lot of time if selected export path is not local
Method StlAPI_Writer::Write() is reimplemented to write triangulation directly, without conversion to StlMesh_Mesh.
New DRAW command "tessellate" is added to generate rapidly triangulation of prescribed size (on surface).
Command "tricheck" is protected to deal correctly with triangulation without UV data.
New tests added: perf de bug26338_1 and _2; bugs stlvrml bug26338
[revision removed]
Detailed log of new commits:
Author: abv
Date: Thu Feb 25 07:30:18 2016 +0300
0026338: STL export (especially binary) needs a lot of time if selected export path is not local
Method StlAPI_Writer::Write() is reimplemented to write triangulation directly, without conversion to StlMesh_Mesh.
New DRAW command "tessellate" is added to generate rapidly triangulation of prescribed size (on surface).
Command "tricheck" is protected to deal correctly with triangulation without UV data.
New tests added: perf de bug26338_1 and _2; bugs stlvrml bug26338
Please test the patch.
Branch [archived branch] has been updated forcibly by Participant.
[revision removed]
[revision removed]
Dear Commenter 1,
Branch CR26338_1 from occt git-repository (and master from products git-repository) was compiled on Linux, MacOS and Windows platforms and tested.
[revision removed]
Number of compiler warnings:
occt component:
Linux: 0 (0 on master)
Windows: 0 (0 on master)
MacOS: 0 (0 on master)
products component:
Linux: 41 (41 on master)
Windows: 0 (0 on master)
Regressions/Differences:
Not detected
Testing cases:
bugs stlvrml bug26338 - OK
http://occt-tests/CR26338-1-master-occt-64/Debian70-64/bugs/stlvrml/bug26338.html
http://occt-tests/CR26338-1-master-occt-64/Windows-64-VC10/bugs/stlvrml/bug26338.html
perf de bug26338_1 - OK
http://occt-tests/CR26338-1-master-occt-64/Debian70-64/perf/de/bug26338_1.html
http://occt-tests/CR26338-1-master-occt-64/Windows-64-VC10/perf/de/bug26338_1.html
perf de bug26338_2 - OK
http://occt-tests/CR26338-1-master-occt-64/Debian70-64/perf/de/bug26338_2.html
http://occt-tests/CR26338-1-master-occt-64/Windows-64-VC10/perf/de/bug26338_2.html
Testing on Linux:
Total MEMORY difference: 90291178 / 90553839 [-0.29%]
Total CPU difference: 19234.78999999991 / 19576.260000000108 [-1.74%]
Testing on Windows:
Total MEMORY difference: 57382759 / 57104393 [+0.49%]
Total CPU difference: 18258.04503799906 / 17709.873124098754 [+3.10%]
Branch CR26338_1 from occt git-repository (and master from products git-repository) was compiled on Linux, MacOS and Windows platforms and tested.
[revision removed]
Number of compiler warnings:
occt component:
Linux: 0 (0 on master)
Windows: 0 (0 on master)
MacOS: 0 (0 on master)
products component:
Linux: 41 (41 on master)
Windows: 0 (0 on master)
Regressions/Differences:
Not detected
Testing cases:
bugs stlvrml bug26338 - OK
http://occt-tests/CR26338-1-master-occt-64/Debian70-64/bugs/stlvrml/bug26338.html
http://occt-tests/CR26338-1-master-occt-64/Windows-64-VC10/bugs/stlvrml/bug26338.html
perf de bug26338_1 - OK
http://occt-tests/CR26338-1-master-occt-64/Debian70-64/perf/de/bug26338_1.html
http://occt-tests/CR26338-1-master-occt-64/Windows-64-VC10/perf/de/bug26338_1.html
perf de bug26338_2 - OK
http://occt-tests/CR26338-1-master-occt-64/Debian70-64/perf/de/bug26338_2.html
http://occt-tests/CR26338-1-master-occt-64/Windows-64-VC10/perf/de/bug26338_2.html
Testing on Linux:
Total MEMORY difference: 90291178 / 90553839 [-0.29%]
Total CPU difference: 19234.78999999991 / 19576.260000000108 [-1.74%]
Testing on Windows:
Total MEMORY difference: 57382759 / 57104393 [+0.49%]
Total CPU difference: 18258.04503799906 / 17709.873124098754 [+3.10%]
Branch [archived branch] has been updated by Participant.
[revision removed]
Detailed log of new commits:
Author: apv
Date: Mon Feb 29 12:26:11 2016 +0300
Correction of testing environment
[revision removed]
Detailed log of new commits:
Author: apv
Date: Mon Feb 29 12:26:11 2016 +0300
Correction of testing environment
Why is the runtime of Tests on Windows worse ([+3.10%])?
These are fluctuations due to instability of testing environment (hardware / software). We are working on improving the tests stability, but it is not an easy subject. Note that tests are executed in parallel thus there is a certain amount of randomness.
Branch [archived branch] has been deleted by Commenter 3.
[revision removed]
[revision removed]
Branch [archived branch] has been deleted by Commenter 3.
[revision removed]
[revision removed]
Related records
- #0028840 · related to · closedData Exchange - rewrite the STL Reader/Writer
- #0028938 · parent of · closedTests - remove useless file artifacts of test cases perf/de/bug26338
- #0027561 · related to · closedData Exchange - since OCCT 7.0.0, exporting a curve to STL creates a file that results in an endless loop when read