Archived issue #0029081
Foundation Classes, OSD_OpenStream - handle UNICODE file paths specifically in case of Mingw-w64
Description
Mingw-w64 does not provide the non-standard Microsoft extension to open ifstreams and ofstreams using wchar_t* file names.
To be able to open the respective streams without the Microsoft extension one would have to subclass std::ifstream and std::ofstream and add an open method to the child classes that accepts wchar_t* file names.
This is exactly what has been done by the boost Filesystem library. If boost was added as an (optional) dependency to Open CASCADE, unicode paths would work also with Mingw-w64.
To be able to open the respective streams without the Microsoft extension one would have to subclass std::ifstream and std::ofstream and add an open method to the child classes that accepts wchar_t* file names.
This is exactly what has been done by the boost Filesystem library. If boost was added as an (optional) dependency to Open CASCADE, unicode paths would work also with Mingw-w64.
Steps to reproduce
Not required
Public activity
25 archived notes
Participants are labeled by their role within this record.
Branch [archived branch] has been created by Participant ul Azam.
[revision removed]
Detailed log of new commits:
Author: Zia ul Azam
Date: Wed Sep 6 10:19:37 2017 +0200
0029081: With Mingw-w64 Unicode Paths Do Not Work
Boost file streams are used on Windows if boost is present to enable
handling unicode paths also with Mingw-w64.
Author: Zia ul Azam
Date: Wed Sep 6 10:14:53 2017 +0200
0029081: With Mingw-w64 Unicode Paths Do Not Work
Added boost as an optional external library.
[revision removed]
Detailed log of new commits:
Author: Zia ul Azam
Date: Wed Sep 6 10:19:37 2017 +0200
0029081: With Mingw-w64 Unicode Paths Do Not Work
Boost file streams are used on Windows if boost is present to enable
handling unicode paths also with Mingw-w64.
Author: Zia ul Azam
Date: Wed Sep 6 10:14:53 2017 +0200
0029081: With Mingw-w64 Unicode Paths Do Not Work
Added boost as an optional external library.
Branch [archived branch] has been updated by Participant ul Azam.
[revision removed]
Detailed log of new commits:
Author: Zia ul Azam
Date: Wed Sep 6 16:09:32 2017 +0200
0029081: With Mingw-w64 Unicode Paths Do Not Work
Added documentation of cmake flags used for using and installing boost.
[revision removed]
Detailed log of new commits:
Author: Zia ul Azam
Date: Wed Sep 6 16:09:32 2017 +0200
0029081: With Mingw-w64 Unicode Paths Do Not Work
Added documentation of cmake flags used for using and installing boost.
Branch [archived branch] has been updated by Author.
[revision removed]
Detailed log of new commits:
Author: Benjamin Bihler
Date: Wed Sep 6 16:14:47 2017 +0200
0029081: With Mingw-w64 Unicode Paths Do Not Work
Corrected boost header file path.
[revision removed]
Detailed log of new commits:
Author: Benjamin Bihler
Date: Wed Sep 6 16:14:47 2017 +0200
0029081: With Mingw-w64 Unicode Paths Do Not Work
Corrected boost header file path.
Branch [archived branch] has been updated by Author.
[revision removed]
Detailed log of new commits:
Author: Benjamin Bihler
Date: Wed Sep 6 16:20:38 2017 +0200
0029081: With Mingw-w64 Unicode Paths Do Not Work
Harmonized capitalization.
[revision removed]
Detailed log of new commits:
Author: Benjamin Bihler
Date: Wed Sep 6 16:20:38 2017 +0200
0029081: With Mingw-w64 Unicode Paths Do Not Work
Harmonized capitalization.
Hello Benjamin and Zia,
Sorry for delay with reply on this issue.
I have tried to apply these changes and, alas, I cannot make it working.
What I have is:
- Windows 10 Pro 64-bit workstation
- MinGW-W64-builds-4.3.3 downloaded from https://netcologne.dl.sourceforge.net/project/mingw-w64/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/7.1.0/threads-win32/sjlj/x86_64-7.1.0-release-win32-sjlj-rt_v5-rev2.7z
- Boost 1.65.1 downloaded from http://www.boost.org[user path removed]/ and built with MinGW (target "gcc") using instructions found at http://www.boost.org/doc/libs/1_65_1/more/getting_started/windows.html#or-build-binaries-from-source
- OCCT current master + fix (branch CR29081_1) built using CMake with generator "MinGW Makefiles"
The problem is that Boost streams do not work correctly under MinGW with neither char* (UTF-8) nor wchar_t* (UTF-16) names. At the end, I reduced the problem to this code:
~~~~~
boost::filesystem::ofstream test1("d:\\\xE6\x9C\x89\xE7\x94\xA8.var1", ios::out);
if (!test1.rdbuf()->is_open()) { std::cout << "Variant 1 failed" << std::endl; }
boost::filesystem::ofstream test2(L"d:\\\x6709\x7528.var2", ios::out);
if (!test2.rdbuf()->is_open()) { std::cout << "Variant 2 failed" << std::endl; }
~~~~~
Both file names represent text "it works" translated to Traditional Chinese using Google.Translate (two hieroglyphs: 有用), first one in UTF-8 and another in UTF-16.
When executed, file opening fails in variant 2; variant 1 produces file with name with UTF-8 string apparently being interpreted as if it were in the current locale.
If I set Boost locale to UTF-8, then variant 2 works just like variant 1.
For setting locale, I tried two variants (seemingly equivalent) found in:
http://www.boost.org/doc/libs/1_62_0/libs/locale/doc/html/default_encoding_under_windows.html
https://svn.boost.org/trac10/ticket/9968
Note that the same code compiled with MSVC 10 using standard streams (i.e. replacing "boost::filesystem" with "std") produces file with the same wrong name as with MinGW in variant 1 (as expected), but produces file with expected correct name "有用.var2" in variant 2.
Thus, from my testing, Boost built for MinGW does not seem to support Unicode path names.
If it works for you, this should be either due to different build of Boost, or some tricks necessary to set it working that I do not know. Please share your knowledge on this.
Sorry for delay with reply on this issue.
I have tried to apply these changes and, alas, I cannot make it working.
What I have is:
- Windows 10 Pro 64-bit workstation
- MinGW-W64-builds-4.3.3 downloaded from https://netcologne.dl.sourceforge.net/project/mingw-w64/Toolchains%20targetting%20Win64/Personal%20Builds/mingw-builds/7.1.0/threads-win32/sjlj/x86_64-7.1.0-release-win32-sjlj-rt_v5-rev2.7z
- Boost 1.65.1 downloaded from http://www.boost.org[user path removed]/ and built with MinGW (target "gcc") using instructions found at http://www.boost.org/doc/libs/1_65_1/more/getting_started/windows.html#or-build-binaries-from-source
- OCCT current master + fix (branch CR29081_1) built using CMake with generator "MinGW Makefiles"
The problem is that Boost streams do not work correctly under MinGW with neither char* (UTF-8) nor wchar_t* (UTF-16) names. At the end, I reduced the problem to this code:
~~~~~
boost::filesystem::ofstream test1("d:\\\xE6\x9C\x89\xE7\x94\xA8.var1", ios::out);
if (!test1.rdbuf()->is_open()) { std::cout << "Variant 1 failed" << std::endl; }
boost::filesystem::ofstream test2(L"d:\\\x6709\x7528.var2", ios::out);
if (!test2.rdbuf()->is_open()) { std::cout << "Variant 2 failed" << std::endl; }
~~~~~
Both file names represent text "it works" translated to Traditional Chinese using Google.Translate (two hieroglyphs: 有用), first one in UTF-8 and another in UTF-16.
When executed, file opening fails in variant 2; variant 1 produces file with name with UTF-8 string apparently being interpreted as if it were in the current locale.
If I set Boost locale to UTF-8, then variant 2 works just like variant 1.
For setting locale, I tried two variants (seemingly equivalent) found in:
http://www.boost.org/doc/libs/1_62_0/libs/locale/doc/html/default_encoding_under_windows.html
https://svn.boost.org/trac10/ticket/9968
Note that the same code compiled with MSVC 10 using standard streams (i.e. replacing "boost::filesystem" with "std") produces file with the same wrong name as with MinGW in variant 1 (as expected), but produces file with expected correct name "有用.var2" in variant 2.
Thus, from my testing, Boost built for MinGW does not seem to support Unicode path names.
If it works for you, this should be either due to different build of Boost, or some tricks necessary to set it working that I do not know. Please share your knowledge on this.
Branch [archived branch] has been created by Participant.
[revision removed]
Detailed log of new commits:
Author: Zia ul Azam
Date: Wed Sep 6 11:14:53 2017 +0300
0029081: With Mingw-w64 Unicode Paths Do Not Work
Added boost as an optional external library.
Added documentation of cmake flags used for using and installing boost.
Boost file streams are used on Windows if boost is present to enable handling unicode paths also with Mingw-w64.
[revision removed]
Detailed log of new commits:
Author: Zia ul Azam
Date: Wed Sep 6 11:14:53 2017 +0300
0029081: With Mingw-w64 Unicode Paths Do Not Work
Added boost as an optional external library.
Added documentation of cmake flags used for using and installing boost.
Boost file streams are used on Windows if boost is present to enable handling unicode paths also with Mingw-w64.
Branch CR29081_1 is the same as CR29081, rebased on current master and with all commits suashed (plus some corrections for building DRAW).
Note that as soon as Boost headers and relevant logic is present in OSD_OpenFile.hxx, there is no need to replicate it whenever streams are used. Instead, we can define typedefs for different kinds of streams used (e.g. Standard_OFStream for either std::ofstream or boost::filesystem::OFStream) in this header, and use them throughout the code. In this case OSD_OpenFile.hxx will be the single place where Boost is mentioned.
Note that as soon as Boost headers and relevant logic is present in OSD_OpenFile.hxx, there is no need to replicate it whenever streams are used. Instead, we can define typedefs for different kinds of streams used (e.g. Standard_OFStream for either std::ofstream or boost::filesystem::OFStream) in this header, and use them throughout the code. In this case OSD_OpenFile.hxx will be the single place where Boost is mentioned.
Hello Andrey,
unfortunately you are right. :-((( I have found this issue: https://svn.boost.org/trac10/ticket/5769 which seems to explain everything. Our solution seems to have worked for us since we have used special characters that are part of our current Windows code page. So the patch is partially beneficial (without Boost it was not even possible to read/write such files), but even with Boost not all unicode paths work with MinGW (they do work with Boost and MSVC).
How to continue? The current patch is still useful for us, but it is not what it has promised to be.
Benjamin
unfortunately you are right. :-((( I have found this issue: https://svn.boost.org/trac10/ticket/5769 which seems to explain everything. Our solution seems to have worked for us since we have used special characters that are part of our current Windows code page. So the patch is partially beneficial (without Boost it was not even possible to read/write such files), but even with Boost not all unicode paths work with MinGW (they do work with Boost and MSVC).
How to continue? The current patch is still useful for us, but it is not what it has promised to be.
Benjamin
> Our solution seems to have worked for us since we have used special characters
> that are part of our current Windows code page.
> So the patch is partially beneficial
> (without Boost it was not even possible to read/write such files),
Working with active CodePage does not require boost - the same could be achieved by patching OSD_OpenFile to convert UTF-8 to CodePage specifically for MinGW
(I think I have written about this workaround somewhere else, but maybe not).
And for opening existing files, one may also use workaround with passing short DOS file path instead of full path (this can be also done using WinAPI functions) - short file names are still generated even on modern Windows systems for NTFS filesystem (can be disabled in system settings).
> that are part of our current Windows code page.
> So the patch is partially beneficial
> (without Boost it was not even possible to read/write such files),
Working with active CodePage does not require boost - the same could be achieved by patching OSD_OpenFile to convert UTF-8 to CodePage specifically for MinGW
(I think I have written about this workaround somewhere else, but maybe not).
And for opening existing files, one may also use workaround with passing short DOS file path instead of full path (this can be also done using WinAPI functions) - short file names are still generated even on modern Windows systems for NTFS filesystem (can be disabled in system settings).
We will look at two other libraries that might offer a solution: https://pocoproject.org/ (Boost Software License) and https://www.guelkerdev.de/projects/pathie/ (GPL).
Or are you already sure that you wouln't accept another optional library dependency even if it was a real solution for the unicode path problem?
Or are you already sure that you wouln't accept another optional library dependency even if it was a real solution for the unicode path problem?
Branch [archived branch] has been created by Commenter 4.
[revision removed]
Detailed log of new commits:
Author: Commenter 4
Date: Wed Sep 6 11:14:53 2017 +0300
0029081: With Mingw-w64 Unicode Paths Do Not Work
Define opencascade::std::fstream implementing wchar_t paths support
on Mingw using __gnu_cxx::stdio_filebuf.
[revision removed]
Detailed log of new commits:
Author: Commenter 4
Date: Wed Sep 6 11:14:53 2017 +0300
0029081: With Mingw-w64 Unicode Paths Do Not Work
Define opencascade::std::fstream implementing wchar_t paths support
on Mingw using __gnu_cxx::stdio_filebuf.
Branch [archived branch] has been created by Commenter 4.
[revision removed]
Detailed log of new commits:
Author: Commenter 4
Date: Wed Sep 6 11:14:53 2017 +0300
0029081: With Mingw-w64 Unicode Paths Do Not Work
OSD_OpenStream now uses __gnu_cxx::stdio_filebuf extension
for opening UNICODE files on MinGW when using C++ file streams.
[revision removed]
Detailed log of new commits:
Author: Commenter 4
Date: Wed Sep 6 11:14:53 2017 +0300
0029081: With Mingw-w64 Unicode Paths Do Not Work
OSD_OpenStream now uses __gnu_cxx::stdio_filebuf extension
for opening UNICODE files on MinGW when using C++ file streams.
Could you please try patch in branch CR29081_3?
For me, it works like a charm.
Branch [archived branch] has been updated forcibly by Commenter 4.
[revision removed]
[revision removed]
Kirill, you are brilliant! I cannot see any limitations, it's just great! :-)
Branch [archived branch] has been created by Participant.
[revision removed]
Detailed log of new commits:
Author: Commenter 4
Date: Wed Sep 6 11:14:53 2017 +0300
0029081: With Mingw-w64 Unicode Paths Do Not Work
OSD_OpenStream() now uses __gnu_cxx::stdio_filebuf extension for opening UNICODE files on MinGW when using C++ file streams.
Variant accepting filebuf returns bool (true if succeeded and false otherwise).
Checks of ofstream to be opened made via calls to low-level ofstream::rdbuf() are replaced by calls to ofstream::is_open(); state of the stream is also checked (to be good).
Unicode name used for test file in test bugs fclasses bug22125 is described (for possibility to check it).
[revision removed]
Detailed log of new commits:
Author: Commenter 4
Date: Wed Sep 6 11:14:53 2017 +0300
0029081: With Mingw-w64 Unicode Paths Do Not Work
OSD_OpenStream() now uses __gnu_cxx::stdio_filebuf extension for opening UNICODE files on MinGW when using C++ file streams.
Variant accepting filebuf returns bool (true if succeeded and false otherwise).
Checks of ofstream to be opened made via calls to low-level ofstream::rdbuf() are replaced by calls to ofstream::is_open(); state of the stream is also checked (to be good).
Unicode name used for test file in test bugs fclasses bug22125 is described (for possibility to check it).
Branch [archived branch] has been updated forcibly by Participant.
[revision removed]
[revision removed]
Reviewed with some amendments (for more consistent handling of possible errors) and tested; see Jenkins job CR29081-master-abv. Please consider branch CR29081_4 for integration.
Combination -
OCCT branch : [archived branch] [revision removed]
Products branch : [archived branch]
was compiled on Linux, MacOS and Windows platforms and tested on optimize mode.
Number of compiler warnings:
No new/fixed warnings
Regressions/Differences/Improvements:
No regressions/differences
CPU differences:
No differences that require special attention
Image differences :
No differences that require special attention
Memory differences :
No differences that require special attention
OCCT branch : [archived branch] [revision removed]
Products branch : [archived branch]
was compiled on Linux, MacOS and Windows platforms and tested on optimize mode.
Number of compiler warnings:
No new/fixed warnings
Regressions/Differences/Improvements:
No regressions/differences
CPU differences:
No differences that require special attention
Image differences :
No differences that require special attention
Memory differences :
No differences that require special attention
Branch [archived branch] has been deleted by Commenter 4.
[revision removed]
[revision removed]
Branch [archived branch] has been deleted by Commenter 4.
[revision removed]
[revision removed]
Branch [archived branch] has been deleted by Commenter 4.
[revision removed]
[revision removed]
Branch [archived branch] has been deleted by Commenter 4.
[revision removed]
[revision removed]
Branch [archived branch] has been deleted by Commenter 4.
[revision removed]
[revision removed]
Related records
- #0027585 · related to · closedIt is not possible to store OCAF documents to paths with special characters in their names
- #0022125 · related to · closedTCollection_ExtendedString: conversion from UTF-8 to unicode
- #0025367 · related to · closedIGES and BRep persistence - support unicode file names on Windows
- #0030403 · parent of · closedFoundation Classes - Overwriting Big "BinOcaf" Files Does Not Reduce Their Size