Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IDN tests failing on Windows / MSYS2 #3747

Closed
MarcelRaad opened this issue Apr 7, 2019 · 1 comment
Closed

IDN tests failing on Windows / MSYS2 #3747

MarcelRaad opened this issue Apr 7, 2019 · 1 comment
Assignees
@MarcelRaad
Labels
cmdline tool KNOWN_BUGS material tests Windows Windows-specific

Comments

@MarcelRaad
Copy link
Member

MarcelRaad commented Apr 7, 2019 •
edited

I did this

See for example Marc's 6DY-SHZ-5KMWP2---- autobuilds, like this one:
https://curl.haxx.se/dev/log.cgi?id=20190407081751-30470

Note this:

=== Start of file stderr165
curl: (3) Failed to convert www.[EF][BF][BD][EF][BF][BD][EF][BF][BD].se to ACE;
curl: (3) Failed to convert www.gro[EF][BF][BD]e.de to ACE;
=== End of file stderr165

which doesn't happen when run in gdb, which is a native MSYS program, via ./runtests.pl -g (and the original URLs are printed correctly then).
So it seems like MSYS2 does some UTF-8-to-something-else conversion for Windows compatibility.

I expected the following

The tests to either succeed or not be executed (but preferably succeed as the MSYS2 terminal itself is UTF-8 capable).

curl/libcurl version

post-7.64.1 master 2f44e94

operating system

MSYS2 with MinGW-w64 on Windows 10
@MarcelRaad MarcelRaad self-assigned this Apr 8, 2019
MarcelRaad added a commit to MarcelRaad/curl that referenced this issue Apr 13, 2019
@MarcelRaad
tool: support UTF-16 command line on Windows
7343ecf 
- use wmain instead of main when _UNICODE is defined [0]
- define argv_item_t as wchar_t * in this case
- use the curl_multibyte gear to convert the command-line arguments to
  UTF-8
- make a few char pointers point to const char instead

This makes it possible to pass parameters with characters outside of
the current locale on Windows, which is required for some tests, e.g.
the IDN tests. Out of the box, this currently only works with the
Visual Studio project files, which default to Unicode.

[0] https://devblogs.microsoft.com/oldnewthing/?p=40643

Ref: curl#3747
Closes
@MarcelRaad MarcelRaad mentioned this issue Apr 13, 2019
Closed
MarcelRaad added a commit to MarcelRaad/curl that referenced this issue Apr 14, 2019
@MarcelRaad
tool: support UTF-16 command line on Windows
fea657a 
- use wmain instead of main when _UNICODE is defined [0]
- define argv_item_t as wchar_t * in this case
- use the curl_multibyte gear to convert the command-line arguments to
  UTF-8

This makes it possible to pass parameters with characters outside of
the current locale on Windows, which is required for some tests, e.g.
the IDN tests. Out of the box, this currently only works with the
Visual Studio project files, which default to Unicode.

[0] https://devblogs.microsoft.com/oldnewthing/?p=40643

Ref: curl#3747
Closes
@MarcelRaad MarcelRaad mentioned this issue Apr 15, 2019
Closed
@MarcelRaad
Copy link
Member Author

The problem is that curl receives its command-line arguments with ANSI encoding even though the perl precheck reports a UTF-8 locale. Using Windows's native UTF-16 for them as done in #3784 fixes this issue.

Only test 165 is then still failing because of ß handling in IDNA 2003 vs. 2008.

@bagder bagder closed this as completed in b8ea432 Oct 9, 2019
MarcelRaad added a commit to MarcelRaad/curl that referenced this issue Jan 2, 2020
@MarcelRaad
tool: support UTF-16 command line on Windows
3b458c9 
- use wmain instead of main when _UNICODE is defined [0]
- define argv_item_t as wchar_t * in this case
- use the curl_multibyte gear to convert the command-line arguments to
  UTF-8

This makes it possible to pass parameters with characters outside of
the current locale on Windows, which is required for some tests, e.g.
the IDN tests. Out of the box, this currently only works with the
Visual Studio project files, which default to Unicode.

[0] https://devblogs.microsoft.com/oldnewthing/?p=40643

Ref: curl#3747
Closes curl#3784
MarcelRaad added a commit to MarcelRaad/curl that referenced this issue Jan 3, 2020
@MarcelRaad
tool: support UTF-16 command line on Windows
177fb0d 
- use wmain instead of main when _UNICODE is defined [0]
- define argv_item_t as wchar_t * in this case
- use the curl_multibyte gear to convert the command-line arguments to
  UTF-8

This makes it possible to pass parameters with characters outside of
the current locale on Windows, which is required for some tests, e.g.
the IDN tests. Out of the box, this currently only works with the
Visual Studio project files, which default to Unicode.

[0] https://devblogs.microsoft.com/oldnewthing/?p=40643

Ref: curl#3747
Closes curl#3784
MarcelRaad added a commit to MarcelRaad/curl that referenced this issue Jan 3, 2020
@MarcelRaad
tool: support UTF-16 command line on Windows
61ec4fb 
- use wmain instead of main when _UNICODE is defined [0]
- define argv_item_t as wchar_t * in this case
- use the curl_multibyte gear to convert the command-line arguments to
  UTF-8

This makes it possible to pass parameters with characters outside of
the current locale on Windows, which is required for some tests, e.g.
the IDN tests. Out of the box, this currently only works with the
Visual Studio project files, which default to Unicode.

[0] https://devblogs.microsoft.com/oldnewthing/?p=40643

Ref: curl#3747
Closes curl#3784
MarcelRaad added a commit to MarcelRaad/curl that referenced this issue Jan 3, 2020
@MarcelRaad
tool: support UTF-16 command line on Windows
cf0702b 
- use `wmain` instead of `main` when `_UNICODE` is defined [0]
- define `argv_item_t` as `wchar_t *` in this case
- use the curl_multibyte gear to convert the command-line arguments to
  UTF-8

This makes it possible to pass parameters with characters outside of
the current locale on Windows, which is required for some tests, e.g.
the IDN tests. Out of the box, this currently only works with the
Visual Studio project files, which default to Unicode.

[0] https://devblogs.microsoft.com/oldnewthing/?p=40643

Ref: curl#3747
Closes curl#3784
@lock lock bot locked as resolved and limited conversation to collaborators Jan 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@MarcelRaad MarcelRaad

Labels
cmdline tool KNOWN_BUGS material tests Windows Windows-specific
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bagder @MarcelRaad