So, for one thing, libtests and curl tool wasn't finding libcurl DLL, that was the next issue. After fixing that, the non-unity builds started running as expected, but their unity counterparts kept failing. They run fine locally with wine64, but crash on startup on real Windows with 0xc0000374 (heap corruption) in "StackHash_a77e". [ Unity binaries are also much smaller, but I attribute that to the more concise debug info in the unity builds. Their size become similar after stripping. ]

So the next issue is to find what causes these shared/debug unity builds to fail on real Windows.

UPDATE: The issue doesn't happen with non-unicode builds:
good: https://ci.appveyor.com/project/curlorg/curl/builds/48142669/job/l64j23l9sk4jsq2p/artifacts
bad: https://ci.appveyor.com/project/curlorg/curl/builds/48142669/job/hup9wvbbiy1krhfo/artifacts

UPDATE 2: The crashing curl tool happens with unicode + debug + unity (tool) builds (both static and shared): https://ci.appveyor.com/project/curlorg/curl/builds/48143670

curl_memory.h, curl_multibyte.h and memdebug.h look suspect, but no evidence or test result.

I'm about to give up finding the issue, and instead disable UNITY for src when ENABLE_UNICODE=ON and ENABLE_DEBUG=ON (= ENABLE_CURLDEBUG=ON = -DCURLDEBUG = TrackMemory) are enabled. It doesn't look a terrible limitation.

UPDATE: It was curl_multibyte.c.

CMake, mingw-w64, gcc 8, Debug x64, Schannel, Static, Unicode, Unity (the build you just enabled unity on) failed in ci due to an ftp test. ftp testing in windows has some flakiness (#12002 for example) so I suspect it's unrelated but I set the job to rerun

The way to track this down was not the most obvious. While doing the first tests I added a way to run curl.exe after each build. This turned out to be non-trivial with the MS-DOS batch, triggering a rewrite in PowerShell that is better, and going into an all new rabbit hole (#11999). But, it resulted in CI jobs able to detect this issue, a more clearly and much quicker than ./runtests.pl failing with could not get curl version!. Locally, it needed a real Windows VM for running curl.exe. Strangely WINE was able to run these filaing curl.exes without issues.

Then it took a long series of builds to narrow this down the mininum conditions.

When this cleared up, I had some guesses where to look, but eyeballing the source didn't help. I was also looking at headers, assuming that these could affect both libcurl and curl, so missed the comment in the affected source file.

At this point I created one build with and one without the issue and planned to compare disassembled binaries. For this I had to learn how to tell CMake to use -O0 without enabling debug. This needed editing its own config file (unintuitively: .../share/cmake/Modules/Compiler/GNU.cmake, for llvm/clang) and not supported with regular CMake options or via CMakeLists.txt. But, this turned out to be a dead-end anyway. Even with non-debug -O0, the resulting objects have internal functions in different order, string optimization was still enabled and cross-references also made up too much difference in the objdump -D outputs.

Next I picked the compiler commands from the build logs and re-run them by replacing -o <obj> with -E for both builds, to compare the actual C code they're ending up with. It was interesting to realize that an almost minimal Windows curl build creates 600MB of compilation input (lib + src) without unity and 11MB (!) with unity. This explains the massive gain in compilation speed. And also the dramatic reduction in the debug info included in debug builds. A large portion of that is likely Windows headers (I haven't made tests with a Linux or macOS build):

-rw-r--r--  1 user  staff  465451081 Oct  3 10:46 lib-non_unity.c.E
-rw-r--r--  1 user  staff    8108339 Oct  3 10:44 lib-unity.c.E
-rw-r--r--  1 user  staff  134004891 Oct  3 10:47 src-non_unity.c.E
-rw-r--r--  1 user  staff    3659820 Oct  3 10:44 src-unity.c.E

Comparing texts this large was a no-go, so picked up the uninclude script from GCC, to strip system headers. Had to modify it to recognize mingw-w64 headers (x86_64-w64-mingw32/include). Since curl keeps no executable code in headers, I took a risk and also stripped include/curl and curl/lib (this will bite a bit later).

This still had too much noise, so restarted the -E pass after making sure that the order of the non-unity build sources matches the ones in Makefile.inc files. (A PR is coming up fixing a few alpha-sorting issues in those. #12014) Sorting was different because I left in -j4 for the build runs that generated the logs with the compiler commands.

At this point it was a matter of sifting through the two diffs for lib and src respectively. Disregaring the remaining header-noise, these were reasonably identical, but not completely so. Notably ((void *)0) might be ((void*)0), and ((NTSTATUS)0x00000000) is also ((NTSTATUS)0x00000000L) depending on unity and non-unity. I surmise because Windows headers' sensitivity to include order.

src diff did not reveal anything suspicious. But lib did:

Left: non-unity, Right: unity

The risk I took to strip curl/lib stuff from the .E output deleted the curl_multbyte.c reference from the src diffs, otherwise I could have caught that as well.

The fix itself was trivial, after realizing that we also use curl_multibyte.c for the curl tool.