Lessons learned: maybe we can modify the CI to check for an expected minimum number of successful tests being run.

TESTFAIL: No tests were performed this in particular should at least be a reason for failure in the CI....

#6053 seems to do its job for the appveyor builds...

/cc @Lekensteyn

When cmake is used on appveyor, make test is not invoked and thus the test servers and tools are not built!?

Was that perhaps done by the default cmake --build before?

Was that perhaps done by the default cmake --build before?

Seems like this commit c2ab249 before ad64169 is the root cause for this.

It seems cmake --build . -- testdeps after the build will build the stuff in tests/.

@bagder

When cmake is used on appveyor, make test is not invoked and thus the test servers and tools are not built!?

Was that perhaps done by the default cmake --build before?

That's what the -DBUILD_TESTING CMake argument is or was for.

Ah right. But that doesn't seem to work (anymore)...

I've tried numerous things now but I can't seem to fix this. I need help!

@Lekensteyn @MarcelRaad can you a take a look at this maybe? Also: would #6036 be an option to solve this?

@Lekensteyn @MarcelRaad can you a take a look at this maybe?

I'm not familiar enough with CMake to help here, sorry.

Perhaps appveyor updated their cmake? https://www.appveyor.com/updates/

I'm not 100% sure but suspect the following removal was the root cause.
So

option(BUILD_TESTING "Build tests" "${PERL_FOUND}")
if(NOT PERL_FOUND)
  message(STATUS "Perl not found, testing disabled.")
elseif(BUILD_TESTING)
  include(CTest)  ### will enable_testing()
  add_subdirectory(tests)
endif()

After digging changes and logs, looks like it was an intentional change to avoid test target name clash.
It looks like the following command should build dependencies and then run tests cmake --build . --target test.
Will try to debug/fix it with AppVeyor on the local project clone.

@mback2k Could you please check if it's looking good for you? https://ci.appveyor.com/project/snikulov/curl-ckq2k/build/job/5hn7cqysanf4df0p and https://ci.appveyor.com/project/snikulov/curl-ckq2k

@snikulov looks better, but the test flags seem to be missing and I don't get the point of your second commit. TFLAGS should work on Windows. For AppVeyor the tests should be build by CMake, but then run using the existing run bash.exe runtests.pl execution.

@mback2k yes, TFLAGS should work, but using ${...} is Unix specific. For Windows native shell it should be %...% AFAIK. I'm opposed bash usage, but you can try to run CMake from bash. Probably it will work.

And my assumption not working :( Adding %TFLAGS% to the command produce the error:

Building Custom Rule C:/projects/curl-ckq2k/tests/CMakeLists.txt
  Unknown option: %TFLAGS%

@bagder @mback2k I'm not familiar with Perl, so could you please explain where env variable TFLAGS comes from?
Adding a custom command in CMake will be something like that
perl runtests.pl [options] env{TFLAGS}.
When and where TFLAGS is defined? Is it available during configuration? Is it possible to pass TFLAGS through file?

Okay, it seems like TFLAGS as an environment variable for make is not currently used in appveyor.yml, instead the following arguments are passed directly to runtests.pl: -a -b$(($(echo '%APPVEYOR_API_URL%' | cut -d'/' -f3 | cut -d':' -f2)+1)) -p !flaky %DISABLED_TESTS%.

You will at least need to find a way to pass the following via cmake to runtests.pl: -a -p !flaky %DISABLED_TESTS%. The parameter -b is not important anymore as the tests use dynamic ports anyway.

yeah... still digging :) Added - cmd: set TFLAGS=%DISABLED_TESTS% to appveyor.yml try to figure out will it helps

@mback2k Could you please check? https://ci.appveyor.com/project/snikulov/curl-ckq2k
Looks like it works now.
I removed curly from ${TFLAGS} placeholder because seems to me it was incorrectly processed with CMake or Perl or something else (was cut off).
Now it works for Win32 as well as on Linux.

@snikulov Looks much better, but I don't yet see the arguments -a and -p being passed into runtests.pl.

@mback2k How can you see this?
See this. Target test-nonflaky will set those switches.
I execute exactly this target

@snikulov ah, thank you very much. I missed that and already wondered if that target is CMake specific and how !flaky is passed.

@mback2k @bagder BTW, should we also enable tests on Travis CMake builds? After this change it's now possible.

Yes, we should!

Ok then. Will submit another PR soon.