appveyor: rewrite batch in PowerShell + CI improvements #11999

vszakats · 2023-09-30T13:39:44Z

Rewrite in PowerShell:

rewrite MS-DOS batch build script in PowerShell.
move some bash operations into native PowerShell.
fixups for PowerShell insisting on failure when a command outputs
something to stderr.
fix to actually run curl -V after every build.
(and exclude ARM64 builds.)
also say why we skipped curl -V if we had to skip.
fix CMake warnings about unused configuration variables, by adapting
these dynamically for build cases.
dedupe OpenSSL path into a variable.
disable test1451 failing with a warning anyway due to missing python
impacket. (after trying and failing to install impacket)
PowerShell promotes these warnings to errors by PowerShell. We can also
suppress they wholesale if they start causing issues in the future,
like we already to with autoreconf and ./configure.

PowerShell is better than MS-DOS batches, so the hope is this makes it
easier to extend and maintain the AppVeyor build logic. POSIX/bash isn't
supported inline by AppVeyor on Windows build machines, but we are okay
to keep it in an external script, so it's also an option.

CI improvements:

enable tests for a "unity" build job.
speed-up CI initialization by using shallow clones of the curl repo.
speed-up CMake MSVC jobs with TrackFileAccess=false.
enable parallelism in VisualStudioSolution builds.
display CMake version before builds.
always show the CPU in job names.
tell which jobs are build-only in job names.
move TESTING: value next to DISABLED_TESTS: in two jobs.
add config.log (autotools) to dumped logs (need to enable manually).

Style:

use single-quotes in YAML like we do in other CI YAML files.
It also allows to drop quoting characters and lighter to write/read.
(keep double quotes for PowerShell strings needing expansion.)

Closes #11999

vszakats · 2023-10-01T14:04:38Z

Here's the bash version, if anyone is interested. It's shorter and sweeter IMO. It's possible to use this as an external script even on AppVeyor Windows machines (needs some fiddling to make it work).

I think it comes down to which is the better language to help curl contributors maintaining this code.

#!/usr/bin/env bash

# install:

# TODO: change pathsep to ':' from ';' in jobs.
[ -n "${ADD_PATH}" ] && export PATH="${ADD_PATH}:${PATH}"

# build_script:

set -x

openssl_root_win='C:/OpenSSL-v111-Win64'
openssl_root="$(cygpath -u "${openssl_root_win}")"
if [ "${BUILD_SYSTEM}" = 'CMake' ]; then
  options=''
  [[ "${TARGET}" = *'ARM64'* ]] && SKIP_RUN='ARM64 architecture'
  [ "${OPENSSL}" = 'ON' ] && options+=" -DOPENSSL_ROOT_DIR=${openssl_root_win}"
  [ "${OPENSSL}" = 'ON' ] && options+=" -DOPENSSL_ROOT_DIR=${openssl_root_win}"
  [ "${PRJ_CFG}" = 'Debug' ] && options+=' -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_DEBUG='
  [ "${PRJ_CFG}" = 'Release' ] && options+=' -DCMAKE_RUNTIME_OUTPUT_DIRECTORY_RELEASE='
  [[ "${PRJ_GEN}" = *'Visual Studio'* ]] && options+=' -DCMAKE_VS_GLOBALS=TrackFileAccess=false'
  cmake --version
  # shellcheck disable=SC2086
  cmake . "-G${PRJ_GEN}" "${TARGET}" ${options} \
    "-DCURL_USE_OPENSSL=${OPENSSL}" \
    "-DCURL_USE_SCHANNEL=${SCHANNEL}" \
    "-DHTTP_ONLY=${HTTP_ONLY}" \
    "-DBUILD_SHARED_LIBS=${SHARED}" \
    "-DBUILD_TESTING=${TESTING}" \
    "-DENABLE_WEBSOCKETS=${WEBSOCKETS}" \
    "-DCMAKE_UNITY_BUILD=${UNITY}" \
    '-DCURL_WERROR=ON' \
    "-DENABLE_DEBUG=${DEBUG}" \
    "-DENABLE_UNICODE=${ENABLE_UNICODE}" \
    '-DCMAKE_INSTALL_PREFIX=C:/CURL' \
    "-DCMAKE_BUILD_TYPE=${PRJ_CFG}"
  # shellcheck disable=SC2086
  cmake --build . --config "${PRJ_CFG}" --parallel 2 --clean-first -- ${BUILD_OPT}
  if [ "${SHARED}" = 'ON' ]; then
    cp -f -p /c/projects/curl/lib/*.dll /c/projects/curl/src/
    cp -f -p /c/projects/curl/lib/*.dll /c/projects/curl/tests/libtest/
  fi
  if [ "${OPENSSL}" = 'ON' ]; then
    cp -f -p "${openssl_root}"/*.dll /c/projects/curl/src/
  fi
  curl='./src/curl.exe'
elif [ "${BUILD_SYSTEM}" = 'VisualStudioSolution' ]; then
  cd projects || exit
  ./generate.bat "${VC_VERSION}"
  msbuild.exe -maxcpucount "-property:Configuration=${PRJ_CFG}" "Windows\\${VC_VERSION}\\curl-all.sln"
  curl="../build/Win32/${VC_VERSION}/${PRJ_CFG}/curld.exe"
elif [ "${BUILD_SYSTEM}" = 'winbuild_vs2015' ]; then
  ./buildconf.bat
  cd winbuild || exit
  cat << EOF > _make.bat
    call "C:\\Program Files\\Microsoft SDKs\\Windows\\v7.1\\Bin\\SetEnv.cmd" /x64
    call "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\vcvarsall.bat" x86_amd64
    nmake /f Makefile.vc mode=dll VC=14 "SSL_PATH=${openssl_root}" WITH_SSL=dll MACHINE=x64 DEBUG=${DEBUG} ENABLE_UNICODE=${ENABLE_UNICODE}
EOF
  ./_make.bat
  rm _make.bat
  curl="../builds/libcurl-vc14-x64-${PATHPART}-dll-ssl-dll-ipv6-sspi/bin/curl.exe"
elif [ "${BUILD_SYSTEM}" = 'winbuild_vs2017' ]; then
  ./buildconf.bat
  cd winbuild || exit
  cat << EOF > _make.bat
    call "C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\VC\\Auxiliary\\Build\\vcvars64.bat"
    nmake /f Makefile.vc mode=dll VC=14.10 "SSL_PATH=${openssl_root}" WITH_SSL=dll MACHINE=x64 DEBUG=${DEBUG} ENABLE_UNICODE=${ENABLE_UNICODE}
EOF
  ./_make.bat
  rm _make.bat
  curl="../builds/libcurl-vc14.10-x64-${PATHPART}-dll-ssl-dll-ipv6-sspi/bin/curl.exe"
elif [ "${BUILD_SYSTEM}" = 'autotools' ]; then
  # shellcheck disable=SC2086
  cd /c/projects/curl && autoreconf -fi && ./configure ${CONFIG_ARGS} && make V=1 && make V=1 examples && cd tests && make V=1
  curl='src/curl.exe'
fi

find /c/projects/curl -name '*.exe' -o -name '*.dll'
if [ -z "${SKIP_RUN}" ]; then
  "${curl}" -V
else
  echo "Skip running curl.exe. Reason: ${SKIP_RUN}"
fi

if false; then
  for log in CMakeFiles/CMakeConfigureLog.yaml CMakeFiles/CMakeOutput.log CMakeFiles/CMakeError.log; do
    [ -r "${log}" ] && cat "${log}"
  done
fi

if [ "${TESTING}" = 'ON' ] && [ "${BUILD_SYSTEM}" = 'CMake' ]; then
  cmake --build . --config "${PRJ_CFG}" --parallel 2 --target testdeps
fi

# test_script:

set -x

if [ "${TESTING}" = 'ON' ]; then
  [ -x /c/msys64/usr/bin/curl.exe ] && acurl='-ac /c/msys64/usr/bin/curl.exe'
  [ -x /c/Windows/System32/curl.exe ] && acurl='-ac /c/Windows/System32/curl.exe'
  if [ "${BUILD_SYSTEM}" = 'CMake' ]; then
    TFLAGS="${acurl} ${DISABLED_TESTS}" cmake --build . --config "${PRJ_CFG}" --target test-ci
  elif [ "${BUILD_SYSTEM}" = 'autotools' ]; then
    cd /c/projects/curl && make V=1 TFLAGS="${acurl} ${DISABLED_TESTS}" test-ci
  else
    # shellcheck disable=SC2086
    cd /c/projects/curl/tests && ./runtests.pl -a -p !flaky -r -rm ${acurl} ${DISABLED_TESTS}
  fi
fi

vszakats · 2023-10-01T17:21:28Z

I plan to split this into 3 separate PRs (or commits), one language swap, one CI speedup/improvement and one formatting.

UPDATE: Separate PRs are painful when they depend on each other, so normalized this PR to have these 3 distinct commits.

We used it already in other CI YAML files, and also in this very file. It's lighter to read and write, and allows to use less character escaping.

vszakats · 2023-10-01T18:21:03Z

Let's stick with PowerShell for now and later move to bash if we want to.

vszakats · 2023-10-01T19:17:31Z

Ref, odd behaviour of PowerShell error handling with external commands: https://stackoverflow.com/questions/57468522/powershell-and-process-exit-codes

Things improved in 7.2 (2021-Nov). It will take a decade+ to reach our oldest-used CI image.

The main issue is that PowerShell considers an external command writing to stderr as a failure, even if the command otherwise returns success. This is accompanied by a red exception text. This also stops execution with $ErrorActionPreference = 'Stop'. This also happens when we redirect stderr to stdout or null on the PowerShell level. The solution that worked is using cmd.exe (or bash.exe) and redirect stderr to stdout inside the external shell. Oddly enough this caused one failed command to NOT cause a stop on failure, so had to add a manual check after running curl.exe and exit manually.

Let's see how real build failures stop the script. If the don't, we need to add more checks.

The linked post also talks about cases where the exit code is falsely returned to the PowerShell code. I haven't hit them while testing, fingers crossed it stays that way.

Regarding stderr: nmake wrote there, fixed by /nologo option. Debug builds of curl write there. autoreconf and ./configure write there. runtests.pl writes warnings there. What I've seen so far.

This is all so backwards, I'm still wondering if I might be holding it wrong.

- rewrite MS-DOS batch build script in PowerShell. - move some bash operations into native PowerShell. - fixups for PowerShell insisting on command failure when something is output to stderr. - fix to actually run `curl -V` after every build. (and exclude ARM64 builds.) - also say why we skipped `curl -V` if we had to skip. - fix CMake warnings about unused configuration variables, by adapting these dynamically for build cases. - dedupe OpenSSL path into a variable. - disable `test1451` failing with a warning anyway due to missing python impacket. (after trying and failing to install impacket) These warnings were promoted to errors by PowerShell. We can also suppress these wholesale if they start causing issues in the future. PowerShell is better than MS-DOS batches, so the hope is this makes it easier to extend and maintain the AppVeyor build logic. POSIX/bash isn't supported inline by AppVeyor on Windows build machines, but we are okay to keep it in an external script, it's also an option.

- enable tests for a "unity" build job. - speed-up CI initialization by using shallow clones of the curl repo. - speed-up CMake MSVC jobs with `TrackFileAccess=false`. - enable parallelism in `VisualStudioSolution` builds. - display CMake version before builds. - always show the CPU in job names. - tell which jobs are build-only in job names. - move `TESTING:` value next to `DISABLED_TESTS:` in two jobs. - add `config.log` (autotools) to dumped logs (need to enable manually).

jay · 2023-10-01T21:42:16Z

Ref, odd behaviour of PowerShell error handling with external commands: https://stackoverflow.com/questions/57468522/powershell-and-process-exit-codes

I'd watch out for that batch file exit code trap, I fell into it once several years ago. https://stackoverflow.com/questions/48739163/cmd-c-batch-file-exit-code-is-sometimes-0-even-on-error

vszakats · 2023-10-01T22:46:52Z

@jay: Thanks, this points to the fragility of command-chaining in batches. It was an important reason that triggered this rewrite. Unless I'm misreading, this is a generic batch issue, not a PowerShell-specific one.

We call into project\generate.bat and .\buildconf.bat (without cmd.exe). They worked before, so I'm guessing they will continue to do so. If we hit an edge case, we can tweak them or might consider replacing them with PowerShell.

The others we have to call are the MSVC/PSDK "setenv" batches (with cmd.exe). We're not interested in their exit code, only the envvars they set, which we pull via a "neat" trick into the PowerShell env. If these break, the compiler will most likely fail in some ways afterwards. There is not much we can do about them, unless there is a better official way to setup the MSVC compiler env. Maybe newer releases offer a PowerShell script (or auto-detect their env?), I don't know. (The manual way is painful.)

I've seen the test stage failing already, so that works. I haven't tested with a broken build, but chances are good that an error from cmake, nmake, msbuild, bash are correctly picked up, too. We will see.

PowerShell works (after a steep development curve), but one property of it stuck and kept causing unresolvable usability issues: With `$ErrorActionPreference=Stop`, it does abort on failures, but shows only the first line of the error message. In `Continue` mode, it shows the full error message, but doesn't stop on all errors. Another issue is PowerShell considering any stderr output as if the command failed (this has been improved in 7.2 (2021-Nov), but fixed versions aren't running in CI and will not be for a long time in all test images.) Thus, we're going with bash. Also: - use `-j2` with autotools tests, making them finish 5-15 minutes per job faster. - omit `POSIX_PATH_PREFIX`. - use `WINDIR`. - prefer forward slashes. Follow-up to: 75078a4 #11999 Ref: #12444 Fixes #12560 Closes #12572

github-actions bot added the CI Continuous Integration label Sep 30, 2023

vszakats force-pushed the appveyor-posix-shell branch 7 times, most recently from ee9aeb2 to 9ddf8f2 Compare October 1, 2023 12:13

vszakats changed the title ~~appveyor: convert to POSIX shell~~ appveyor: convert batch code to PowerShell Oct 1, 2023

vszakats force-pushed the appveyor-posix-shell branch from 9ddf8f2 to 8e45f3a Compare October 1, 2023 12:44

vszakats changed the title ~~appveyor: convert batch code to PowerShell~~ appveyor: convert batch to PowerShell + improvements Oct 1, 2023

vszakats added the Windows Windows-specific label Oct 1, 2023

vszakats force-pushed the appveyor-posix-shell branch from ceac28c to e7e94a9 Compare October 1, 2023 13:35

vszakats changed the title ~~appveyor: convert batch to PowerShell + improvements~~ appveyor: convert batch to PowerShell (or bash) + improvements Oct 1, 2023

vszakats changed the title ~~appveyor: convert batch to PowerShell (or bash) + improvements~~ appveyor: rewrite batch to PowerShell (or bash) + improvements Oct 1, 2023

appveyor: use single quotes

498412a

We used it already in other CI YAML files, and also in this very file. It's lighter to read and write, and allows to use less character escaping.

vszakats force-pushed the appveyor-posix-shell branch from bc96087 to 9262af0 Compare October 1, 2023 18:14

vszakats changed the title ~~appveyor: rewrite batch to PowerShell (or bash) + improvements~~ appveyor: rewrite batch in PowerShell + improvements Oct 1, 2023

vszakats changed the title ~~appveyor: rewrite batch in PowerShell + improvements~~ appveyor: rewrite batch in PowerShell + CI improvements Oct 1, 2023

vszakats force-pushed the appveyor-posix-shell branch from 9262af0 to 41d1905 Compare October 1, 2023 18:50

vszakats added 2 commits October 1, 2023 20:16

vszakats force-pushed the appveyor-posix-shell branch from 41d1905 to 52031ad Compare October 1, 2023 20:16

vszakats closed this in 75078a4 Oct 2, 2023

github-actions bot added the hacktoberfest-accepted label Oct 2, 2023

vszakats deleted the appveyor-posix-shell branch October 2, 2023 22:51

vszakats removed the hacktoberfest-accepted label Oct 2, 2023

vszakats mentioned this pull request Oct 3, 2023

cmake: fix unity with Windows Unicode + TrackMemory #11928

Closed

This was referenced Dec 20, 2023

AppVeyor build failures do not always show in log output #12560

Closed

appveyor: replace PowerShell with bash + parallel autotools #12572

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

appveyor: rewrite batch in PowerShell + CI improvements #11999

appveyor: rewrite batch in PowerShell + CI improvements #11999

vszakats commented Sep 30, 2023 •

edited

vszakats commented Oct 1, 2023 •

edited

vszakats commented Oct 1, 2023 •

edited

vszakats commented Oct 1, 2023

vszakats commented Oct 1, 2023 •

edited

jay commented Oct 1, 2023

vszakats commented Oct 1, 2023 •

edited

appveyor: rewrite batch in PowerShell + CI improvements #11999

appveyor: rewrite batch in PowerShell + CI improvements #11999

Conversation

vszakats commented Sep 30, 2023 • edited

vszakats commented Oct 1, 2023 • edited

vszakats commented Oct 1, 2023 • edited

vszakats commented Oct 1, 2023

vszakats commented Oct 1, 2023 • edited

jay commented Oct 1, 2023

vszakats commented Oct 1, 2023 • edited

vszakats commented Sep 30, 2023 •

edited

vszakats commented Oct 1, 2023 •

edited

vszakats commented Oct 1, 2023 •

edited

vszakats commented Oct 1, 2023 •

edited

vszakats commented Oct 1, 2023 •

edited