Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmake: add support for single libcurl compilation pass #11546

Closed
wants to merge 3 commits into from

Conversation

vszakats
Copy link
Member

@vszakats vszakats commented Jul 30, 2023

Before this patch CMake builds used two separate compilation passes to
build the shared and static libcurl respectively. This patch allows to
reduce that to a single pass if the target platform and build settings
allow it.

This reduces CMake build times when building both static and shared
libcurl at the same time, making these dual builds an almost zero-cost
option.

Enable this feature for Windows builds, where the difference between the
two passes was the use of __declspec(dllexport) attribute for exported
API functions for the shared builds. This patch replaces this method
with the use of libcurl.def at DLL link time.

Also update Makefile.mk to use libcurl.def to export libcurl API
symbols on Windows. This simplifies (or fixes) this build method (e.g.
in curl-for-win, which generated a libcurl.def from .h files using
an elaborate set of transformations).

libcurl.def has the maintenance cost of keeping the list of public
libcurl API symbols up-to-date. This list seldom changes, so the cost
is low.

Closes #11546


  • curl previously had a libcurl.def placed in the lib directory. I opted for the root directory, because in lib it may be overwritten by .def files generated by the DLL linker command when using certain build methods/options.

  • single-pass libcurl builds can be freely extended to other platforms, if both shared/static libs have position-independent code enable and symbol exports can be fixed up via a linker option.

  • it might be an option to make libcurl.def an OS-agnostic raw list of public API symbols (= without the top EXPORTS line) for re-use for other purposes. CMake could handle that with some extra lines. In Makefile.mk less trivially, but also seems doable.

@vszakats vszakats added build cmake Windows Windows-specific labels Jul 30, 2023
@vszakats vszakats marked this pull request as draft July 30, 2023 13:08
@dfandrich
Copy link
Contributor

dfandrich commented Jul 30, 2023 via email

@dfandrich
Copy link
Contributor

dfandrich commented Jul 30, 2023 via email

@vszakats
Copy link
Member Author

vszakats commented Jul 30, 2023

We do use declspec(), yes, and this is what forces separate compile passes for shared and static libs. The goal of this PR is to avoid that double work. Another positive side effect is that Makefile.mk builds can also use this libcurl.def, without having to generate it manually. Makefile.mk always used a single compile pass, so a .def file was always necessary for correct results (on Windows).

The file was deleted because it wasn't used. This time it will be used by two build systems.

@vszakats vszakats marked this pull request as ready for review July 30, 2023 22:20
@vszakats vszakats changed the title cmake: add support for single libcurl compilation pass on Windows cmake: add support for single libcurl compilation pass Jul 31, 2023
@vszakats vszakats closed this in 2ebc74c Aug 1, 2023
@vszakats vszakats deleted the build-add-libcurl-def branch August 1, 2023 08:15
vszakats added a commit to curl/curl-for-win that referenced this pull request Aug 1, 2023
Stop dynamically generating `libcurl.def` with the next curl release,
which provides it and uses it automatically in GNU Make builds.

curl/curl@2ebc74c
curl/curl#11546
vszakats added a commit to vszakats/curl that referenced this pull request Aug 8, 2023
2ebc74c curl#11546 introduced sharing
libcurl objects for shared and static targets.

The above automatically enabled for Windows builds, with an option to
disable with `SHARE_LIB_OBJECT=OFF`.

This patch extend this feature for all platforms as a manual option.
You can enable it by setting `SHARE_LIB_OBJECT=ON`. Shared objects will
be built in PIC mode, meaning the static lib will also have PIC code.

EXPERIMENTAL.

Closes #xxxxx
vszakats added a commit that referenced this pull request Aug 9, 2023
2ebc74c #11546 introduced sharing
libcurl objects for shared and static targets.

The above automatically enabled for Windows builds, with an option to
disable with `SHARE_LIB_OBJECT=OFF`.

This patch extend this feature to all platforms as a manual option.
You can enable it by setting `SHARE_LIB_OBJECT=ON`. Then shared objects
are built in PIC mode, meaning the static lib will also have PIC code.

[EXPERIMENTAL]

Closes #11627
ptitSeb pushed a commit to wasix-org/curl that referenced this pull request Sep 25, 2023
Before this patch CMake builds used two separate compilation passes to
build the shared and static libcurl respectively. This patch allows to
reduce that to a single pass if the target platform and build settings
allow it.

This reduces CMake build times when building both static and shared
libcurl at the same time, making these dual builds an almost zero-cost
option.

Enable this feature for Windows builds, where the difference between the
two passes was the use of `__declspec(dllexport)` attribute for exported
API functions for the shared builds. This patch replaces this method
with the use of `libcurl.def` at DLL link time.

Also update `Makefile.mk` to use `libcurl.def` to export libcurl API
symbols on Windows. This simplifies (or fixes) this build method (e.g.
in curl-for-win, which generated a `libcurl.def` from `.h` files using
an elaborate set of transformations).

`libcurl.def` has the maintenance cost of keeping the list of public
libcurl API symbols up-to-date. This list seldom changes, so the cost
is low.

Closes curl#11546
ptitSeb pushed a commit to wasix-org/curl that referenced this pull request Sep 25, 2023
2ebc74c curl#11546 introduced sharing
libcurl objects for shared and static targets.

The above automatically enabled for Windows builds, with an option to
disable with `SHARE_LIB_OBJECT=OFF`.

This patch extend this feature to all platforms as a manual option.
You can enable it by setting `SHARE_LIB_OBJECT=ON`. Then shared objects
are built in PIC mode, meaning the static lib will also have PIC code.

[EXPERIMENTAL]

Closes curl#11627
vszakats added a commit to curl/curl-for-win that referenced this pull request Oct 25, 2023
Switch from `curl-gnumake.sh` to `curl-cmake.sh` with upcoming curl
release v8.5.0.

cmake builds are now _faster_ for Windows builds than raw gnumake
(aka `Makefile.mk`). They also use 'unity' mode, which makes builds
run fast with the side-effect of also creating potentially more
efficient binaries by allowing better compiler optimizations.

This also makes curl-for-win use the same build system for all target
platforms (`Makefile.mk` is not available for *nix platforms).

Initially on 2022-07-04 cmake was 25% slower than gnumake. By
2022-09-26 this reduced to 20%, by 2023-07-29 to 18% and after the
latest round of updates gnumake came out 7% slower than cmake.
This is for Windows, for a triple x64, arm64 and x86 build. In
absolute times this is 27m22s for cmake and 29m11s for gnumake.

(FWIW autotools builds are 52% slower than cmake unity builds now.)

Making cmake builds fast was a multi-year effort with these major steps:

1. add support for cmake builds in curl-for-win.
   420e73c
2. bring it in-line with gnumake and autotools builds and tidy-up
   as much as possible. Scattered to a many commits.
3. delete a whole bunch of unused feature detections.
   curl/curl@4d73854
   curl/curl#9044
   (and a lot more smaller commits)
4. speed up picky warning option initialization by avoiding brute-force
   all options. (first in libssh2, then in curl, then applied also
   ngtcp2, nghttp3, nghttp2)
   curl/curl@9c543de
   curl/curl#10973
5. implement single build run for shared and static libcurl + tool
   (first in libssh2)
   curl/curl@1199308
   curl/curl#11505
   53dcd49
6. implement single build pass for shared and static libcurl
   (for Windows initially)
   curl/curl@2ebc74c
   curl/curl#11546
7. extend above to non-Windows (e.g. macOS)
   curl/curl@fc9bfb1
   curl/curl#11627
   bafa77d (mac)
   1b27304 (linux)
8. implement unity builds: single compiler invocation for libcurl + tool
   curl/curl@3f8fc25
   curl/curl#11095
   curl/curl@f42a279
   curl/curl#11928
   67d7fd3
9. speed up 4x the cmake initialization phase (for Windows)
   curl/curl@2100d9f
   curl/curl#12044
10. speed up cmake initialization even more by pre-filling
   detection results for our well-known mingw-w64 environments.
   74dd967
   5a43c61

+1: speed up autotools initialization by mitigating a brute-force
   feature detection on Windows. This reduced total build times
   by 5 minutes at the time, for the 3 Windows targets in total.
   curl/curl@6adcff6
   curl/curl#9591

Also update build times for cmake-unity and gnumake, based on runs:
cmake-unity: https://ci.appveyor.com/project/curlorg/curl-for-win/builds/48357875
gnumake: https://ci.appveyor.com/project/curlorg/curl-for-win/builds/48358005
vszakats added a commit to vszakats/curl that referenced this pull request Dec 12, 2023
- 898b012 curl#1288
- 5de6848 curl#10023
- bunch of others that are completed
- `NTLM_WB_ENABLED` is implemented in a basic form, and now also
  scheduled for removal, so a TODO at this point isn't useful.

And this 'to-check' item:

Q: "The cmake build selected to run gcc with -fPIC on my box while the
   plain configure script did not."

A: With CMake, since 2ebc74c curl#11546
   and fc9bfb1 curl#11627, we explicitly
   enable PIC for libcurl shared lib. Or when building libcurl for
   shared and static lib in a single pass. We do this by default for
   Windows or when enabled by the user via `SHARE_LIB_OBJECT`.
   Otherwise we don't touch this setting. Meaning the default set by
   CMake (if any) or the toolchain is used. On Debian Bookworm, this
   means that PIC is disabled for static libs by default. Some platforms
   (like macOS), has PIC enabled by default.
   autotools supports the double-pass mode only, and in that case
   CMake seems to match PIC behaviour now (as tested on Linux with gcc.)

Closes #xxxxx
vszakats added a commit to vszakats/curl that referenced this pull request Dec 12, 2023
- 898b012 curl#1288
- 5de6848 curl#10023
- bunch of others that are completed
- `NTLM_WB_ENABLED` is implemented in a basic form, and now also
  scheduled for removal, so a TODO at this point isn't useful.

And this 'to-check' item:

Q: "The cmake build selected to run gcc with -fPIC on my box while the
   plain configure script did not."

A: With CMake, since 2ebc74c curl#11546
   and fc9bfb1 curl#11627, we explicitly
   enable PIC for libcurl shared lib. Or when building libcurl for
   shared and static lib in a single pass. We do this by default for
   Windows or when enabled by the user via `SHARE_LIB_OBJECT`.
   Otherwise we don't touch this setting. Meaning the default set by
   CMake (if any) or the toolchain is used. On Debian Bookworm, this
   means that PIC is disabled for static libs by default. Some platforms
   (like macOS), has PIC enabled by default.
   autotools supports the double-pass mode only, and in that case
   CMake seems to match PIC behaviour now (as tested on Linux with gcc.)

Follow-up to 5d5dfdb curl#12500

Closes #xxxxx
vszakats added a commit that referenced this pull request Dec 13, 2023
- manual completed: 898b012 #1288
- soname completed: 5de6848 #10023
- bunch of others that are completed
- `NTLM_WB_ENABLED` is implemented in a basic form, and now also
  scheduled for removal, so a TODO at this point isn't useful.

And this 'to-check' item:

Q: "The cmake build selected to run gcc with -fPIC on my box while the
   plain configure script did not."

A: With CMake, since 2ebc74c #11546
   and fc9bfb1 #11627, we explicitly
   enable PIC for libcurl shared lib. Or when building libcurl for
   shared and static lib in a single pass. We do this by default for
   Windows or when enabled by the user via `SHARE_LIB_OBJECT`.
   Otherwise we don't touch this setting. Meaning the default set by
   CMake (if any) or the toolchain is used. On Debian Bookworm, this
   means that PIC is disabled for static libs by default. Some platforms
   (like macOS), has PIC enabled by default.
   autotools supports the double-pass mode only, and in that case
   CMake seems to match PIC behaviour now (as tested on Linux with gcc.)

Follow-up to 5d5dfdb #12500

Reviewed-by: Jay Satiro
Closes #12509
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

None yet

2 participants