That doesn't fix it. You're still setting DYLD_LIBRARY_PATH to /opt/local/lib (CURL_LIBRARY_PATH) which you must not do.

What problem was ba0657c trying to solve? It mentions a "non-standard installation of openssl" but does not elaborate what that means. Could it perhaps be a broken installation of openssl that curl needn't concern itself with supporting? Any installed library that requires you to set DYLD_LIBRARY_PATH to use it is broken.

When you tell configure to look for a library at a custom location and it finds it to be there - it uses the linker to verify that the library is there and works to link with.

Later in the configure script, it will run an executable for some tests. When configure runs an executable the previously detected libraries need to be found at run-time as opposed to build-time. That's what it sets the variable for.

Why does it find the library in the non-standard directory /opt/local/lib and yet can't set the variable to search for it there? (Edit: The answer is of course that it then wrongly also finds other libs there that you have there but don't want to use.)

Any installed library that requires you to set DYLD_LIBRARY_PATH to use it is broken.

If I install a build of a library in ~/funky, how is using DYLD_LIBRARY_PATH wrong for that?

Your problem is probably that you're using one library in that directory but then you also have other libs in that directory that are using the same names as ones you have in another path, and you only want that one lib used from that directory.

I might be naive but that sounds like a recipe just waiting for problems like this. I don't have a short term fix for a setup like that, other than perhaps avoid having configure detect that directory specifically.

Your problem is

My problem is that the curl build system is setting DYLD_LIBRARY_PATH=/opt/local/lib. It must not do so. It causes the problem I reported.

you also have other libs in that directory

Yes, MacPorts installs libs primarily in /opt/local/lib, just as most other package managers I expect primarily install their libs in their own prefix or like a user might install several libraries in /usr/local. MacPorts has done so since 2002. It is not a problem.

a recipe just waiting for problems like this

It is the decision of the developers of libiconv that a "third party" build (such as the one in MacPorts) shall have symbols beginning with _libiconv so as not to conflict with a "vendor" iconv implementation (which may or may not be GNU libiconv) that has symbols beginning with _iconv. The difference is transparent to programs that use the iconv.h header correctly. The problem only appears here because you are subverting normal library loading by setting DYLD_LIBRARY_PATH=/opt/local/lib. Don't do that.

Later in the configure script, it will run an executable for some tests. When configure runs an executable the previously detected libraries need to be found at run-time as opposed to build-time. That's what it sets the variable for.

I do not understand why setting DYLD_LIBRARY_PATH=/opt/local/lib would be thought to be necessary to achieve that. macOS dynamic libraries do not work like Linux shared libraries. When a macOS executable is linked with a dynamic library, the library's install_name is copied into the executable. For most libraries, such as those installed by the OS or MacPorts, the install_name is most usually an absolute path, so if you link an executable with such a library, it knows where to find the library by its absolute path, no DYLD_LIBRARY_PATH needed at build time or runtime. Setting DYLD_LIBRARY_PATH at runtime tells the OS to ignore the library paths that were baked into the executable, which is not something you usually want to do, unless as I said you're trying to run an executable you just built that is linked with a library you just built whose install_name references its ultimate install location but it has not yet been installed there. I am assuming that is not the case here.

Again, I'd like to understand what specific issue the commit that introduced the problem was meant to correct.

If I install a build of a library in ~/funky, how is using DYLD_LIBRARY_PATH wrong for that?

The install_name of the library should be set to its absolute install path. Then setting DYLD_LIBRARY_PATH is not necessary. Or if you want the library and the executable linking with it to be easily relocatable there are solutions involving @executable_path, @loader_path, or @rpath which are outside the scope of the normal building of a library like curl.

I do not understand why

#8028 is why

I saw #8028 but it does not explain what is meant by "non-standard installation of openssl".

I presume that means you build openssl with something like:

./config --prefix=/home/daniel/build-openssl

I can't see how setting DYLD_LIBRARY_PATH would be necessary to support such an installation. Is there a reproduction recipe for the problem that the commit was trying to solve?

I failed to configure 7.81.0 on macOS, with libjpeg in my case.
I'm not using MacPorts or other package managers but I have libjpeg-turbo installed under /usr/local. Had no problem with 7.80.0.

I remembered that DYLD_LIBRARY_PATH has been a source of problem from the old days. You will find examples caused by this on Google.
I may be wrong, but I personally understand that this variable is for debugging or testing purposes and should usually be avoided.

Does it help if we change it to use DYLD_FALLBACK_LIBRARY_PATH instead?

Sure, that would avoid the problem, but it's still unclear why setting any variable would be needed.

Setting DYLD_LIBRARY_PATH is indeed something that should not be done on macOS [*]. DYLD_FALLBACK_LIBRARY_PATH is a bit better, but is still a sign that something elsewhere is broken. My guess is that #8028 is actually caused by a misconfigured OpenSSL, and having to set DYLD_* variables to get around it is the wrong fix for this.

[*] Well, fine, you can use it for debugging purposes if you want to highjack the libraries an executable intends to link against. The use of the term "highjack" here is intentional -- there is a reason why binaries that live in SIPed directories (e.g. /bin, /usr/bin) clear DYLD_LIBRARY_PATH (and DYLD_FALLBACK_LIBRARY_PATH) from the environment before executing as a security measure.

I dusted off my mac and did some tests and no, I cannot understand why the original #8028 was necessary. Have a look at #8247 for my proposed way forward.

@bwncp @mspncp do you guys have anything to add

@jay Thanks for mentioning me. Sorry for breaking your build. I will verify why the install_name was not set in my case.

Ok, was able to reproduce the old problem now. We call configure as

./configure --prefix=/Users/bwalle/devel/enterprise-build/3p/macos/x86_64/install/curl --with-ssl=/Users/bwalle/devel/enterprise-build/3p/macos/x86_64/install/openssl --with-zlib=/Users/bwalle/devel/enterprise-build/3p/macos/x86_64/install/zlib --enable-ipv6 --enable-smtp --disable-ftp --disable-file --disable-ldap --disable-ldaps --disable-rtsp --disable-dict --disable-telnet --disable-tftp --disable-pop3 --disable-imap --disable-gopher --disable-manual --without-libidn2 --without-ca-path --without-ca-bundle --without-libssh2 --without-librtmp --without-libpsl --without-brotli --disable-static --enable-shared

The compilation of conftest can be shortened as

clang -o conftest -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk -I/Users/bwalle/devel/enterprise-build/3p/macos/x86_64/install/openssl/include -L/Users/bwalle/devel/enterprise-build/3p/macos/x86_64/install/openssl/lib conftest.c -lssl

which results in

dyld[82303]: Library not loaded: @rpath/libssl.1.0.0.dylib
  Referenced from: /Users/bwalle/devel/enterprise/conftest
  Reason: tried: '/usr/local/lib/libssl.1.0.0.dylib' (no such file), '/usr/lib/libssl.1.0.0.dylib' (no such file)
[1]    82303 abort      ./conftest

The contents of the openssl installation directory is

drwxr-xr-x  14 bwalle  staff       448 10 Jan 16:48 engines
-r-xr-xr-x   1 bwalle  staff   3731216 10 Jan 16:48 libcrypto.1.0.0.dylib
-rw-r--r--   1 bwalle  staff  10029960 10 Jan 16:48 libcrypto.a
lrwxr-xr-x   1 bwalle  staff        21 10 Jan 16:48 libcrypto.dylib -> libcrypto.1.0.0.dylib
-r-xr-xr-x   1 bwalle  staff    650160 10 Jan 16:48 libssl.1.0.0.dylib
-rw-r--r--   1 bwalle  staff   2445528 10 Jan 16:48 libssl.a
lrwxr-xr-x   1 bwalle  staff        18 10 Jan 16:48 libssl.dylib -> libssl.1.0.0.dylib
drwxr-xr-x   5 bwalle  staff       160 10 Jan 16:48 pkgconfig

What are we doing wrong here? Of course I can set DYLD_FALLBACK_LIBRARY_PATH in the build scripts, but what's the clean solution here @ryandesign?

Ok, so your build of openssl was created using @rpath. @rpath is a newer feature in macOS, introduced in Mac OS X 10.5. Both because it was not available in earlier macOS releases and because of the way that it tends to cause failures like the one you experienced, I don't like @rpath and advocate wherever I can for not using it in MacPorts.

If you want to use your @rpath build of openssl, then every time you want to link something with that openssl library you'll have to tell the linker what @rpath should expand to, by passing -rpath /wherever/the/library/is in LDFLAGS. (I think this is right but I have little experience actually using @rpath-based libraries myself.)

Setting rpath globally to a single value doesn't make sense in that build system. Each library has its own installation path (well, didn't design that build system but I can't rewrite anything now just to build curl).

But indeed, the openssl build system is patched by us:

SHAREDFLAGS="$$SHAREDFLAGS -install_name @rpath/$$SHLIB$(SHLIB_EXT)"

That explains why the problem happens only here. Unfortunately, setting DYLD_FALLBACK_LIBRARY_PATH from outside is not possible because of System Integrity Protection (see this discussion on StackOverflow). 🤔

I'm told you can set multiple rpaths, e.g. -rpath /some/path -rpath /another/path. Each path will be recorded into the executable as LD_RUNPATH_SEARCH_PATH. The program will look for @rpath libraries in those paths at runtime. The paths you specify with the -rpath flag could also contain @executable_path or @loader_path if you're trying to make something that's relocatable.

DYLD_* environment variables work fine with your own executables with System Integrity Protection enabled, they're just not passed down to subprocesses like they were without SIP, and they're cleared if you're using an executable that Apple provides with the OS.

Passing -rpath is a much better choice here. You can indeed pass multiple -rpath flags. This is technically a flag consumed by ld, so you may wish to pass it as -Wl,-rpath,/path/to/dir/containing/libssl, but clang is smart enough these days to just forward this to ld.

To use @loader_path, you'll probably need to pass something like

-Wl,-rpath,@loader_path/relative_path_to_openssl_dir_from_curl_install_dir,-rpath,absolute_path_to_openssl_dir

The latter -rpath is needed for conftest to find libssl during configure (presumably your build directory is different from where you'll eventually install curl). You can use install_name_tool to delete the extra rpath when you're ready to install curl. (Or you can leave it alone, but your security person might not like that.)

Thanks for all that useful information. I'm not a Mac expert, my home is Linux 😀

DYLD_* environment variables work fine with your own executables with System Integrity Protection enabled, they're just not passed down to subprocesses like they were without SIP, and they're cleared if you're using an executable that Apple provides with the OS.

That's true, but the shell executing the configure script is from Apple and cleans DYLD_* variables.

However, adding -rpath /Users/bwalle/devel/enterprise-build/3p/macos/x86_64/install/openssl/lib to the LDFLAGS (and only that directory, not multiple -rpath options, makes configure to succeed, but later in the build process I get

libtool: warning: ignoring multiple '-rpath's for a libtool library
.libs/libcurl_la-amigaos.o: no symbols
...
clang: error: no such file or directory: '/Users/bwalle/devel/enterprise-build/3p/macos/x86_64/install/curl/lib/libcurl.4.dylib'

Any ideas?

libtool: warning: ignoring multiple '-rpath's for a libtool library

Googling this error, I found the suggestion to use -Wl,-rpath,/the/path instead of -rpath /the/path, since the latter will be interpreted by libtool, which you don't want, whereas the former will only go to ld.

the shell executing the configure script is from Apple and cleans DYLD_* variables

Right, there shouldn't be any reason why anyone would need to set a DYLD_* variable and then call a configure script. But if the author of a configure script needed to use a DYLD_* variable somewhere within the configure script when running some just-compiled executable that linked against a just-compiled library, that would be fine.

libtool: warning: ignoring multiple '-rpath's for a libtool library

Googling this error, I found the suggestion to use -Wl,-rpath,/the/path instead of -rpath /the/path, since the latter will be interpreted by libtool, which you don't want, whereas the former will only go to ld.

This works indeed. I think that's the best solution for me.

Thanks for all the hints and explanations.

Thanks for your extensive help here.

@ryandesign @bwncp I presume you are then fine with #8247 ?

@ryandesign @bwncp I presume you are then fine with #8247 ?

Yes, you can close the issue from my point of view, reverting my commit is ok.

@ryandesign @bwncp I presume you are then fine with #8247 ?

Yes that works for me, thanks!