Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make in /docs fails sometimes for 8.6.0 #12829

Closed
jonrumsey opened this issue Jan 31, 2024 · 6 comments
Closed

make in /docs fails sometimes for 8.6.0 #12829

jonrumsey opened this issue Jan 31, 2024 · 6 comments
Labels

Comments

@jonrumsey
Copy link
Contributor

I did this

Builds sometimes failing on Linux x/p/z and AIX where it was always clean on 8.5.0

make[4]: Entering directory '/build/curl/64/docs/cmdline-opts'
  GENERATE ../../docs/curl.1
mv: cannot stat './manpage.tmp': No such file or directory
make[4]: *** [Makefile:909: ../../docs/curl.1] Error 1

As this is intermittent, is there a problem with the makefile dependencies for this rule being incomplete?

$(MANPAGE): $(DPAGES) $(SUPPORT) mainpage.idx Makefile.inc gen.pl
	$(GEN)(rm -f $(MANPAGE) && cd $(srcdir) && @PERL@ ./gen.pl mainpage $(DPAGES) > $(builddir)/manpage.tmp && mv $(builddir)/manpage.tmp $(MANPAGE))

I expected the following

Clean build

curl/libcurl version

curl 8.6.0

operating system

Ubuntu 22.04.3 LTS & Others

@bagder bagder added the build label Jan 31, 2024
@bagder
Copy link
Member

bagder commented Jan 31, 2024

That makefile logic was introduced in 17153e1, almost a year ago. It is odd that it causes problems now!

If you run make V=1 in the docs/cmdline-opts directory, does it help to reveal anything about the failure that must have happened that caused manpage.tmp not to be created (but yet not cause the command to return error) ?

@jonrumsey
Copy link
Contributor Author

Thanks for the suggestion - with make V=1 I get the following;

Making all in ../docs make[2]: Entering directory '/curl/32/docs' Making all in . make[3]: Entering directory '/curl/32/docs' if test "..x" != "/curl/src/curl-8.6.0x" -a -e "/curl/src/curl-8.6.0/docs/curl.1"; then \ /usr/bin/install -c -m 644 "/curl/src/curl-8.6.0/docs/curl.1" /curl/32/docs/curl.1 \ && touch -r "/curl/src/curl-8.6.0/docs/curl.1" /curl/32/docs/curl.1; fi /curl/src/curl-8.6.0/scripts/cd2nroff /curl/src/curl-8.6.0/docs/mk-ca-bundle.md >mk-ca-bundle.1 cd cmdline-opts && /tools/gnumake/4.4.1/bin/make make[4]: Entering directory '/curl/32/docs/cmdline-opts' (rm -f ../../docs/curl.1 && cd /curl/src/curl-8.6.0/docs/cmdline-opts && /usr/bin/perl ./gen.pl mainpage abstract-unix-socket.md alt-svc.md anyauth.md append.md aws-sigv4.md basic.md ca-native.md cacert.md capath.md cert-status.md cert-type.md cert.md ciphers.md compressed-ssh.md compressed.md config.md connect-timeout.md connect-to.md continue-at.md cookie-jar.md cookie.md create-dirs.md create-file-mode.md crlf.md crlfile.md curves.md data-ascii.md data-binary.md data-raw.md data-urlencode.md data.md delegation.md digest.md disable-eprt.md disable-epsv.md disable.md disallow-username-in-url.md dns-interface.md dns-ipv4-addr.md dns-ipv6-addr.md dns-servers.md doh-cert-status.md doh-insecure.md doh-url.md dump-header.md egd-file.md engine.md etag-compare.md etag-save.md expect100-timeout.md fail-early.md fail-with-body.md fail.md false-start.md form-escape.md form-string.md form.md ftp-account.md ftp-alternative-to-user.md ftp-create-dirs.md ftp-method.md ftp-pasv.md ftp-port.md ftp-pret.md ftp-skip-pasv-ip.md ftp-ssl-ccc-mode.md ftp-ssl-ccc.md ftp-ssl-control.md get.md globoff.md happy-eyeballs-timeout-ms.md haproxy-protocol.md haproxy-clientip.md head.md header.md help.md hostpubmd5.md hostpubsha256.md hsts.md http0.9.md http1.0.md http1.1.md http2-prior-knowledge.md http2.md http3.md http3-only.md ignore-content-length.md include.md insecure.md interface.md ipfs-gateway.md ipv4.md ipv6.md json.md junk-session-cookies.md keepalive-time.md key-type.md key.md krb.md libcurl.md limit-rate.md list-only.md local-port.md location-trusted.md location.md login-options.md mail-auth.md mail-from.md mail-rcpt-allowfails.md mail-rcpt.md manual.md max-filesize.md max-redirs.md max-time.md metalink.md negotiate.md netrc-file.md netrc-optional.md netrc.md next.md no-alpn.md no-buffer.md no-clobber.md no-keepalive.md no-npn.md no-progress-meter.md no-sessionid.md noproxy.md ntlm-wb.md ntlm.md oauth2-bearer.md output-dir.md output.md parallel-immediate.md parallel-max.md parallel.md pass.md path-as-is.md pinnedpubkey.md post301.md post302.md post303.md preproxy.md progress-bar.md proto-default.md proto-redir.md proto.md proxy-anyauth.md proxy-basic.md proxy-ca-native.md proxy-cacert.md proxy-capath.md proxy-cert-type.md proxy-cert.md proxy-ciphers.md proxy-crlfile.md proxy-digest.md proxy-header.md proxy-http2.md proxy-insecure.md proxy-key-type.md proxy-key.md proxy-negotiate.md proxy-ntlm.md proxy-pass.md proxy-pinnedpubkey.md proxy-service-name.md proxy-ssl-allow-beast.md proxy-ssl-auto-client-cert.md proxy-tls13-ciphers.md proxy-tlsauthtype.md proxy-tlspassword.md proxy-tlsuser.md proxy-tlsv1.md proxy-user.md proxy.md proxy1.0.md proxytunnel.md pubkey.md quote.md random-file.md range.md rate.md raw.md referer.md remote-header-name.md remote-name-all.md remote-name.md remote-time.md remove-on-error.md request-target.md request.md resolve.md retry-all-errors.md retry-connrefused.md retry-delay.md retry-max-time.md retry.md sasl-authzid.md sasl-ir.md service-name.md show-error.md silent.md socks4.md socks4a.md socks5-basic.md socks5-gssapi-nec.md socks5-gssapi-service.md socks5-gssapi.md socks5-hostname.md socks5.md speed-limit.md speed-time.md ssl-allow-beast.md ssl-auto-client-cert.md ssl-no-revoke.md ssl-reqd.md ssl-revoke-best-effort.md ssl.md sslv2.md sslv3.md stderr.md styled-output.md suppress-connect-headers.md tcp-fastopen.md tcp-nodelay.md telnet-option.md tftp-blksize.md tftp-no-options.md time-cond.md tls-max.md tls13-ciphers.md tlsauthtype.md tlspassword.md tlsuser.md tlsv1.0.md tlsv1.1.md tlsv1.2.md tlsv1.3.md tlsv1.md tr-encoding.md trace-ascii.md trace-config.md trace-ids.md trace-time.md trace.md unix-socket.md upload-file.md url.md url-query.md use-ascii.md user-agent.md user.md variable.md verbose.md version.md write-out.md xattr.md > ./manpage.tmp && mv ./manpage.tmp ../../docs/curl.1) mv: cannot stat './manpage.tmp': No such file or directory make[4]: *** [Makefile:909: ../../docs/curl.1] Error 1 make[4]: Leaving directory '/curl/32/docs/cmdline-opts' make[3]: *** [Makefile:937: /curl/32/docs/curl.1] Error 2 make[3]: Leaving directory '/curl/32/docs' make[2]: *** [Makefile:654: all-recursive] Error 1 make[2]: Leaving directory '/curl/32/docs' make[1]: *** [Makefile:1553: all-recursive] Error 1 make[1]: Leaving directory '/curl/32/src' make: *** [Makefile:1262: all-recursive] Error 1

I build both 32 and 64-bit versions of curl in parallel from a single source tree - so I think that my issue is that this common source tree /curl/src is being changed (the perl script runs in the /curl/src/curl-8.6.0/docs/cmdline-opts directory) and there is a race between the 32 and 64-bit builds to create/move the temp file in the common source tree, rather than working in their output trees of /curl/32 and /curl/64 respectively. For some reason with 8.6.0 that race seems to be more competitive!

I will change my build process so that the two parallel builds have their own modifiable source trees to confirm that resolves the issue.

@bagder
Copy link
Member

bagder commented Feb 1, 2024

We could make it use a unique temp file to reduce this risk?

diff --git a/docs/cmdline-opts/Makefile.am b/docs/cmdline-opts/Makefile.am
index e9b35ac05..08d19a015 100644
--- a/docs/cmdline-opts/Makefile.am
+++ b/docs/cmdline-opts/Makefile.am
@@ -36,9 +36,9 @@ GN_1 =
 GN_ = $(GN_0)
 
 all: $(MANPAGE)
 
 $(MANPAGE): $(DPAGES) $(SUPPORT) mainpage.idx Makefile.inc gen.pl
-       $(GEN)(rm -f $(MANPAGE) && cd $(srcdir) && @PERL@ ./gen.pl mainpage $(DPAGES) > $(builddir)/manpage.tmp && mv $(builddir)/manpage.tmp $(MANPAGE))
+       $(GEN)(rm -f $(MANPAGE) && cd $(srcdir) && @PERL@ ./gen.pl mainpage $(DPAGES) > $(builddir)/manpage.tmp.$$$$ && mv $(builddir)/manpage.tmp.$$$$ $(MANPAGE))
 
 listhelp:
        ./gen.pl listhelp $(DPAGES) > $(top_builddir)/src/tool_listhelp.c

bagder added a commit that referenced this issue Feb 1, 2024
By appending the pid number two different runs at the same time will not
trample over the same file.

Reported-by: Jon Rumsey
Fixes #12829
@jonrumsey
Copy link
Contributor Author

Thanks - I've confirmed that giving each of the builds their own source tree also resolves the issue and yes appending a pid to the filename for this make rule would also prevent any interactions on this temp file.

I'm note sure how $(builddir) got expanded to '.' but because the current directory is in the source tree that means the build will be writing (albeit temporarily) to the source tree rather than under the --prefix provided on the configure.

@bagder
Copy link
Member

bagder commented Feb 1, 2024

I'm note sure how $(builddir) got expanded to '.' but because the current directory is in the source tree that means the build will be writing (albeit temporarily) to the source tree

That variable will expand to a separate directory if you build outside of the source tree. Like this:

$ mkdir this/somewhere
$ cd this/somewhere
$ ../../curl-8.6.0/configure [options]
$ make -sj

@jonrumsey
Copy link
Contributor Author

Thanks

bagder added a commit that referenced this issue Feb 1, 2024
By appending the pid number two different runs at the same time will not
trample over the same file.

Reported-by: Jon Rumsey
Fixes #12829
Closes #12839
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

Successfully merging a pull request may close this issue.

2 participants