Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Curl fails to build reproducibly #13242

Closed
charles2910 opened this issue Mar 30, 2024 · 7 comments
Closed

Curl fails to build reproducibly #13242

charles2910 opened this issue Mar 30, 2024 · 7 comments
Labels
script

Comments

@charles2910
Copy link
Contributor

I did this

We have in debian a CI pipeline running for each commit in our repo (hosted in Debian's gitlab instance). We do have reprotest which builds the package twice changing the build environment and checks if they are bit by bit the same. Though they aren't the same because of the autogenerated manpages:

 charles  …  debian  output  reprotest  2  SIGINT  diffoscope control/source-root/curl_8.7.1-1+salsaci+20240330+129_amd64.deb experiment-1/source-root/curl_8.7.1-1+salsaci+20240330+129_amd64.deb
--- control/source-root/curl_8.7.1-1+salsaci+20240330+129_amd64.deb
+++ experiment-1/source-root/curl_8.7.1-1+salsaci+20240330+129_amd64.deb
├── file list
│ @@ -1,3 +1,3 @@
│  -rw-r--r--   0        0        0        4 2024-03-27 19:02:14.000000 debian-binary
│  -rw-r--r--   0        0        0     1028 2024-03-27 19:02:14.000000 control.tar.xz
│ --rw-r--r--   0        0        0   344140 2024-03-27 19:02:14.000000 data.tar.xz
│ +-rw-r--r--   0        0        0   344172 2024-03-27 19:02:14.000000 data.tar.xz
├── control.tar.xz
│ ├── control.tar
│ │ ├── ./md5sums
│ │ │ ├── ./md5sums
│ │ │ │┄ Files differ
├── data.tar.xz
│ ├── data.tar
│ │ ├── file list
│ │ │ @@ -7,11 +7,11 @@
│ │ │  drwxr-xr-x   0 root         (0) root         (0)        0 2024-03-27 19:02:14.000000 ./usr/share/doc/curl/
│ │ │  -rw-r--r--   0 root         (0) root         (0)      573 2024-03-27 19:02:14.000000 ./usr/share/doc/curl/README.Debian
│ │ │  -rw-r--r--   0 root         (0) root         (0)     8731 2024-03-27 19:02:14.000000 ./usr/share/doc/curl/changelog.Debian.gz
│ │ │  -rw-r--r--   0 root         (0) root         (0)   113339 2024-03-27 08:03:42.000000 ./usr/share/doc/curl/changelog.gz
│ │ │  -rw-r--r--   0 root         (0) root         (0)    21639 2024-03-27 19:02:14.000000 ./usr/share/doc/curl/copyright
│ │ │  drwxr-xr-x   0 root         (0) root         (0)        0 2024-03-27 19:02:14.000000 ./usr/share/man/
│ │ │  drwxr-xr-x   0 root         (0) root         (0)        0 2024-03-27 19:02:14.000000 ./usr/share/man/man1/
│ │ │ --rw-r--r--   0 root         (0) root         (0)    63973 2024-03-27 19:02:14.000000 ./usr/share/man/man1/curl.1.gz
│ │ │ +-rw-r--r--   0 root         (0) root         (0)    63982 2024-03-27 19:02:14.000000 ./usr/share/man/man1/curl.1.gz
│ │ │  drwxr-xr-x   0 root         (0) root         (0)        0 2024-03-27 19:02:14.000000 ./usr/share/zsh/
│ │ │  drwxr-xr-x   0 root         (0) root         (0)        0 2024-03-27 19:02:14.000000 ./usr/share/zsh/vendor-completions/
│ │ │  -rw-r--r--   0 root         (0) root         (0)    15680 2024-03-27 19:02:14.000000 ./usr/share/zsh/vendor-completions/_curl
│ │ ├── ./usr/share/man/man1/curl.1.gz
│ │ │ ├── curl.1
│ │ │ │ @@ -20,15 +20,15 @@
│ │ │ │  .\" *
│ │ │ │  .\" * SPDX-License-Identifier: curl
│ │ │ │  .\" *
│ │ │ │  .\" **************************************************************************
│ │ │ │  .\"
│ │ │ │  .\" DO NOT EDIT. Generated by the curl project managen man page generator.
│ │ │ │  .\"
│ │ │ │ -.TH curl 1 "March 27 2024" "curl 8.7.1" "curl Manual"
│ │ │ │ +.TH curl 1 "märts 28 2024" "curl 8.7.1" "curl Manual"
│ │ │ │  .SH NAME
│ │ │ │  curl \- transfer a URL
│ │ │ │  .SH SYNOPSIS
│ │ │ │  \fBcurl [options / URLs]\fP
│ │ │ │  .SH DESCRIPTION
│ │ │ │  \fBcurl\fP is a tool for transferring data from or to a server using URLs. It
│ │ │ │  supports these protocols: DICT, FILE, FTP, FTPS, GOPHER, GOPHERS, HTTP, HTTPS,
│ │ │ │ ├── encoding
│ │ │ │ │ @@ -1 +1 @@
│ │ │ │ │ -us-ascii
│ │ │ │ │ +utf-8

The cd2nroff script is correctly using SOURCE_DATE_EPOCH, but it's failing for 2 reasons, timezone variation and language variation:

INFO:reprotest.build:TIMEZONE variation: TZ=GMT+12

Makes the day change from 27 to 28.

INFO:reprotest.build:LOCALE variation: LC_ALL = et_EE.UTF-8 LANGUAGE = et_EE.UTF-8:fr

Is making the month different (english vs estonian).

I expected the following

To pass reprotest test and be reproducible.

curl/libcurl version

curl 8.7.1 (x86_64-pc-linux-gnu) libcurl/8.7.1 OpenSSL/3.1.5 zlib/1.2.13 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.4 libpsl/0.21.2 libssh2/1.11.0 nghttp2/1.60.0 librtmp/2.3 OpenLDAP/2.5.16
Release-Date: 2024-03-27, security patched: 8.7.1-2
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

operating system

Debian unstable
@bagder bagder added the script label Mar 30, 2024
@bagder
Copy link
Member

bagder commented Mar 30, 2024 •
edited

I suppose we should switch to gmtime when SOURCE_DATE_EPOCH is set.

Why does it need to build the same for different TZ settings? If you want to reproduce shouldn't you stick to the same TZ?

@bagder
Copy link
Member

bagder commented Mar 31, 2024

Sorry, I meant locale. Why does it need to build the same output when you change locale?

bagder added a commit that referenced this issue Mar 31, 2024
@bagder
cd2nroff: use UTC timezone when SOURCE_DATE_EPOCH is set
61e7ff4 
Makes it independent of the TZ setting.

Reported-by: Carlos Henrique Lima Melara
Ref: #13242
@bagder bagder mentioned this issue Mar 31, 2024
Closed
@charles2910
Copy link
Contributor Author

Sorry, I meant locale. Why does it need to build the same output when you change locale?

I think it's a goal set by the reproducible builds efforts. By their description:

First, the build system needs to be made entirely deterministic: transforming a given source must always create the same result. For example, the current date and time must not be recorded and output always has to be written in the same order.

Second, the set of tools used to perform the build and more generally the build environment should either be recorded or pre-defined.

Third, users should be given a way to recreate a close enough build environment, perform the build process, and validate that the output matches the original build.

The rationale would be to allow anyone to check if the provided binary matches a "build from my machine".

@charles2910
Copy link
Contributor Author

I guess an easy way to solve would be to use the ISO date as yyyy-mm-dd instead of relying on the %B which depends on the locale

@bagder
Copy link
Member

bagder commented Mar 31, 2024

Yeah, good thinking. I made this change as well in #13243

@bagder
Copy link
Member

bagder commented Mar 31, 2024

@charles2910 does #13243 fix the problems for you?

@bagder bagder closed this as completed in afdd112 Mar 31, 2024
@charles2910
Copy link
Contributor Author

Yes, it did fix the pipeline. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
script
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bagder @charles2910