New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
dist: set -eu
, fix shellcheck, make reproducible and smaller tarballs
#13299
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
Haven't found yet where the .tar file is created — depending on how its done |
The tarball is made by the |
That was my fear :) I'll have to look... After looking: tardir=curl-- && ${TAR-tar} chof - "$tardir" | [...]
As for how to tell autotools to do the right thing, I have not explored yet. I assume it's impossible or at least painful. edit:
Ref: https://mgorny.pl/articles/portability-of-tar-features.html |
set -eu
, fix shellcheck warnings, smaller & reproducible .zip
set -eu
, shellcheck fixes, reproducibility, smaller .zip
I think we can settle with shipping the content reproducible as start, and not the tarball files themselves. |
If we want the tarball images themselves reproducible I figure that's a rather easy thing to add. |
FWIW curl-for-win uses this: TZ=UTC tar --create \
--format=ustar \
--owner=0 --group=0 --numeric-owner \
--files-from "${_FLS}" where Also needs to install and remap I'd vote to make the tar images reproducible, if we're tackling this part anyway. (in a separate PR though) By easy, which method would you go? |
set -eu
, shellcheck fixes, reproducibility, smaller .zip
set -eu
, fix shellcheck, reproducibility, shrink .zip
in maketgz
I figure untaring and retaring with the proper options should work fine. Then we can let |
Maybe like this? diff --git a/maketgz b/maketgz
index 602f1071b..089d1bfb8 100755
--- a/maketgz
+++ b/maketgz
@@ -174,10 +174,28 @@ res=$?
if test "$res" != 0; then
echo "make dist failed"
exit 2
fi
+retar() {
+ tempdir=$1
+ rm -rf $tempdir
+ mkdir $tempdir
+ cd $tempdir
+ gzip -dc ../$targz | tar -xf -
+ find curl-* | sort > files
+ tar --create --format=ustar --owner=0 --group=0 --numeric-owner --files-from files -zf out.tar.gz
+ rm files
+ mv out.tar.gz ../
+ cd ..
+ rm -rf $tempdir
+}
+
+retar ".tarbuild"
+echo "replace $targz with out.tar.gz"
+mv out.tar.gz "$targz"
+
############################################################################
#
# Now make a bz2 archive from the tar.gz original
#
|
Oh well, |
This reverts commit c194e8e.
- set file timestamps in tarball/zip reproducible. - use POSIX tar format (ustar) to avoid supply chain vulnerability: https://seclists.org/oss-sec/2021/q4/0 - make file order in tarball reproducible. - make uid/gid in tarball reproducible. - omit owner user/group names from tarball for reproducibility and privacy. - set file timestamps of tarballs/zip reproducible. - omit current timestamp from .gz header for reproducibility. .tar.gz also became smaller in the process: 4462311 -> 4148249 bytes Requires GNU tar, GNU date.
set -eu
, fix shellcheck, reproducibility, shrink .zip
in maketgz
set -eu
, fix shellcheck, make source tarballs reproducible and smaller in maketgz
Source tarballs and zip should be fully reproducible now. (Assuming
Testing on macOS needs this snippet (plus manually nugding autotools with some command, then copying a missing autotools file into the build directory): case "$(uname)" in
Darwin*)
date() { gdate "$@"; }
tar() { gtar "$@"; }
;;
esac |
set -eu
, fix shellcheck, make source tarballs reproducible and smaller in maketgz
set -eu
, fix shellcheck, make source tarballs reproducible and smaller
This reverts commit 6898b30.
set -eu
, fix shellcheck, make source tarballs reproducible and smallerset -eu
, fix shellcheck, make tarballs reproducible and smaller
set -eu
, fix shellcheck, make tarballs reproducible and smallerset -eu
, fix shellcheck, reproducible and smaller tarballs
set -eu
, fix shellcheck, reproducible and smaller tarballsset -eu
, fix shellcheck, make reproducible and smaller tarballs
Follow-up to 860cd5f curl#13299 Closes #xxxxx
In the initial implementation of reproducible tarballs, they were missing directory entries, while .zip archives had them. It meant that on extracting the tarball, on-disk directory entries got the current timestamp. This patch fixes this by including directory entries in the tarball, with reproducible timestamps. It also moves sorting inside tar, to ensure reproducible directory entry timestamps on extract (without the need of `--delay-directory-restore` option, when extracting with GNU tar. BSD tar got that right by default.) GNU tar 1.28 (2014-07-28) introduced `--sort=`. Ref: #13299 (comment) Follow-up to 860cd5f #13299 Closes #13322
-eu
and fix fallouts.SOURCE_DATE_EPOCH
for reproducibility.Authored-by: Daniel J. H.
Ref: dist: add SOURCE_DATE_EPOCH env var to maketgz script #13280
TZ=UTC
andLC_ALL=C
for reproducibility.ustar
tarball format to avoid supply chain vulnerability:https://seclists.org/oss-sec/2021/q4/0
.tar.gz
also became smaller in the process: 4,462,311 -> 4,148,249 bytes (8.7.1)Requires GNU tar, GNU date,
sha256sum
.Ref: #13250
Closes #13299