Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move most CI builds away from Cirrus #11771

Closed
wants to merge 5 commits into from
Closed

Conversation

dfandrich
Copy link
Contributor

Cirrus is drastically reducing their free tier on Sept. 1, so they will no
longer perform all these builds for us. Most builds are moved other services in
this PR, with some caveats:

  • Windows builds have not yet been replaced
  • macOS ARM builds will be running on x86_64 instead of ARM for a few months

Doing this should preserve the ability to continue to run a FreeBSD build on
Cirrus without disruption. I'm not aware of another CI service offering FreeBSD
builds (for free), so keeping it going is important for test coverage.

Cirrus is reducing their free tier to next to nothing, so we must move
builds elsewhere.
Cirrus' new free tier won't let us have many builds, so drop the
nonessential ones. The FreeBSD 13.X build will still give us the most
relevant FreeBSD coverage.
@github-actions github-actions bot added the CI Continuous Integration label Aug 31, 2023
dfandrich added a commit that referenced this pull request Aug 31, 2023
If we don't do this, all coverage on Cirrus will cease in a few days. By
removing the Windows builds, the FreeBSD one should still continue
as before. The Windows builds will need be moved to another service to
maintain test coverage.

Closes #11771
@dfandrich
Copy link
Contributor Author

The libressl build failure looks legit. There are dyld[NNN]: missing symbol called errors on the failing tests, which points to some kind of link problem. Is it linking against the wrong library at run-time? Is there a system LibreSSL library available on macOS that could be causing this confusion?

@icing
Copy link
Contributor

icing commented Aug 31, 2023

macOS older versions used libressl, yes.

@bagder
Copy link
Member

bagder commented Aug 31, 2023

macOS older versions used libressl, yes.

Don't they still build their curl versions to use libressl?

But if this is a link problem, why does it only happen in a few tests?

Copy link
Member

@bagder bagder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

@vszakats
Copy link
Member

As of 12.x Monterey, macOS ships with LibreSSL and builds curl with it:

curl 7.87.0 (x86_64-apple-darwin21.0) libcurl/7.87.0 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.11 nghttp2/1.45.1
Release-Date: 2022-12-21
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe UnixSockets

AFAIK this continues with Ventura.

@pheiduck
Copy link
Contributor

Yep @vszakats

curl 8.1.2 (x86_64-apple-darwin22.0) libcurl/8.1.2 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.11 nghttp2/1.51.0
Release-Date: 2023-05-30
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe UnixSockets

@dfandrich
Copy link
Contributor Author

But if this is a link problem, why does it only happen in a few tests?

These errors are happening at run time, so I'm guessing those tests are the only ones that enter code paths that need those symbols. This would make sense if macOS is doing lazy symbol binding. The error message doesn't give the missing symbol name so it might not be a LibreSSL one, too; I recall a discussion a couple of months ago about accessing some functions on macOS using some kind of run-time determination.

As the comment in the config file notes, a previous run of this test failed even harder during compilation with --enable-debug on unit1303, so clearly there's some kind of odd linking issue a work here.

Cirrus is drastically reducing their free tier on Sept. 1, so they will
no longer perform all these builds for us.

One important note about this change is that Circle CI is currently
directing all these builds to x86_64 hardware, despite them requesting
ARM. This is because ARM nodes are scheduled to be available on the
free tier only in December. This reduces our architectural diversity
until then but it should automatically come back once those machines are
enabled.
If we don't do this, all coverage on Cirrus will cease in a few days. By
removing the Windows builds, the FreeBSD one should still continue
as before. The Windows builds will need be moved to another service to
maintain test coverage.

Closes #11771
@dfandrich
Copy link
Contributor Author

I wasn't able to reproduce this on a macOS box I have access to; LibreSSL from Brew worked just fine. I've just disabled the problematic tests until someone is able to figure it out.

@dfandrich
Copy link
Contributor Author

Disabling the failing tests on the LibreSSL build worked, but looking at the version output in the banner it appears that the system isn't even using LibreSSL at run-time but rather trying to use OpenSSL (the version banner says OpenSSL/not available instead of LibreSSL/3.7.3). It compiles, links and runs fine when I try it on a macOS box manually. Since it looks like it's not even using LibreSSL, and who knows, possibly not even building with it, I'm going to drop that build altogether since it doesn't seem to be adding anything of value.

@dfandrich dfandrich closed this in 63f23fa Aug 31, 2023
@dfandrich dfandrich deleted the dfandrich/reducecirrus branch August 31, 2023 22:29
@MarcelRaad
Copy link
Member

Looks like moving the macOS builds to CircleCI is unfortunate for forks. On every push to my fork, I'm now spammed with error e-mails from CirceCI because:

Resource class macos for macos.m1.medium.gen1, image xcode:15.0.0 is not available for your project, or is not a valid resource class. This message will often appear if the pricing plan for this project does not support macos use.

Do others also get that or am I doing something wrong?

@MarcelRaad
Copy link
Member

MarcelRaad commented Sep 5, 2023

Hmm, https://discuss.circleci.com/t/macos-intel-support-deprecation-in-january-2024/48718 says that "Xcode 14.2 is the last supported version for Gen1". How does this work here at all? 😕
Looks like Gen2 is used even though the config file explicitly specifies Gen1:
https://app.circleci.com/pipelines/github/curl/curl/6037/workflows/fd44dd35-d306-4c5e-9b31-1661dfc4476b/jobs/39563

@vszakats

This comment was marked as off-topic.

@dfandrich
Copy link
Contributor Author

@vszakats if you're running into local build issues and not CI problems please open a new issue.

@MarcelRaad I haven't been getting spammed by CircleCI, but I also know that there is an issue with the resource class. Their ARM machines aren't available to users in the free tier yet so it's been silently (from my perspective, anyway) replacing that with x86_64 machines. I didn't bother changing that because it seemed harmless and the whole reason for moving builds there was for ARM support, but if it's causing issues I'll change that to reflect reality and hopefully stop the spam.

@vszakats
Copy link
Member

vszakats commented Sep 5, 2023

@dfandrich: I just run into this and felt relevant given that a similar issue was raised here that might be related. I'll link to this post from the commit I fixed this on the build side. It's not my local build but curl-for-win builds (now supporting macOS, Linux) that I'm at the moment testing locally. A little bit of goodwill would be appreciated.

@dfandrich
Copy link
Contributor Author

dfandrich commented Sep 5, 2023 via email

dfandrich added a commit that referenced this pull request Sep 6, 2023
The ARM machines aren't ready for us and requesting them now causes
warnings e-mails to be sent to some PR pushers.

Ref: #11771
dfandrich added a commit that referenced this pull request Sep 6, 2023
The ARM machines aren't ready for us and requesting them now causes
warnings e-mails to be sent to some PR pushers.

Ref: #11771
dfandrich added a commit that referenced this pull request Sep 6, 2023
The ARM machines aren't ready for us and requesting them now causes
warnings e-mails to be sent to some PR pushers.

Ref: #11771
@dfandrich
Copy link
Contributor Author

I've switched the Circle macOS builds to an x86 class where those things should not be issues. Please say something if you still get those warning e-mails.

@MarcelRaad
Copy link
Member

I can confirm that the builds on my fork run again, thanks Dan!

ptitSeb pushed a commit to wasix-org/curl that referenced this pull request Sep 25, 2023
If we don't do this, all coverage on Cirrus will cease in a few days. By
removing the Windows builds, the FreeBSD one should still continue
as before. The Windows builds will need be moved to another service to
maintain test coverage.

Closes curl#11771
ptitSeb pushed a commit to wasix-org/curl that referenced this pull request Sep 25, 2023
The ARM machines aren't ready for us and requesting them now causes
warnings e-mails to be sent to some PR pushers.

Ref: curl#11771
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI Continuous Integration
Development

Successfully merging this pull request may close these issues.

None yet

6 participants