From the current CI test results:

1. I suppose new tests should be disabled for rustls ?
2. I suspect we have a race problem when starting the protocol server just before stunnel. runtests.pl should probably make sure the server is running before starting stunnel. For now, move imaps/pop3s/smtps tests after first corresponding clear protocol tests.

I suppose new tests should be disabled for rustls ?

Why? rustls is supposed to generally work fine.

I suspect we have a race problem

Hm, yes it is supposed to first verify that the non-stunnel server works before it starts stunnel. See startservers for FTPS for example, it first runs runpingpongserver (which also verifies that it works) before it runs runftpsserver that starts stunnel.

Why? rustls is supposed to generally work fine.

It has 12 disabled tests that are supposed to work with. That's why I asked.

Hm, yes it is supposed to first verify that the non-stunnel server works before it starts stunnel. See startservers for FTPS for example, it first runs runpingpongserver (which also verifies that it works) before it runs runftpsserver that starts stunnel.

I've reused the ftps code for the mail protocols (I even renamed runftpsserver as runsecureserver) thus I expect new protos behave the same. Investigating.

It has 12 disabled tests that are supposed to work with. That's why I asked.

Right, but we use TLS in many more tests than just those. These are just tests that for some reason fail with rustls. It is still expected to work just like the other backends unless there's some anomaly.

There are test failures now with NSS and rustls. Are they permafails or flaky?

There are test failures now with NSS and rustls. Are they permafails or flaky?

They seem to be non-flaky.
I've succeeded in reproducing the NSS failure locally and it shows the transfer never ends (stuck idle until timeout).
I did not try with rustls as it's way too complicated to compile/install locally.
I'm now 80% sure this does not come from the PR changes themselves, but rather are backend-specific problems wich we did not experience before as we were not able to run TLS mail tests. Help would be appreciated.
In all cases this PR is not ready for commit.

In all cases this PR is not ready for commit.

Maybe make it a "draft" PR for the time being then to make it more visible to observers?

I've succeeded in reproducing the NSS failure locally and it shows the transfer never ends (stuck idle until timeout).

Since NSS is already deprecated and scheduled for deletion in August, it feels like a boring job to try to work on a bugfix now for an issue that probably has existed for a long time already and obviously has not disturbed any users enough to report it to us. It feels attractive to just mark this single test disabled for NSS builds.

I did not try with rustls as it's way too complicated to compile/install locally.

rustls also already has a range of tests disabled so it wouldn't be too terrible to add a few more to that list. This backend is stilled marked experimental for this reason.

It generally takes someone with a little insight into rustls and its API to work on these issues.

Thanks for your comment.

In the meantime I've found the problem with NSS: the data_pending method is not implemented and Curl_none_data_pending is used for it, always telling there is no data ready for input.

When some TLS data is buffered by the backend, there's therefore no chance to see it and wake the handle. The problem is likely to occur with all backends not specifically implementing data_pending.

I've managed to make it work with NSS in two ways:

1. By adding a delay in ftpserver.pl between the imap data output and postfetch output: this forces the postfetch output to be transmitted in a separate packet and therefore not buffered. This was just a confirmation test and not a problem solution.
2. By effectively implementing nss_data_pending:.

rustls does implement data_pending but I suspect it does not work as it should.

I then consider this problem as the first one being detected by this PR and already existing before it. Do you want a separate issue report for it ? I think it should be fixed (for all backends) before we commit the current PR to avoid "parasitic" reports in the CI tests.

Do you want a separate issue report for it ?

I think at least a separate PR would make sense, yes!

I think at least a separate PR would make sense, yes!.

I can produce one for NSS only. However the problem should probably be handled more generally, either for each backend or by supporting (blocking?) data reads from backends that do not implement data_pending.

The rustls backend code is out of my skill.

Should be OK now.
New tests successful with NSS after commits ee0f739 and f22cd67 applied.
Still fail with rustls: timeout in hello handshake --> rustls problem; new tests disabled for it.
Gskit should fail too because data_pending method is not implemented.
Other backends do not seem to have problems.

Thanks! and sorry for taking so long

Thanks! and sorry for taking so long

Never mind and thanks for pulling.