New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tool and tests: force flush of all buffers at end of program #8516
Conversation
2782c39
to
820f38e
Compare
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
a0d401d
to
de87e31
Compare
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
On Windows data can be lost in buffers in case of abnormal program termination, especially in process chains as seen due to flaky tests. Therefore flushing all buffers manually should avoid this data loss. In the curl tool we play the safe game by only flushing write buffers, but in the testsuite where we manage all buffers, we flush everything. This should drastically reduce the CI and testsuite flakiness. Supersedes curl#7833 and curl#6064
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
I don't understand why this works. If the program is aborting then these flush calls shouldn't matter. Is it possible it is a coincidence that it appears to work? If a program exits normally (return from main) then the files should be flushed and closed without these additional calls. |
Me neither. I think it most likely works by avoiding data getting lost in process / output forwarding chains. Remember, we are intermixing a couple of different more Unix-based technologies in the testsuite even if it is running under msys. So we have at least the native Win32 binaries, msys-based Perl and Shell, with a couple of scripts in each of those interpreters being intertwined and nested. My theory is that sometimes somewhere in those process chains, the output data get's lost in a buffer. |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
@jay I would like to land this for now and observe the long term statistics on Azure Pipelines after ~30 days, what do you think? |
ok |
The intermediate picture looks quite good so far, not a single test failure on Azure CI since merging this PR: |
On Windows data can be lost in buffers in case of abnormal program
termination, especially in process chains as seen due to flaky tests.
Therefore flushing all buffers manually should avoid this data loss.
In the curl tool we play the safe game by only flushing write buffers,
but in the testsuite where we manage all buffers, we flush everything.
This should drastically reduce the CI and testsuite flakiness.
Supersedes #7833 and #6064