-Z / --parallel breaks --fail-early --fail #6939
Comments
|
Ref: #6921 It might be a bug, I'm not sure myself. It seems --fail-early wasn't meant for parallel transfers. The transfers in progress would need to be properly cleaned up. |
|
Haha, OK. So I was a couple of days short on my master build. I see that the error code is not discarded anymore. But the I'll agree that I'm not sure what the best thing to do would be: finish in-flight transfers, or close everything down. But I certainly feel that starting new transfers is not what should happen. |
luke-clifton
changed the title
It was added long before we did parallel transfers and I think we just haven't adapted parallel mode properly to deal with it. I think it sounds like a proper bug... |
|
We could set a flag in the multi loop and then abort from the progress callback. diff --git a/src/tool_progress.c b/src/tool_progress.c
index da6c2bc..4123e93 100644
--- a/src/tool_progress.c
+++ b/src/tool_progress.c
@@ -101,6 +101,9 @@ int xferinfo_cb(void *clientp,
per->ultotal = ultotal;
per->ulnow = ulnow;
+ if(per->abort)
+ return -1;
+
if(config->readbusy) {
config->readbusy = FALSE;
curl_easy_pause(per->curl, CURLPAUSE_CONT); |
|
We'd have to discuss what the option should do about other already started transfers... |
|
I'd say That being said, my usecase is 1000's of tiny transfers, so I don't care 🤷 |
|
I'm fine with killing off the already started ones too, as long as we document that this is to be expected when |
Aborting from the progress callback would kill them off |
|
@jay will you make a PR based on your proposed fix? |
(Draft.) - Abort via progress callback to fail early on parallel transfers. Fixes curl#6939 Closes #xxxx
#6984 to abort via progress callback but it would mean setting CURLOPT_NOPROGRESS to 0L. I'm not sure what to do about the return code in the latter case edit: maybe for the return code skip setting aborted by callback if another error is already set diff --git a/src/tool_operate.c b/src/tool_operate.c
index 8db6a46..5219eaa 100644
--- a/src/tool_operate.c
+++ b/src/tool_operate.c
@@ -2333,7 +2333,7 @@ static CURLcode parallel_transfers(struct GlobalConfig *global,
ended->startat = delay ? time(NULL) + delay/1000 : 0;
}
else {
- if(tres)
+ if(tres && (!result || tres != CURLE_ABORTED_BY_CALLBACK))
result = tres;
if(result && global->fail_early)
wrapitup = TRUE;
@@ -2359,7 +2359,7 @@ static CURLcode parallel_transfers(struct GlobalConfig *global,
CURLcode tres = add_parallel_transfers(global, multi, share,
&more_transfers,
&added_transfers);
- if(tres)
+ if(tres && (!result || tres != CURLE_ABORTED_BY_CALLBACK))
result = tres;
if(added_transfers)
/* we added new ones, make sure the loop doesn't exit yet */(^^ I've added the above change to the draft) |
(Draft.) - Abort via progress callback to fail early on parallel transfers. Fixes curl#6939 Closes #xxxx
- Abort via progress callback to fail early during parallel transfers. When a critical error occurs during a transfer (eg --fail-early constraint) then other running transfers will be aborted via progress callback and finish with error CURLE_ABORTED_BY_CALLBACK (42). In this case, the callback error does not become the most recent error and a custom error message is used for those transfers: curld --fail --fail-early --parallel https://httpbin.org/status/404 https://httpbin.org/delay/10 curl: (22) The requested URL returned error: 404 curl: (42) Transfer aborted due to critical error in another transfer > echo %ERRORLEVEL% 22 Fixes curl#6939 Closes #xxxx
- Abort via progress callback to fail early during parallel transfers. When a critical error occurs during a transfer (eg --fail-early constraint) then other running transfers will be aborted via progress callback and finish with error CURLE_ABORTED_BY_CALLBACK (42). In this case, the callback error does not become the most recent error and a custom error message is used for those transfers: curld --fail --fail-early --parallel https://httpbin.org/status/404 https://httpbin.org/delay/10 curl: (22) The requested URL returned error: 404 curl: (42) Transfer aborted due to critical error in another transfer > echo %ERRORLEVEL% 22 Fixes curl#6939 Closes #xxxx
- Abort via progress callback to fail early during parallel transfers. When a critical error occurs during a transfer (eg --fail-early constraint) then other running transfers will be aborted via progress callback and finish with error CURLE_ABORTED_BY_CALLBACK (42). In this case, the callback error does not become the most recent error and a custom error message is used for those transfers: curld --fail --fail-early --parallel https://httpbin.org/status/404 https://httpbin.org/delay/10 curl: (22) The requested URL returned error: 404 curl: (42) Transfer aborted due to critical error in another transfer > echo %ERRORLEVEL% 22 Fixes curl#6939 Closes #xxxx
- Abort via progress callback to fail early during parallel transfers. When a critical error occurs during a transfer (eg --fail-early constraint) then other running transfers will be aborted via progress callback and finish with error CURLE_ABORTED_BY_CALLBACK (42). In this case, the callback error does not become the most recent error and a custom error message is used for those transfers: curld --fail --fail-early --parallel https://httpbin.org/status/404 https://httpbin.org/delay/10 curl: (22) The requested URL returned error: 404 curl: (42) Transfer aborted due to critical error in another transfer > echo %ERRORLEVEL% 22 Fixes curl#6939 Closes #xxxx
I did this
I expected the following
Stop as soon as one as we know something failed, and return a non-0 exit code.
What actually happens
curl will attempt every single url, and then exit with a 0.
curl/libcurl version
I did it on master:
But also other version.
operating system
Darwin BLAH 19.6.0 Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64 x86_64 i386 MacBookPro15,1 Darwin
The text was updated successfully, but these errors were encountered: