Hi Daniel,

So what we try to do follows more or less the tutorial in https://curl.haxx.se/libcurl/c/libcurl-tutorial.html, chapter "HTTP POSTing"
- We publish a boundary signature in the header before the payload transmission
- the transmission should end when the boundary signature is found in the next body (after the payload, which is binary data)

I have consolidated the code a little bit:

curl_easy_setopt(etaso_DMHttp_Prv_session_ast[idx_u16].curl_pst, CURLOPT_READFUNCTION, cfgUser_pst->readCb_pfct);
curl_easy_setopt(etaso_DMHttp_Prv_session_ast[idx_u16].curl_pst, CURLOPT_READDATA, &etaso_DMHttp_Prv_session_ast[idx_u16].idModuleSession_u16);
curl_easy_setopt(etaso_DMHttp_Prv_session_ast[idx_u16].curl_pst, CURLOPT_POST, 1);
                    
// Set the expected POST size.
// the contextLen is the actual paylod plus the final boundary signature)
curl_easy_setopt(etaso_DMHttp_Prv_session_ast[idx_u16].curl_pst, CURLOPT_POSTFIELDSIZE_LARGE, cfgUser_pst->txContentLen_u32);
curl_easy_setopt(etaso_DMHttp_Prv_session_ast[idx_u16].curl_pst, CURLOPT_COPYPOSTFIELDS, cfgUser_pst->userData_pchr);
 
curl_easy_setopt(etaso_DMHttp_Prv_session_ast[idx_u16].curl_pst, CURLOPT_VERBOSE, 1L);
 // Adding HTTP header to curl handler
curl_easy_setopt(etaso_DMHttp_Prv_session_ast[idx_u16].curl_pst, CURLOPT_HTTPHEADER, etaso_DMHttp_Prv_session_ast[idx_u16].headerList_pst);

if (CURLM_OK == curl_multi_add_handle(etaso_DMHttp_Prv_session_ast[idx_u16].curlM_pst, etaso_DMHttp_Prv_session_ast[idx_u16].curl_pst))
{
    // internal session handling
}
  

>> Do you set CURLOPT_INFILESIZE?
No, we go with POSTFIELDSIZE_LARGE, similar to the example in the tutorial

>> Do you set Content-Length yourself ?
Yes, in my understanding of the mechanism: we add the length of the boundary signature / header information to the payload -> gives the content-length

>> Do you upload chunked-encoded?
No - I was not the original implementor of our code, but I think initially it was desired to also support HTTP 1.0

I am not sure that we do it entirely correct, and maybe we are making it more complicated than necessary, that's why I am asking ;-)

Now, I attached to log snippets (the libcurl logging is intermixed with our own log messages)

1) with libcurl 7.61-0. I tend to think that this is the way it is supposed to work:
The bare file size is , with the boundary information and header stuff it adds up to 29108 bytes. After the last callback call, where the boundary is finally transmitted (34 bytes), it says "We are completely uploaded and fine"

[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_StartUpload] Upload URL: http://localhost/upload.php
[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_StartUpload] Upload file path: /var/lib/OBUA/Uploads/logfile12345.zip
[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_StartUpload] Upload file size: 28928
[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_StartUpload] Upload requester: 200
[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_StartUpload] Upload UUID 15215991-C11B-46CC-AB82-CC95F4F174FD. Session id: 0
* Uses proxy env variable no_proxy == 'hi0vm068.de.bosch.com'
* Uses proxy env variable http_proxy == 'http://localhost:3128'
* Trying ::1...
* TCP_NODELAY set
* connect to ::1 port 3128 failed: Connection refused
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 3128 (#0)
> POST http://localhost/upload.php HTTP/1.1
Host: localhost
Proxy-Connection: Keep-Alive
Connection: keep-alive
Content-Type: multipart/form-data; boundary=--bG9nZmlsZTEyMzQ1LnppcA==
Accept: application/vnd.syncml.dm+xml
Content-Length: 29108

[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_Prv_TxDataCb] callback invokation (step) with size, nrMem: 0 1 16384
[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_Prv_TxDataCb] callback returns upload Size: 146
[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_Prv_TxDataCb] callback invokation (step) with size, nrMem: 1 1 16384
[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_Prv_TxDataCb] callback returns upload Size: 8192
[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_Prv_TxDataCb] callback invokation (step) with size, nrMem: 1 1 16384
[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_Prv_TxDataCb] callback returns upload Size: 8192
[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_Prv_TxDataCb] callback invokation (step) with size, nrMem: 1 1 16384
[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_Prv_TxDataCb] callback returns upload Size: 8192
[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_Prv_TxDataCb] callback invokation (step) with size, nrMem: 1 1 16384
[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_Prv_TxDataCb] callback returns upload Size: 4352
[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_Prv_TxDataCb] callback invokation (step) with size, nrMem: 1 1 16384
 [INFO][2018-08-31 10:27:30][ULC][etaso_ULC_Prv_TxDataCb] callback returns upload Size: 34
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Date: Fri, 31 Aug 2018 08:27:30 GMT
< Server: Apache/2.4.7 (Ubuntu)
< X-Powered-By: PHP/5.5.9-1ubuntu4.25
< Content-Length: 45
< Keep-Alive: timeout=5, max=100
< Connection: keep-alive
< Content-Type: text/html
< Proxy-Connection: keep-alive
<
* Connection #0 to host localhost left intact
[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_MainFunction] Uploading successful in session: 0
[INFO][2018-08-31 10:27:30][ULC][etaso_ULC_MainFunction] Upload state machine switching to IDLE session: 0

----------------------------------------------------------------------------------------------------------------------------------------------------

2) exactly the same client code, but we link against libcurl 7.35.0
This time, after the 200/OK packet, there is yet another READFUNCTION callback invocation, where there is no data to transmit, so we return 0

[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_StartUpload] Upload URL: http://localhost/upload.php
[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_StartUpload] Upload file path: /var/lib/OBUA/Uploads/logfile12345.zip
[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_StartUpload] Upload file size: 28928
[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_StartUpload] Upload requester: 200
[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_StartUpload] Upload UUID 5F15392C-5A1F-4BDB-AF6C-2EED256C4EB9. Session id: 0
* Hostname was NOT found in DNS cache
* Trying ::1...
* connect to ::1 port 3128 failed: Connection refused
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 3128 (#0)
> POST http://localhost/upload.php HTTP/1.1
Host: localhost
Proxy-Connection: Keep-Alive
Connection: keep-alive
Content-Type: multipart/form-data; boundary=--bG9nZmlsZTEyMzQ1LnppcA==
Accept: application/vnd.syncml.dm+xml
Content-Length: 29108

[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_Prv_TxDataCb] callback invokation (step) with size, nrMem: 0 1 16384
[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_Prv_TxDataCb] callback returns upload Size: 146
[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_Prv_TxDataCb] callback invokation (step) with size, nrMem: 1 1 16384
[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_Prv_TxDataCb] callback returns upload Size: 8192
[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_Prv_TxDataCb] callback invokation (step) with size, nrMem: 1 1 16384
[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_Prv_TxDataCb] callback returns upload Size: 8192
[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_Prv_TxDataCb] callback invokation (step) with size, nrMem: 1 1 16384
[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_Prv_TxDataCb] callback returns upload Size: 8192
[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_Prv_TxDataCb] callback invokation (step) with size, nrMem: 1 1 16384
[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_Prv_TxDataCb] callback returns upload Size: 4352
[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_Prv_TxDataCb] callback invokation (step) with size, nrMem: 1 1 16384
[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_Prv_TxDataCb] callback returns upload Size: 34
< HTTP/1.1 200 OK
< Date: Fri, 31 Aug 2018 09:36:56 GMT
* Server Apache/2.4.7 (Ubuntu) is not blacklisted
< Server: Apache/2.4.7 (Ubuntu)
< X-Powered-By: PHP/5.5.9-1ubuntu4.25
< Content-Length: 45
< Keep-Alive: timeout=5, max=100
< Connection: keep-alive
< Content-Type: text/html
< Proxy-Connection: keep-alive
<
[INFO][2018-08-31 11:36:56][ULC][etaso_ULC_Prv_TxDataCb] callback invokation (step) with size, nrMem: 2 1 16384
 [INFO][2018-08-31 11:36:56][ULC][etaso_ULC_Prv_TxDataCb] callback returns upload Size: 0
* Connection #0 to host localhost left intact

So, to summarize again:
- in both cases the upload works fine, after the final 34 bytes are transmitted, the uploaded file is correct and intact on the upload server
- but: with libcurl 7.35.0 there is an extra invokation of the READFUNCTION callback, in which we return 0

But our design currently forces us to treat the two cases differently (there is a state machine involved, which does or does not (7.61) expect the extra READFUNCTION invocation). If we link against 7.61.0 and keep waiting for the final 0-byte READFUNCTION, we would wait forever, so we check libcurl versions currently as a workaround - but that is certainly not nice...

So we were wondering:
- what is the reason for the different behaviour ?
- from when did it possibly change ?
- how can we make our client so generic that it works with all libcurl versions say between 7.35 and 7.61 ?

Thanks again,

Mit freundlichen Grüßen / Best regards

 Stefan Syberichs

Engineering RTA Solutions - Consulting (ETAS/ERS-CON)
Tel. +49 711 3423-2948 | Fax +49 711 811 | Stefan.Syberichs_at_etas.com

-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2018-08-31