Use both --etag-compare and --etag-save with the same etag file name #5179

kwon-young · 2020-04-03T18:06:05Z

I did this

On the first run where test.etag and test.ext does not exist:

$ curl --etag-compare test.etag --etag-save test.etag -o test.ext -v 'http://myurl.com/test.ext'

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ip:80...
* Connected to myurl.com (ip) port 80 (#0)
> GET /test.ext HTTP/1.1
> Host: myurl.com
> User-Agent: curl/7.69.1
> Accept: */*
> If-None-Match: ""
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 03 Apr 2020 17:43:36 GMT
< Content-Type: application/ext
< Content-Length: 3840872
< Connection: keep-alive
< Set-Cookie: __cfduid=d8d26cdd5d1359a2545e22facdeb5c54f1585935816; expires=Sun, 03-May-20 17:43:36 GMT; path=/; domain=.myurl.com; HttpOnly; SameSite=Lax
< Last-Modified: Tue, 25 Feb 2014 02:26:16 GMT
< ETag: "530bff48-3a9b68"
< Cache-Control: max-age=14400
< CF-Cache-Status: REVALIDATED
< Accept-Ranges: bytes
< Server: cloudflare
< CF-RAY: 57e497c43ee9ee8d-CDG
< 
{ [967 bytes data]
100 3750k  100 3750k    0     0  2976k      0  0:00:01  0:00:01 --:--:-- 2976k
* Connection #0 to host myurl.com left intact

On the second run, test.ext is downloaded again.

$ curl --etag-compare test.etag --etag-save test.etag -o test.ext -v 'http://myurl.com/test.ext'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ip:80...
* Connected to myurl.com (ip) port 80 (#0)
> GET /test.ext HTTP/1.1
> Host: myurl.com
> User-Agent: curl/7.69.1
> Accept: */*
> If-None-Match: ""
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 03 Apr 2020 17:46:33 GMT
< Content-Type: application/ext
< Content-Length: 3840872
< Connection: keep-alive
< Set-Cookie: __cfduid=d2c4673cee3162b5136f49cde885137ca1585935993; expires=Sun, 03-May-20 17:46:33 GMT; path=/; domain=.myurl.com; HttpOnly; SameSite=Lax
< Last-Modified: Tue, 25 Feb 2014 02:26:16 GMT
< ETag: "530bff48-3a9b68"
< Cache-Control: max-age=14400
< CF-Cache-Status: HIT
< Age: 177
< Accept-Ranges: bytes
< Server: cloudflare
< CF-RAY: 57e49c182b07a8c1-CDG
< 
{ [965 bytes data]
100 3750k  100 3750k    0     0  3015k      0  0:00:01  0:00:01 --:--:-- 3015k
* Connection #0 to host myurl.com left intact

I expected the following

The second curl command should not download text.ext again.

curl/libcurl version

curl 7.69.1 (x86_64-pc-linux-gnu) libcurl/7.69.1 OpenSSL/1.1.1e zlib/1.2.11 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh2/1.9.0 nghttp2/1.40.0
Release-Date: 2020-03-11
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

operating system

Linux edwin 5.5.13-1-MANJARO #1 SMP PREEMPT Wed Mar 25 17:14:28 UTC 2020 x86_64 GNU/Linux

Discussion

After going through the pull-request #4678 that implemented using both --etag-compare and --etag-save, I understood that the --etag-save is processed first and open for writing test.etag (see

curl/src/tool_operate.c

Lines 908 to 976 in 57476a9

    
           /* --etag-save */ 
        
           etag_save = &per->etag_save; 
        
           etag_save->stream = stdout; 
        
           if(config->etag_save_file) { 
        
             /* open file for output: */ 
        
             if(strcmp(config->etag_save_file, "-")) { 
        
               FILE *newfile = fopen(config->etag_save_file, "wb"); 
        
               if(!newfile) { 
        
                 warnf( 
        
                   config->global, 
        
                   "Failed to open %s\n", config->etag_save_file); 
        
                 result = CURLE_WRITE_ERROR; 
        
                 break; 
        
               } 
        
               else { 
        
                 etag_save->filename = config->etag_save_file; 
        
                 etag_save->s_isreg = TRUE; 
        
                 etag_save->fopened = TRUE; 
        
                 etag_save->stream = newfile; 
        
               } 
        
             } 
        
             else { 
        
               /* always use binary mode for protocol header output */ 
        
               set_binmode(etag_save->stream); 
        
             } 
        
           } 
        
           /* --etag-compare */ 
        
           if(config->etag_compare_file) { 
        
             char *etag_from_file = NULL; 
        
             char *header = NULL; 
        
             /* open file for reading: */ 
        
             FILE *file = fopen(config->etag_compare_file, FOPEN_READTEXT); 
        
             if(!file) { 
        
               errorf(config->global, 
        
                      "Failed to open %s\n", config->etag_compare_file); 
        
               result = CURLE_READ_ERROR; 
        
               break; 
        
             } 
        
             if((PARAM_OK == file2string(&etag_from_file, file)) && 
        
                etag_from_file) { 
        
               header = aprintf("If-None-Match: \"%s\"", etag_from_file); 
        
               Curl_safefree(etag_from_file); 
        
             } 
        
             else 
        
               header = aprintf("If-None-Match: \"\""); 
        
             if(!header) { 
        
               if(file) 
        
                 fclose(file); 
        
               errorf(config->global, 
        
                      "Failed to allocate memory for custom etag header\n"); 
        
               result = CURLE_OUT_OF_MEMORY; 
        
               break; 
        
             } 
        
             /* add Etag from file to list of custom headers */ 
        
             add2list(&config->headers, header); 
        
             Curl_safefree(header); 
        
             if(file) { 
        
               fclose(file); 
        
             } 
        
           }

).
This results into creating test.etag if it does not exists or truncating test.etag if it exists.
Truncating test.etag then make the --etag-compare option useless.

I believe this behavior is a bug because it is different to the behavior described here: https://daniel.haxx.se/blog/2019/12/06/curl-speaks-etag/ where this example is given:

curl --etag-compare etag.txt --etag-save etag.txt https://example.com -o saved-file

I would like to propose a pull-request where --etag-compare is processed before --etag-save which will results in the expected behavior.

I suppose curl should not fail if --etag-compare test.etag is specified but test.etag does not exist and --etag-save is specified.

Thank you for this awesome tool!

The text was updated successfully, but these errors were encountered:

kwon-young mentioned this issue Apr 3, 2020

Use both --etag-compare and --etag-save with the same etag file name #5180

Closed

bagder added cmdline tool HTTP labels Apr 4, 2020

bagder closed this as completed in a448a4c Apr 4, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use both --etag-compare and --etag-save with the same etag file name #5179

Use both --etag-compare and --etag-save with the same etag file name #5179

kwon-young commented Apr 3, 2020 •

edited

Use both --etag-compare and --etag-save with the same etag file name #5179

Use both --etag-compare and --etag-save with the same etag file name #5179

Comments

kwon-young commented Apr 3, 2020 • edited

I did this

I expected the following

curl/libcurl version

operating system

Discussion

kwon-young commented Apr 3, 2020 •

edited