curl_easy_reset() does not reset text/binary option for FTP transfer #6577

bodob · 2021-02-08T13:59:26Z

Transfer after curl_easy_reset() uses incorrect options, thus still happening in ASCII mode, even when explicitly setting CURLOPT_TRANSFERTEXT to 0.
In addition, this also causes '\r' (carriage returns) to be written to the output on Unix, which is wrong for both ASCII and binary mode.

To reproduce

Transfer (upload) uses ASCII mode by using FTP command "TYPE A" via CURLOPT_PREQUOTE.
curl_easy_reset() to reset all options
Next transfer (download)

Example C code

#include <curl/curl.h>
int main(int argc, char *argv[])
{
  CURLcode ret;
  CURL *hnd = NULL;
  struct curl_slist *cmdlist = NULL;

  hnd = curl_easy_init();
  curl_easy_setopt(hnd, CURLOPT_URL, "ftp://ftpserver.example.com//tmp/myfile.txt");
  curl_easy_setopt(hnd, CURLOPT_USERPWD, "anonymous:xyz");
  curl_easy_setopt(hnd, CURLOPT_UPLOAD, 1L);
  // Using CURLOPT_TRANSFERTEXT=1 would be easier, but we want to use/test CURLOPT_PREQUOTE
  cmdlist = curl_slist_append(cmdlist, "TYPE A"); // ASCII transfer
  curl_easy_setopt(hnd, CURLOPT_PREQUOTE, cmdlist);
  curl_easy_setopt(hnd, CURLOPT_VERBOSE, 1L);
  ret = curl_easy_perform(hnd);

  fprintf(stderr, "\nUpload done, now doing download\n\n.");

  // Resetting  all options before download
  curl_easy_reset(hnd);
  // Workaround (instead of the above call):
  // curl_easy_cleanup(hnd);
  // hnd = curl_easy_init();

  curl_easy_setopt(hnd, CURLOPT_URL, "ftp://ftpserver.example.com//tmp/myfile.txt");
  curl_easy_setopt(hnd, CURLOPT_USERPWD, "anonymous:xyz");
  curl_easy_setopt(hnd, CURLOPT_TRANSFERTEXT, 0L); // Trying explicitly set binary mode - does not work
  // Note: Even CURLOPT_TRANSFERTEXT=1 did not correctly download the file -  output still contains \r characters
  curl_easy_setopt(hnd, CURLOPT_VERBOSE, 1L);
  ret = curl_easy_perform(hnd);

  curl_easy_cleanup(hnd);
  hnd = NULL;

  return (int)ret;
}

Example usage

From bash (ftpdld is the name of the executable created from code above):

printf "Line1\nLine2" | ftpdld | od -cx

Workaround

Instead of curl_easy_reset() use a new curl handle for the download.

Expected output

Verbose output (stderr) should indicate binary mode (TYPE I) being used for download.
File downloaded only contains newline (no carriage return) as end-of-line characters.

Actual output

Verbose output (stderr) indicates that ASCII mode is used for download.
File downloaded contains carriage return + newline as end-of-line characters.

curl/libcurl version

libcurl/7.74.0

operating system

Linux xxxxx 3.10.0-1062.el7.x86_64 #1 SMP Thu Jul 18 20:25:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

The text was updated successfully, but these errors were encountered:

bagder · 2021-02-08T14:03:07Z

I renamed it to be more specific

bagder · 2021-02-08T14:54:23Z

I don't think your example case shows any bug. Your code inserts a TYPE A as a command there behind curl's back so it doesn't know you changed the mode and therefore it will also not change it back even if you set it to use binary, because as far as curl knows it already uses binary there.

If you instead do curl_easy_setopt(hnd, CURLOPT_TRANSFERTEXT, 1L); for the first transfer and then reset, you'll see that the second transfer will set it to TYPE I correctly.

bodob · 2021-02-08T15:27:28Z

Well, the "TYPE A" command is also issued via curl's CURLOPT_PREQUOTE option), so it's actually not behind curl's back.
And even when I tell curl afterwards (during the download), that this is now ASCII by using curl_easy_setopt(hnd, CURLOPT_TRANSFERTEXT, 1L);, then the output is still containing carriage returns (coming from the FTP protocol), which are not removed by curl, which is wrong on Unix.

At least it should be documented that commands executed by the CURLOPT_PREQUOTE and CURLOPT_POSTQUOTE options are executed unnoticed by curl and not unset by the curl_easy_reset() function, which can cause problems afterwards, thus the developer is responsible for unsetting anything changed by these commands.
The documentation for curl_easy_reset currently states:

This puts back the handle to the same state as it was in when it was just created with curl_easy_init.

This is obviously not the case.

Of course setting CURLOPT_TRANSFERTEXT=1 during upload the step would be the obvious cure,
but I was in the step of testing CURLOPT_PREQUOTE and CURLOPT_POSTQUOTE options (with curl_slist), and the "TYPE A" command was just the first one I tried - and immediately got into unexpected behaviour, so I am wondering if there are other problems to expect.

jzakrzewski · 2021-02-08T15:46:19Z

Well, the "TYPE A" command is also issued via curl's CURLOPT_PREQUOTE option), so it's actually not behind curl's back.

Yes, but libcurl does not parse your commands, so you're effectively changing state of the connection without informing libcurl about it.

And even when I tell curl afterwards (during the download), that this is now ASCII by using curl_easy_setopt(hnd, CURLOPT_TRANSFERTEXT, 1L);, then the output is still containing carriage returns (coming from the FTP protocol), which are not removed by curl, which is wrong on Unix.

Yep, and it's a known and documented limitation "that nobody has rectified" to quote the docs.

not unset by the curl_easy_reset() function

Is it really so or is the "TYPE A" simply a sticky command that is in effect until you actually issue "TYPE I"?

bagder · 2021-02-08T15:50:16Z

This puts back the handle to the same state as it was in when it was just created with curl_easy_init.
This is obviously not the case.

I still claim it is the case even if it is a bit of a word game. The handle knows what state it wants and the reset restores that "wanted state" to default. What happens isn't related to the handle's state, what happened was that the connection has been modified to be in a different state than libcurl knows about.

... so passed in commands may confuse libcurl's knowledge of state. Reported-by: Bodo Bergmann Fixes #6577

bodob · 2021-02-08T16:08:55Z

So, why is libcurl still producing carriage returns in the output on Unix when I explicitly tell it, that this is an ASCII transfer (so it should remove any incoming carriage returns)?

And I still claim, that it is at least a documentation issue - if not for curl_easy_reset(), then for the CURLOPT_PREQUOTE and CURLOPT_POSTQUOTE options.
Something mentioning that the developer is responsible for unsetting anything in the connection state set by these commands, i.e. for commands that change something the curl handle usually knows about.
IMHO, a "bug" doesn't always mean a bug in the product (which I admit it's probably not, apart from the strange behavior mentioned above), but could also mean a problem in the documentation, where users expect a certain behavior.
A little more explanation in the documentation could prevent such expectations.

bagder · 2021-02-08T16:16:09Z

why is libcurl still producing carriage returns in the output on Unix when I explicitly tell it, that this is an ASCII transfer

That's unrelated to this bug report though and is about curl not doing the necessary text-translations for FTP ASCII transfers. It is mentioned in the CURLOPT_TRANSFERTEXT option man page. This is a known flaw.

bodob · 2021-02-08T16:45:08Z

I do not actually expect a code change, I am fine with a clarifying documentation change for the CURLOPT_PREQUOTE and CURLOPT_POSTQUOTE options (not sure if other options are also affected) to help others to avoid problems like this, i.e. as the libcurl documentation is refered to by others (e.g. wrapper libraries for other languages).

BTW, thanks for the very quick response.

bagder · 2021-02-08T17:04:17Z

#6580 attempts to clarify the curl-not-parsing-quote-commands situation

bagder added the FTP label Feb 8, 2021

bagder changed the title ~~curl_easy_reset() does not reset all options correctly, i.e. for FTP transfer~~ curl_easy_reset() does not reset text/binary option for FTP transfer Feb 8, 2021

bagder added a commit that referenced this issue Feb 8, 2021

CURLOPT_QUOTE.3: clarify that libcurl doesn't parse what's sent

522e9b3

... so passed in commands may confuse libcurl's knowledge of state. Reported-by: Bodo Bergmann Fixes #6577

bagder mentioned this issue Feb 8, 2021

CURLOPT_QUOTE.3: clarify that libcurl doesn't parse what's sent #6580

Closed

bagder closed this as completed in ff9ec4e Feb 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

curl_easy_reset() does not reset text/binary option for FTP transfer #6577

curl_easy_reset() does not reset text/binary option for FTP transfer #6577

bodob commented Feb 8, 2021

bagder commented Feb 8, 2021

bagder commented Feb 8, 2021

bodob commented Feb 8, 2021

jzakrzewski commented Feb 8, 2021

bagder commented Feb 8, 2021

bodob commented Feb 8, 2021

bagder commented Feb 8, 2021

bodob commented Feb 8, 2021

bagder commented Feb 8, 2021

curl_easy_reset() does not reset text/binary option for FTP transfer #6577

curl_easy_reset() does not reset text/binary option for FTP transfer #6577

Comments

bodob commented Feb 8, 2021

To reproduce

Example C code

Example usage

Workaround

Expected output

Actual output

curl/libcurl version

operating system

bagder commented Feb 8, 2021

bagder commented Feb 8, 2021

bodob commented Feb 8, 2021

jzakrzewski commented Feb 8, 2021

bagder commented Feb 8, 2021

bodob commented Feb 8, 2021

bagder commented Feb 8, 2021

bodob commented Feb 8, 2021

bagder commented Feb 8, 2021