Daniel Stenberg via curl-library (Mon, 13 Aug 2018 23:37:56 +0200
(CEST)):
>On Mon, 13 Aug 2018, Jan Ehrhardt via curl-library wrote:
>
>> Daniel Stenberg (Thu, 9 Aug 2018 16:55:38 +0200 (CEST)):
>> -#define UPLOAD_BUFSIZE CURL_MAX_WRITE_SIZE
>> +#define UPLOAD_BUFSIZE (512*1024)
>>
>> @Daniel Stenberg: is there a reason that is stopping you from changing this
>> in curl?
>
>Yes. (Pretty much explained already in the TODO:
>https://curl.haxx.se/docs/todo.html#Modified_buffer_size_approach)

Clear. But what strikes me: only SFTP is mentioned, but my tests
indicated that increasing this value also influenced the FTP upload
speeds. From 10 to 5 seconds on XP in Daniel Jelinski's testcurl
uploads. And down to less than a second on Win 7 and Win 10.

>1. This buffer is allocated as part of the regular easy handle and increasing
>that allocation this much in one go I fear will cause problems to some
>existing applications out there. Especially those doing a huge amount of
>parallel transfers.

I can imagine this. In my iOS app (using libssh2, not curl), there are
no parallel transfers, but only single transfers of typically 200MB+
files.

>2. There's basically only SFTP that needs this huge buffer so it seems
>wasteful to always use such a large buffer if it isn't necessary. I would like
>to see it conditionally enlarged if SFTP is indeed used.

Sure? What about FTP uploads? See
https://curl.haxx.se/mail/lib-2018-08/0150.html

>3. I think it might be a good idea to let the application control the SFTP
>size, so that the 1,000 parallel transfer applications can use a smaller
>buffer and the single connection apps can use a larger.

The iOS NMSSH library did just that. The buffersize is configurable from
the calling app.

>Besides all that, it would also be interesting to work out what an "ideal"
>size might be. Is 512KB large enough? Is 256KB enough for 99% of the cases?
>Would 1MB help in the extremely high speed high latency cases?

In my iOS tests I settled for 320KB, but that was was for a 4-5 ms
latency connection. I have no idea whet the effect would be for high
latency connections. And the app crashed with 512KB, most likely due to
memory allocation issues. Debugging on iOS is no easy job, so I did nit
investigate further.

>Would it be sensible to attempt to detect the situation and dynamicly grow the
>size if deemed fine?

Ouch. Seems a hell of a job to auto-detect an optimal size. Better make
it configurable from outside (API, command line option).

-- 
Jan
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html