Ray,

Here is a sample program that illustrates the problem. I tested the
performance with an 11MB file. This sample program consistently takes
11-17 seconds to complete. If I upload the same file through Firefox, it
takes about 4.5 seconds.

http://www.mediafire.com/view/9rb4aac4hnhma47/curl_filedrop_upload.c

On Tue, Apr 14, 2015 at 9:14 PM, Bryan Christ <bryan.christ_at_gmail.com>
wrote:

> Ray,
>
> Thanks for the reply. It would be quite difficult to create an isolated
> test case due the inherit cost of setting up a RESTful POST to the server.
>
> The problem is very much reproducible. Several users have reported this
> issue. It's not hard to see the problem if you download and build the
> MediaFire FUSE client over at GitHub.
>
> I have seen the notes about the TODO item and I have seen the posts that
> seem to regard this as a SFTP only problem. I suspect that if I build
> libcurl from source and change the define, the performance will go. If
> that be the case, would you accept a patch for curl_easy_setopt() to allow
> this to be configured at run-time?
>
> As for the server, it doesn't support compression or the user-agent
> header. Also, I have a direct connection to our data center. Those are
> definitely not issues.
>
>
> On 4/13/2015 10:01 PM, Bryan Christ wrote:
> > I've been trying to figure out why http POST uploads are so slow with
> > libcurl. Upload speeds continually perform at about 1/10th of the
> > expected performance (or less). Many users have reported this behavior
> > on our forum. I suspect it has a lot to do with CURL_MAX_WRITE_SIZE
> > being set to 16k. Uploads to these same servers through other means
> > (JavaScript for example) reach their expected throughput. The code in
> > question can be seen here:
> >
> > https://github.com/MediaFire/mediafire-fuse/blob/master/utils/http.c
> > (at approx line 314)
> >
> > Assuming the issue is the 16K buffer limit, are there any other
> > options? Asking users to recompile a custom libcurl with a larger
> > buffer size is not very palatable.
>
> If you want help on the list your best bet is a self contained example
> that can be used to reproduce and the details at [1]. The buffer issue
> is in the TODO [2] but from what I see there and elsewhere the
> significance is SFTP related.
>
> Continually or continuously? Is it 100% reproducible? A few ideas:
> Maybe your uploads are compressed when they go through the browser, but
> they are not compressed when uploaded through libcurl. There is no
> compression built in libcurl upload (as far as I know), you would have
> to do it manually and attach the header for content encoding gzip.
> A different user agent (or lack of one -- the default) when you use
> libcurl causes different treatment by the server. This applies to any
> header, really.
> The IP address returned to the browser is different than the IP address
> returned to curl via DNS because the DNS request was made differently.
> Or you just got a different IP address because they rotate.
> I/O in your program. eg posting a FILE but the I/O is backed up.
> Proxy setting is different in the browser than it is in libcurl.
>
> Once you have a way to reproduce try using the curl tool and see if you
> get the same result. Also try the latest version.
>
>
> [1]: http://curl.haxx.se/docs/bugs.html#What_to_report
> [2]: http://curl.haxx.se/docs/todo.html#Modified_buffer_size_approach
>
>
> On Tue, Apr 14, 2015 at 3:52 AM, Aleksandar Lazic <al-curllibrary_at_none.at>
> wrote:
>
>> Dear Bryan
>>
>> Am 14-04-2015 04:01, schrieb Bryan Christ:
>>
>> I've been trying to figure out why http POST uploads are so slow with
>>> libcurl. Upload speeds continually perform at about 1/10th of the expected
>>> performance (or less). Many users have reported this behavior on our forum.
>>> I suspect it has a lot to do with CURL_MAX_WRITE_SIZE being set to 16k.
>>> Uploads to these same servers through other means (JavaScript for example)
>>> reach their expected throughput. The code in question can be seen here:
>>>
>>> https://github.com/MediaFire/mediafire-fuse/blob/master/utils/http.c
>>> [1] (at approx line 314)
>>>
>>> Assuming the issue is the 16K buffer limit, are there any other options?
>>> Asking users to recompile a custom libcurl with a larger buffer size is not
>>> very palatable.
>>>
>>
>> Which version of libcurl is used?
>>
>> http://curl.haxx.se/libcurl/c/curl_version.html
>>
>> BR Aleks
>>
>> --
>>>
>>> Bryan
>>> <><
>>>
>>> -------------------------------------------------------------------
>>> List admin: http://cool.haxx.se/list/listinfo/curl-library [2]
>>> Etiquette: http://curl.haxx.se/mail/etiquette.html [3]
>>>
>>
>>
>> Links:
>> ------
>> [1] https://github.com/MediaFire/mediafire-fuse/blob/master/utils/http.c
>> [2] http://cool.haxx.se/list/listinfo/curl-library
>> [3] http://curl.haxx.se/mail/etiquette.html
>>
>
>
>
> --
> Bryan
> <><
>

-- 
Bryan
<><

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2015-05-20