Ray,

Thanks for the reply. It would be quite difficult to create an isolated
test case due the inherit cost of setting up a RESTful POST to the server.

The problem is very much reproducible. Several users have reported this
issue. It's not hard to see the problem if you download and build the
MediaFire FUSE client over at GitHub.

I have seen the notes about the TODO item and I have seen the posts that
seem to regard this as a SFTP only problem. I suspect that if I build
libcurl from source and change the define, the performance will go. If
that be the case, would you accept a patch for curl_easy_setopt() to allow
this to be configured at run-time?

As for the server, it doesn't support compression or the user-agent
header. Also, I have a direct connection to our data center. Those are
definitely not issues.

On 4/13/2015 10:01 PM, Bryan Christ wrote:
> I've been trying to figure out why http POST uploads are so slow with
> libcurl. Upload speeds continually perform at about 1/10th of the
> expected performance (or less). Many users have reported this behavior
> on our forum. I suspect it has a lot to do with CURL_MAX_WRITE_SIZE
> being set to 16k. Uploads to these same servers through other means
> (JavaScript for example) reach their expected throughput. The code in
> question can be seen here:
>
> https://github.com/MediaFire/mediafire-fuse/blob/master/utils/http.c
> (at approx line 314)
>
> Assuming the issue is the 16K buffer limit, are there any other
> options? Asking users to recompile a custom libcurl with a larger
> buffer size is not very palatable.

If you want help on the list your best bet is a self contained example
that can be used to reproduce and the details at [1]. The buffer issue
is in the TODO [2] but from what I see there and elsewhere the
significance is SFTP related.

Continually or continuously? Is it 100% reproducible? A few ideas:
Maybe your uploads are compressed when they go through the browser, but
they are not compressed when uploaded through libcurl. There is no
compression built in libcurl upload (as far as I know), you would have
to do it manually and attach the header for content encoding gzip.
A different user agent (or lack of one -- the default) when you use
libcurl causes different treatment by the server. This applies to any
header, really.
The IP address returned to the browser is different than the IP address
returned to curl via DNS because the DNS request was made differently.
Or you just got a different IP address because they rotate.
I/O in your program. eg posting a FILE but the I/O is backed up.
Proxy setting is different in the browser than it is in libcurl.

Once you have a way to reproduce try using the curl tool and see if you
get the same result. Also try the latest version.

[1]: http://curl.haxx.se/docs/bugs.html#What_to_report
[2]: http://curl.haxx.se/docs/todo.html#Modified_buffer_size_approach

On Tue, Apr 14, 2015 at 3:52 AM, Aleksandar Lazic <al-curllibrary_at_none.at>
wrote:

> Dear Bryan
>
> Am 14-04-2015 04:01, schrieb Bryan Christ:
>
> I've been trying to figure out why http POST uploads are so slow with
>> libcurl. Upload speeds continually perform at about 1/10th of the expected
>> performance (or less). Many users have reported this behavior on our forum.
>> I suspect it has a lot to do with CURL_MAX_WRITE_SIZE being set to 16k.
>> Uploads to these same servers through other means (JavaScript for example)
>> reach their expected throughput. The code in question can be seen here:
>>
>> https://github.com/MediaFire/mediafire-fuse/blob/master/utils/http.c [1]
>> (at approx line 314)
>>
>> Assuming the issue is the 16K buffer limit, are there any other options?
>> Asking users to recompile a custom libcurl with a larger buffer size is not
>> very palatable.
>>
>
> Which version of libcurl is used?
>
> http://curl.haxx.se/libcurl/c/curl_version.html
>
> BR Aleks
>
> --
>>
>> Bryan
>> <><
>>
>> -------------------------------------------------------------------
>> List admin: http://cool.haxx.se/list/listinfo/curl-library [2]
>> Etiquette: http://curl.haxx.se/mail/etiquette.html [3]
>>
>
>
> Links:
> ------
> [1] https://github.com/MediaFire/mediafire-fuse/blob/master/utils/http.c
> [2] http://cool.haxx.se/list/listinfo/curl-library
> [3] http://curl.haxx.se/mail/etiquette.html
>

-- 
Bryan
<><

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2015-04-15