Hello,
After some more investigations I found that this problem can be avoided in
user program with thiss call:
    curl_easy_setopt(pMyCurlHandle, CURLOPT_BUFFERSIZE, myMaxPackSize);

This value
#define CURL_MAX_WRITE_SIZE 16384

if just default value in case user didnt specify something else with
CURLOPT_BUFFERSIZE option.

No how this value was defined (16384)?

I checked libssh2 and found this

/* Channel API */
#define LIBSSH2_CHANNEL_WINDOW_DEFAULT 65536
#define LIBSSH2_CHANNEL_PACKET_DEFAULT 16384
#define LIBSSH2_CHANNEL_MINADJUST 1024

So, it is same as packet size. Now I debuged the libssh2 and in default case
(for instance you use ./curl -u username sftp://something.com/bigfile.txt)
varibles from this code have values:

                if (session->packAdd_channel->remote.packet_size <
                    (datalen - session->packAdd_data_head)) {
                    /*
                     * Spec says we MAY ignore bytes sent beyond
                     * packet_size
                     */
                    libssh2_error(session,
                                  LIBSSH2_ERROR_CHANNEL_PACKET_EXCEEDED,
                                  "Packet contains more data than we offered
to receive, truncating",
                                  0);
                    datalen =
                        session->packAdd_channel->remote.packet_size +
                        session->packAdd_data_head;
                }

session->packAdd_channel->remote.packet_size = 16384
datalen = 16406 !!!
session->packAdd_data_head = 9

so datalen comes from here,
            /* we now have the initial blocksize bytes decrypted,
             * and we can extract packet and padding length from it
             */
            p->packet_length = libssh2_ntohu32(block);
looks like sftp server sent slightly more (13 bytes)!

Next:
>> bytesfromsocket = MIN((long)sizerequested, conn->data->set.buffer_size
>> ?
>> conn->data->set.buffer_size : BUFSIZE);
in default case sizerequested == (conn->data->set.buffer_size
?conn->data->set.buffer_size : BUFSIZE)
so it is irrelevant.

Also interesting that even curl cant receive data from that sftp server,
original problem is gone!!! looks like I just discovered another one. But
most important the original failure was happening sometimes, like transfer
works for a while then fails, week later starts to work again, then fails
and all I can see with Unix pstack that it is sitting in recv(), which
returns -1 non stop.
Regards
Aleksey

----- Original Message -----
From: "Daniel Stenberg" <daniel_at_haxx.se>
To: "the curl tool" <curl-users_at_cool.haxx.se>
Cc: "libcurl hacking" <curl-library_at_cool.haxx.se>
Sent: Thursday, September 18, 2008 3:46 PM
Subject: Re: SFTP fails (Failure when receiving data from the peer)

> On Thu, 18 Sep 2008, hippocat_at_verizon.net wrote:
>
> I'm cc'ing my reply to the curl-library list which really is more suitable
> for this topic! Please take follow-ups on that list.
>
> Your mail did confuse me. Here's a bunch of questions and comments...
>
>> And finally:
>>
>> bytesfromsocket = MIN((long)sizerequested, conn->data->set.buffer_size
>> ?
>> conn->data->set.buffer_size : BUFSIZE);
>>
>> if(conn->protocol & PROT_SFTP)
>> nread = Curl_sftp_recv(conn, num, buffertofill, bytesfromsocket);
>>
>> Ok, curl requesting buffer with magic number 16384 + ?head data?
>
> Can you explain to me how it can ask for more than 16384 bytes with this
> code?
>
> As I read it, the code asks for up to 16384 bytes. Possibly less - but
> never more.
>
>> and it is always generating error LIBSSH2_ERROR_CHANNEL_PACKET_EXCEEDED
>> for that specific server (looks like it sends extra data which is ?Spec
>> says we MAY ignore bytes sent beyond? ? just legal, but not for curl).
>
> If libssh2 returns an error libcurl will consider that to be an error. If
> the case is fine by a protocol spec libssh2 should not return an error in
> the first place.
>
>> Then data truncated and makes data out of sync with data len
>
> How is that? libssh2 returns an error then isn't libcurl just bailing out?
>
> And if not, how is it getting out of sync?
>
>> curl getting crazy and connection dropped by sftp server.
>
> How is "curl getting crazy" because of this?
>
>> Simple fix is to reduce CURL_MAX_WRITE_SIZE and it works.
>
> How come 16K is such a magic limit for the SSH2 code, do you know?
>
> I disagree it being a fix though, it's just a work-around!
>
>> How many bytes ? I believe it depends on server, but something like 16000
>> will work.
>
> Hm, does this mean we can force the bug to happen if we instead bump the
> buffer size to something like 32K or so? I mean as a help to fix this
> "properly" in the libssh2 end?
>
> I can consider adding the work-around for the SFTP case just to make
> libcurl play better with existing libssh2 but I would prefer to make a
> real fix (as well).
>
>> Or just disable data truncation in libssh2 which is also works.
>
> But that will make libssh2 send more data than its windows allow, doesn't
> it? Or did I misunderstand that check?
>
>> I beleive there is better patch but I dont have time to dig cURL, so
>> probably developers could take care about this special case ?
>
> I'll try!
>
> --
>
> / daniel.haxx.se
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
> FAQ: http://curl.haxx.se/docs/faq.html
> Etiquette: http://curl.haxx.se/mail/etiquette.html
>

-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2008-09-19