Daniel,
You proposed exactly what I had in mind. Changing libssh2 to let know the
direction from outside and adapting curl to it. Seems to be the only way.
While debugging I thought about a quick patch to solve the problem. Actually
if I could find an easy way I probably would not send the message to the
mailing list ^^...

The call stack in the bugged situation is:
recv() that returns -1
libssh2_packet_read
libssh2_channel_read_ex
libssh2_sftp_packet_read
libssh2_sftp_packet_requirev
libssh2_sftp_readdir_ex that returns LIBSSH2_ERROR_EAGAIN
ssh_statemach_act that returns CURLE_OK
ssh_easy_statemach
sftp_perform
ssh_do
Curl_do
Curl_perform
curl_easy_perform

ssh_conn::state is SSH_SFTP_READDIR (but this seems to be not important)

If any libssh2 function returns LIBSSH2_ERROR_EAGAIN ssh_statemach_act
breaks returning CURLE_OK. And then ssh_easy_statemach just continues the
next step of the loop. No select or whatever.
I'll try to make a small patch to libssh2 + libcurl. However as I am a
neither libcurl or libssh2 experienced developer my patch can be just a
draft pattern for someone's final one.
To see 100% CPU usage you can use my small tunneling software that stops
transmitting after it receives a certain amount of bytes:
http://fmot.ru/tcp_forward.cpp
So if run
  ./a.out 23 127.0.0.1 22
where sftp works. And then
  curl sftp://127.0.0.1:23 -u [username]
curl will use 100% CPU time.

And thanks for a quick reply, Daniel.

Best regards, Vlad.

On Thu, Sep 25, 2008 at 6:14 PM, Daniel Stenberg <daniel_at_haxx.se> wrote:

> On Thu, 25 Sep 2008, Vlad Grachov wrote:
>
> When libssh2 is set to non-blocking mode its functions return
>> LIBSSH2_ERROR_EAGAIN
>> if recv/send returned EAGAIN. Upon receiving LIBSSH2_ERROR_EAGAIN curl's
>> state machine just repeats the same step (without a call to select/poll).
>>
>
> It does? Can you detail more about what the particular code/function
> sequence that is (not) doing this? It is certainly a bug. Perhaps you even
> have a patch to propose?
>
> But you should also be aware of the fact that there's still a signigicant
> flaw in the libcurl <=> libssh2 communication that will lead to occaional
> busy-loops even when poll() is used:
>
> libcurl waits for socket actions based on what direction the transfer is.
> Alas, if you download data libcurl will wait for the socket to get readable
> and then call libssh2 to get that data. But libssh2 itself both recieves and
> sends data during this call and it can very well return EAGAIN when it
> _sent_ data and then libcurl can immediately discover that the socket is
> indeed still readable and call libssh2 again that returns EAGAIN... until
> the socket becomes writable and libssh2 can continue with its operations.
>
> The fix for this problem has to be made in the libssh2 end and then libcurl
> needs to adapt to it. My suggested approach is to add a function to libssh2
> that says in what direction in wants data when EAGAIN is returned, so that
> libcurl can wait for the proper situation to occur.
>
> I'd love some help on getting all these problems fixed. (lib)curl is
> continuously getting more and more users and attention and my list of things
> to do is growing faster than I manage to complete previously added issues.
>
> BTW, consider taking the further details on the curl-library list where
> most of the hard core libcurl hackers hang out.
>
> --
>
> / daniel.haxx.se
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
> FAQ: http://curl.haxx.se/docs/faq.html
> Etiquette: http://curl.haxx.se/mail/etiquette.html
>

-------------------------------------------------------------------
List admin: http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2008-09-25