Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

7.69, 7.70 busy-loops when doing SOCKS connection #5532

Closed
zloi-user opened this issue Jun 6, 2020 · 15 comments
Closed

7.69, 7.70 busy-loops when doing SOCKS connection #5532

zloi-user opened this issue Jun 6, 2020 · 15 comments

Comments

@zloi-user
Copy link

zloi-user commented Jun 6, 2020

curl loads the cpu, with many threads this is especially noticeable

on version 7.68.0 the problem is not observed

@zloi-user zloi-user changed the title 7.69, 7.70 loads the processor 7.69, 7.70 loads the cpu Jun 6, 2020
@bagder
Copy link
Member

bagder commented Jun 6, 2020

Then please give us a lot more details about the problem and how to reproduce!

@bagder
Copy link
Member

bagder commented Jun 7, 2020

That's not very helpful.

@zloi-user
Copy link
Author

zloi-user commented Jun 7, 2020

I have a libcurl-based proxy checker, when testing part of the bad proxies 7.69 and 7.70 - it loads the system 100%, on version 7.68 only ~2%

sorry for my english, used google translator

perl script example

#!/usr/bin/perl -w
use strict;
use WWW::Curl::Easy;
use POSIX ':sys_wait_h';

use constant MAX_FORKs => 100;
$SIG{CHLD}='IGNORE';

my @apids;

print "Run 'top -p $$' test start in 15 seconds\n";
sleep 15;

my @aproxy=(
'95.216.33.245:9514',
'186.226.172.115:63253',
'181.101.51.44:1080',
'41.164.169.106:61671',
'187.73.68.14:53281',
'177.73.47.110:4145',
'115.88.138.251:4145',
'181.102.25.18:1080',
'125.26.108.12:38150',
'92.245.104.154:49384',
'94.139.176.179:1085',
'134.0.63.134:1723',
'165.16.54.244:31618',
'121.230.103.6:38801',
'186.225.32.97:1080',
'120.28.110.216:4145',
'211.35.73.254:4145',
'200.212.2.125:61283',
'181.102.116.157:7071',
'123.168.67.70:54321',
'117.69.147.113:38801',
'178.150.194.243:4153',
'181.3.111.84:1080',
'138.68.240.218:3128',
'178.218.58.234:4145',
'199.188.93.247:8000',
'95.68.115.202:53281',
'91.247.250.215:4145',
'45.232.249.73:4145',
'183.161.145.168:38801',
'185.134.99.66:51327',
'114.104.182.54:38801',
'200.152.78.48:38543',
'27.22.132.77:38801',
'118.174.220.32:51399',
'196.28.234.66:1080',
'181.5.241.176:7071',
'195.154.106.167:80',
'187.103.15.17:40559',
'175.41.44.43:34079',
'36.66.61.7:46118',
'181.102.28.40:7071',
'184.168.146.10:43227',
'168.181.134.119:41351',
'94.124.193.243:48258',
'41.215.37.230:4145',
'182.52.51.47:51449',
'186.84.172.7:60657',
'91.236.251.131:9251',
'181.102.182.177:1080',
'143.137.126.37:63253',
'181.102.84.217:7071',
'176.107.251.132:4145',
'125.25.165.22:4153',
'109.162.241.109:1085',
'185.85.219.74:61068',
'212.1.97.254:4145',
'103.239.254.114:50409',
'185.162.0.74:10801',
'175.111.15.2:33159',
'190.128.26.99:45412',
'61.145.48.38:4216',
'150.129.52.74:6667',
'178.219.173.193:4145',
'186.225.194.78:34110',
'43.248.73.122:31308',
'209.16.78.27:54321',
'103.210.142.54:4145',
'14.116.208.123:38801',
'181.211.38.62:35069',
'181.3.102.155:1080',
'113.12.202.50:53324',
'112.4.232.37:1080',
'195.22.253.238:4145',
'49.85.97.57:38801',
'86.34.197.6:23500',
'85.15.179.235:4145',
'37.26.136.181:33578',
'165.227.20.194:9050',
'177.125.206.227:4145',
'177.200.159.213:32817',
'178.47.131.202:4145',
'52.68.105.182:80',
'199.188.93.91:9000',
'81.89.113.142:44493',
'60.169.213.35:38801',
'192.158.15.201:50877',
'216.109.197.49:36974',
'45.228.4.90:4145',
'117.65.99.236:38801',
'181.3.63.8:7071',
'181.7.223.7:1080',
'176.192.8.206:61339',
'177.11.156.246:4145',
'124.156.98.172:80',
'203.77.240.75:4145',
'114.33.206.143:36346',
'45.112.11.25:1080',
'190.144.116.34:32877',
'150.129.201.30:6667'
);


for(my $i_pid=0;$i_pid<MAX_FORKs; $i_pid++){
 fork_curl($aproxy[$i_pid]) unless $apids[$i_pid]=fork;
}

for my $cur_pid (@apids){
	while($cur_pid && waitpid($cur_pid,WNOHANG)==0){
		sleep 2
	}
}

sub fork_curl {
 my $addres_porxy=shift;
 my $response_body='';
 my $curl=WWW::Curl::Easy->new;
 $curl->setopt(CURLOPT_WRITEDATA,\$response_body);
 $curl->setopt(CURLOPT_PROXY,'socks4://'.$addres_porxy);
 $curl->setopt(CURLOPT_URL,'https://www.google.com/favicon.ico');
 $curl->perform();
 exit 0;
}

@jay
Copy link
Member

jay commented Jun 8, 2020

Can you reproduce without forking? Is there a specific proxy connection causing the problem?

@zloi-user
Copy link
Author

zloi-user commented Jun 8, 2020

Can you reproduce without forking? Is there a specific proxy connection causing the problem?

try the proxy 115.88.138.251:4145, I have this proxy currently loads the cpu core 100%

curl --proxy 'socks4://115.88.138.251:4145' 'https://www.google.com/favicon.ico' -o - >>/dev/null

if it does not load the processor, try to run several times

@jzakrzewski
Copy link
Contributor

I've quickly tried it out. It doesn't hang each time but it happens

me@machine /tmp $ curl --proxy 'socks4://115.88.138.251:4145' 'https://www.google.com/favicon.ico'
Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.
me@machine /tmp $ curl --proxy 'socks4://115.88.138.251:4145' 'https://www.google.com/favicon.ico' -v
*   Trying 115.88.138.251:4145...
* SOCKS4 communication to www.google.com:443
* SOCKS4 connect to IPv4 216.58.215.228 (locally resolved)
^C
me@machine /tmp $ curl -V
curl 7.70.0 (x86_64-pc-linux-gnu) libcurl/7.70.0 OpenSSL/1.1.1g zlib/1.2.11 libssh2/1.9.0_DEV
Release-Date: 2020-04-29
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s rtsp scp sftp smtp smtps tftp 
Features: HTTPS-proxy Largefile libz NTLM SSL TLS-SRP UnixSockets
me@machine /tmp $ 

During the second run I had 96% on one core used by curl.

@jay
Copy link
Member

jay commented Jun 8, 2020

if it does not load the processor, try to run several times

...

I've quickly tried it out. It doesn't hang each time but it happens

I ran it a few twenty times and I don't see anything unusual here. Sometimes the transfer succeeds and sometimes it doesn't due to RST (shown as "Failed receiving connect request ack").

curl 7.71.0-DEV (i386-pc-win32) libcurl/7.71.0-DEV OpenSSL/1.0.2t nghttp2/1.40.0

@zloi-user
Copy link
Author

zloi-user commented Jun 8, 2020

if it does not load the processor, try to run several times

...

I've quickly tried it out. It doesn't hang each time but it happens

I ran it a few twenty times and I don't see anything unusual here. Sometimes the transfer succeeds and sometimes it doesn't due to RST (shown as "Failed receiving connect request ack").

curl 7.71.0-DEV (i386-pc-win32) libcurl/7.71.0-DEV OpenSSL/1.0.2t nghttp2/1.40.0

I tried on windows to reproduce the problem - it did not work, it seems this problem is linux curl implementation

sorry for my english, used google translator

bagder added a commit that referenced this issue Jun 8, 2020
The SOCKS4/5 state machines weren't properly terminated when the proxy
connection got closed, leading to a busy-loop.

Reported-By: zloi-user on github
Fixes #5532
bagder added a commit that referenced this issue Jun 8, 2020
The SOCKS4/5 state machines weren't properly terminated when the proxy
connection got closed, leading to a busy-loop.

Reported-By: zloi-user on github
Fixes #5532
@bagder
Copy link
Member

bagder commented Jun 8, 2020

It is system agnostic and happens when the SOCKS proxy closes the connection while curl runs the SOCKS state machine. See my proposed fix in #5542.

@bagder bagder self-assigned this Jun 8, 2020
@bagder bagder changed the title 7.69, 7.70 loads the cpu 7.69, 7.70 busy-loops when doing SOCKS connection Jun 8, 2020
@bagder
Copy link
Member

bagder commented Jun 8, 2020

@zloi-user: it would be awesome if you could try that patch and tell us if it fixes the problem you see!

@zloi-user
Copy link
Author

compiled 7.69.1 with a patch

problem still exists :(

@bagder
Copy link
Member

bagder commented Jun 8, 2020

That was the second of the two commits, and the first one was the one that fixed the problem for me. You should rather merge both of them!

@bagder
Copy link
Member

bagder commented Jun 8, 2020

I rebased the PR and squashed the two commits into a single one. Try that instead!

bagder added a commit that referenced this issue Jun 8, 2020
The SOCKS4/5 state machines weren't properly terminated when the proxy
connection got closed, leading to a busy-loop.

Reported-By: zloi-user on github
Fixes #5532
@zloi-user
Copy link
Author

compiled 7.69.1 with two patches
1,2

cheers, the problem does not occur :)

@jay
Copy link
Member

jay commented Jun 8, 2020

It is system agnostic and happens when the SOCKS proxy closes the connection while curl runs the SOCKS state machine. See my proposed fix in #5542.

Both Windows and Linux recv return 0 repeatedly but then Windows after a second will return -1 WSAECONNABORTED which is why it doesn't busy loop.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging a pull request may close this issue.

4 participants