-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I have looked around the Internet and I haven't found an answer. Maybe noone
has seen this, or maybe I don't know what to call it. I have a perl script
on a website that calls curl through a pipe and retrieves web pages. When
this script is under a load from many users, it acts inappropriately. The
top command shows that my cpu is using 98%+ of its power in I/O wait, over
100% of my cpu is being used (up to even 150%) and my server quits responding
to input. The userspace cpu usage is usually less than 15% when this
happens, and my server is usually under 30% userspace usage. In httpd.conf,
my MaxClients is 150, my MaxSpareServers is 20 and MaxSpareThreads is 75.
This is not the only website on the server, but it is the busiest.

In my script, I use an open(WEBFILE, "curl options url |"); to open the url.
Then, depending on what content type is requested, the script goes into one
of two loops:

  while ( $line = <WEBFILE> ) {
   $line = parse_line($line);

   $BYTES += length($line);
   print $line;
   select(undef, undef, undef, length($line)/51200);
  }

or the following...

  while ( $bytesread = read(WEBFILE, $data, 4096) ) {
   print $data;
   $BYTES += length($data);
   select(undef, undef, undef, 0.08);
  }

The select command is the documented way to cause a sleep in Perl with
granularity of smaller than a second. The final argument in the select
command is used to restrict the bandwidth to 50K per second. What could be
the problem here?

- -Joseph
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBsNv3mXZROF+EADURAuWRAJ9c4qcOon3Fi7hPSUw98bH+W3au7ACfZLMN
PXIZsItA0ZK163tWtJGSyKs=
=vpi0
-----END PGP SIGNATURE-----
Received on 2004-12-03