On Sun, 21 Jan 2001 Kuce35_at_aol.com wrote:

> I have a rather curious problem that curl keeps throwing at me. I'm
> trying to retrieve a set of webpages by sending them a cookie. Imagine a
> set of 15 pages with individual stock information on them that each
> require the *exact* same cookie to retrieve. pretty simple.. huh?

Sounds simple enough, yes.

> When I run the script, I get sporadic results. Occasionally, it will grab
> all the data fine. Other times, curl will return no output for all the
> pages. More frequently, however, curl will return no output for a couple
> of pages, but get the rest. There is no pattern as to which pages it
> doesn't return. I found that if I put a for loop after the curl command
> that iterates 500 times, curl gets better results.

Without knowing for sure, I do think this sounds as if the problem is not in
curl. To curl, there can be absolutely no difference to how fast you reinvoke
it. Curl doesn't care if you run it 1000 times/second or once each 15th
second. It does the same operation.

So if you get different results returned depending on how fast you run curl
or how long you wait between the invokes, it points to a problem that isn't
curl-related. If you're using a proxy/firewall between curl and the server,
that might be an explanation otherwise I can't see any other obvious point of
failure than the remote server itself.

> I'm sorry if this quesiton is ambigous. I'm at wits end and I don't know
> of a better way to describe because I can't find any pattern as to why it
> almost "chooses" which pages to return each time.

Of course there might be some "smart" application in the other end that does
some magic and decides to skip some of the requests if they're too rapid. I
can't possibly guess from this...

-- 
  Daniel Stenberg -- curl project maintainer -- http://curl.haxx.se/