> Are you using the multi interface?
> If not, that would solve TONS of problems if set up properly (i.e.
> using curl_multi_select & stuff)

I am not using multi interface, as it all runs on a cron job that places one
request at a time, ~12 seconds apart, so I don't see how it would solve tons
of problems (besides,k I only have the 1 problem, and it's only on <1% of
sites)

> Is this huge list of options really necessary?

I doubt it, but I've tried a bunch of variations, and thats the settings
where I left off, and, as I said, it is working on most sites, it's just a
small few, and as browsers (and I know google) seem to have no trouble
connecting to these sites that I am having issues with, I took the approach
of trying to get as close to somethign acceptable as possible.

> Why are you trying to appear like the GoogleBot?
> Maybe some sites reject you because you don't have GoogleBot's IP!

Well, I've tried leaving this blank, or trying other strings, this one was
reccomended and seems to work ok otherwise, and isn't the problem with these
few sites.
As for why I picked it as the final one, as I am doing something similar to
indexing, and want to see whatever google would see, assuming they have
content handlers based on source, that's why, but if you want me to try a
specific set of different options/headers, I'll give anything a shot.

> And why are you forcing HTTP/1.0?

Um. because it was reccomended to me on php.net I think. Again, I'm happy to
use any set of opts that works, really just wondering specifically why I
would get this error on a select few sites that I can browse to in a regular
web browser just fine.

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-php
Received on 2009-06-18