At 09:53 -0500 2/1/11, <dsisk_at_nc.rr.com> wrote:
>Actually, CBOE apparently doesn't want people running automated scripts to download option quotes in CSV format...they've made it very convoluted and difficult to do from anything except a browser. That said, if I can somehow figure out exactly what a browser is doing behind the scenes, then I should be able to make curl "trick" it into thinking it's just a browser (right?). I'm a DBA and not a developer, but I think if I could somehow trap what the browser is doing behind the scenes then I could make curl do the same thing. Not exactly sure how to "trap" that though.
>
>Any guidance here? This works easily with other sites (like Yahoo Finance for instance...but the only way to get an option chain from Yahoo is to return it as HTML, not CSV)...CBOE is a stubborn site!
>
>Dave

It may not be easy. I know of one brokerage that is so fierce about it that, during your login, an encrypted key is sent to a paid third site that responds to you with another encrypted key that you must send back to the original site. It has to match the key that is sent directly back without coming through you.

I find that combining curl with perl works well. You can examine cookies and items within an html file to figure out how to send cookies that are not delivered in an http header. You can invoke multiple calls to cure from within perl with the backticks operator.

Live http headers for Firefox helps a whole lot.

-- 
--> Protons have mass? I didn't even know they were Catholic. <--
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ:        http://curl.haxx.se/docs/faq.html
Etiquette:  http://curl.haxx.se/mail/etiquette.html