I downloaded a copy of Lynx (console-mode text-based browser) and I've got that working despite CBOE's purposeful complications. For future reference, you launch it and record a command script like this:

lynx -cmd_log=cboe_cmds.txt http://www.cboe.com/DelayedQuote/QuoteTableDownload.aspx

Interactively walk through the necessary keystrokes, and quit. Voila...command script that can then be replayed like so:

lynx -cmd_script=cboe_cmds.txt http://www.cboe.com/DelayedQuote/QuoteTableDownload.aspx

Launched with the command script, it will walk through the same commands, download the file, and exit. It isn't very quick nor elegant, but it gets the job done.

That said, I'd still rather use curl...come on, some of you experts step up to the plate and provide an example that handles CBOE's complexities! [Said with a smile, of course. :-P ]

Cheers,
Dave

---- dsisk_at_nc.rr.com wrote:
> Yes, does not appear to be easy at all. I'm wondering if it might just be easier to get option quotes from Yahoo...I would have to somehow isolate the HTML table containing the quote data and extract the data from that HTML table. Is there any utility similar to curl that will turn an HTML page with tables into a CSV file? ;-)
>
> Cheers!
> Dave
>
> ---- Doug McNutt <douglist_at_macnauchtan.com> wrote:
> > At 09:53 -0500 2/1/11, <dsisk_at_nc.rr.com> wrote:
> > >Actually, CBOE apparently doesn't want people running automated scripts to download option quotes in CSV format...they've made it very convoluted and difficult to do from anything except a browser. That said, if I can somehow figure out exactly what a browser is doing behind the scenes, then I should be able to make curl "trick" it into thinking it's just a browser (right?). I'm a DBA and not a developer, but I think if I could somehow trap what the browser is doing behind the scenes then I could make curl do the same thing. Not exactly sure how to "trap" that though.
> > >
> > >Any guidance here? This works easily with other sites (like Yahoo Finance for instance...but the only way to get an option chain from Yahoo is to return it as HTML, not CSV)...CBOE is a stubborn site!
> > >
> > >Dave
> >
> >
> > It may not be easy. I know of one brokerage that is so fierce about it that, during your login, an encrypted key is sent to a paid third site that responds to you with another encrypted key that you must send back to the original site. It has to match the key that is sent directly back without coming through you.
> >
> > I find that combining curl with perl works well. You can examine cookies and items within an html file to figure out how to send cookies that are not delivered in an http header. You can invoke multiple calls to cure from within perl with the backticks operator.
> >
> > Live http headers for Firefox helps a whole lot.
> >
> > --
> > --> Protons have mass? I didn't even know they were Catholic. <--
> > -------------------------------------------------------------------
> > List admin: http://cool.haxx.se/list/listinfo/curl-users
> > FAQ: http://curl.haxx.se/docs/faq.html
> > Etiquette: http://curl.haxx.se/mail/etiquette.html
>
> -------------------------------------------------------------------
> List admin: http://cool.haxx.se/list/listinfo/curl-users
> FAQ: http://curl.haxx.se/docs/faq.html
> Etiquette: http://curl.haxx.se/mail/etiquette.html

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2011-02-01