curl-users
Re: Scripting curl from the inside out (Re: curl the next few years)
Date: Fri, 20 Jun 2014 05:22:13 -0700
>I have been toying with the idea of an option which would allow control of
>curl via scripting *between* multiple requests, allowing a single curl
>instance to perform multiple transfers to one or more hosts. Because curl
>would not die between commands, connections can stay open, DNS caches can be
>maintained, etc., making the repeated accesses more efficient.
>Imagine a "--exec <program>" option which would invoke a program (script) at
>the completion of a command. Curl would pass the just completed command
>string, the result and other useful info to the program as arguments and/or
>environment variables. The program would check the result of the curl run,
>optionally operate on any data and then communicate back to curl what it
>wants to do next (if anything) by sending commands back to curl through a
>pipe.
I would LOVE this feature. The way I use curl in bash with 2-way TLS requires ~4 seconds per transfer because of all the handshake/auth time. eliminating that would give me (I think) more than a 4-fold speed increase. Overnight tasks could drop to 1 hour tasks.
Right now I care little how it happens, just any way to keep the connections open for future posts while scripting around curl would be great. I process in excess of 100,000 transactions at a time and 4 seconds per is mind numbing. (which is probably why I should learn to program instead of scripting everything.)
--Von
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2014-06-20