cURL / Mailing Lists / curl-users / Single Mail

curl-users

Re: Scripting curl from the inside out (Re: curl the next few years)

From: Von Hawkins <vonhawkins_at_yahoo.com>
Date: Fri, 20 Jun 2014 05:22:13 -0700

>I have been toying with the idea of an option which would allow control of  >curl via scripting *between* multiple requests, allowing a single curl >instance to perform multiple transfers to one or more hosts.  Because curl >would not die between commands, connections can stay open, DNS caches can be >maintained, etc., making the repeated accesses more efficient. >Imagine a "--exec <program>" option which would invoke a program (script) at >the completion of a command.  Curl would pass the just completed command >string, the result and other useful info to the program as arguments and/or >environment variables.  The program would check the result of the curl run, >optionally operate on any data and then communicate back to curl what it >wants to do next (if anything) by sending commands back to curl through a >pipe. I would LOVE this feature.  The way I use curl in bash with 2-way TLS requires ~4 seconds per transfer because of all the handshake/auth time.  eliminating that would give me (I think) more than a 4-fold speed increase.  Overnight tasks could drop to 1 hour tasks. Right now I care little how it happens, just any way to keep the connections open for future posts while scripting around curl would be great.  I process in excess of 100,000 transactions at a time and 4 seconds per is mind numbing.  (which is probably why I should learn to program instead of scripting everything.) --Von

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-users
FAQ: http://curl.haxx.se/docs/faq.html
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2014-06-20