Re: curl_easy_cmdline ?

From: Timothe Litt <litt_at_acm.org>
Date: Thu, 25 Mar 2021 14:07:14 -0400

I've been off-line, so this is a late reply.  Hopefully not too late.

On 03-Dec-20 17:38, Daniel Stenberg wrote:On Thu, 3 Dec 2020, Timothe
Litt via curl-library wrote:
>
> Thanks Timothe, you really took this further!
>
>>    int curl_easy_getopt(CURL *handle, CURLoption option, size_t
>>    bufsize, void *buffer)
>>
>>        Returns any settable option's current value in buffer, with the
>>        same type as curl_easy_setopt().  Returns CURLE_UNKNOWN_OPTION,
>>        CURLE_NOT_BUILT_IN, CURLE_OVERFLOW, or CURLE_OK.  OVERFLOW if
>>        buffer too small for value.
>
> I like the core of your thinking here, but I don't think exposing a
> function this "raw" for exporting the contents of a single option will
> do a lot of good and the API is much too crude and error-prone when
> people will try to extract the values from that buffer. And I would
> bet good money that people would.
>
This has other uses, and allows higher-level functions to be built from
it.  It's the best way to build an API.

I don't see why it's any more error-prone than curl_easy_setopt(). 
Returns the data as it was set by the user, modulo the standard "if you
don't know how much buffer space to allocate, try a size and increase it
until the return value fits".  This is pretty standard - used in many
APIs. 

Of course, the reason I included curl_easy_listopt() was to minimize the
issue, since it provides the type.

The other standard approach is to make bufsize an in-out; the caller
specifies the allocated size; the function returns the size of the item,
which can be larger than the buffer if CURLE_OVERFLOW.  E.g.

  int curl_easy_getopt(CURL *handle, CURLoption option, size_t
   *bufsize, void *buffer)

I actually prefer the second approach since for an unknown value, it
guarantees no more than 2 calls to get the value.  However, I went for
symmetry with curl_easy_setopt().  In most cases, the caller will have a
good idea of how big the return value is; it's only the generic
extraction code that would have to guess.

>>    curl_easy_write_handle_state(CURL *handle, FILE *fh), where fh is
>>    open in binary mode.
>
> This function however, seems much more like the golden middle-way
> worth exploring further.
>
> Extracting all the contents from an easy handle to a well-documented
> binary format would be much less code than the *cmdline() version, it
> would remove the libcurl-to-curl mapping that none of us were really
> comfortable with and it could rather allow a "binary-to-cmdline"
> tool/option to be created and to be fancy and without the restrictions
> a library function would have.
>
> I can also predict that binding authors and others would rather have
> the ability to get that "snapshot" put into a buffer in memory to
> avoid storing temp files...

My point was that this can be implemented in terms of curl_easy_getopt()
and listopt().

If you want a few values, use curl_easy_getopt() directly.  If you want
a snapshot, use a file.  You really can't use a buffer efficiently
because there are so many variable size items.  The library would either
have to compute and sum the size of every item & return that to the
caller, or work down the list item by item and return overflow as
encountered - perhaps multiple times.  Further, with async operations,
things can change.

You can make the fh be a pipe if you don't want to have a temp file.

>
>>    curl --gencmd statefile
>>
>>    curl --genxml statefile
>
> Right. And --gencmd combined with --libcurl would get the C program
> for it... =)

Sure, and --gencobol would write the COBOL program :=)

Seriously, once you have the state, you can do anything you like with
it.  Writing a gdb file might be interesting to some.  But what I
actually had in mind was that --gencmd was to satisfy your original
request, and let --genxml handle everything else.  Once the data is in
XML, generating gdb files, COBOL, C, statistics, web pages, or whatever
else you can dream of is a separate program.  Write it in Perl, Python,
C, whatever.  But it's never a burden on the curl tool or libcurl.

If I had my choice, it would ONLY be --genxml, and you could supply the
curlxml2cmd script/program as a separate tool.  Assuming the XML was
properly specified, consumers would only need to use a standard XML
parser to extract information.  These are available in pretty much any
programming language, and the file is portable.  So no worries about
network byte order or item sizes if you want to aggregate data from
multiple hosts, or simply process data elsewhere on the network.

But these would be minimally intrusive to both the curl library and
tool, while letting ambitious people access the data directly.

>
>>    This would make a pass over curl_easy_listopt to obtain the mapping
>>    of codes to names and types.
>
> Note that we already provide curl_easy_option_next(), which can
> iterate over all known easy options and return info about them. And
> curl_easy_option_by_id() to look up an option by its id. So a lot of
> pieces are already there...
>
That's news to me.  I just checked, and none of these are listed on
https://curl.se/libcurl/c/ today, although if I explicitly search for
them google will find their pages.

If they're meant to be public, they need to be in the public API
documentation...


>> The curl_easy_getopt function has other uses - e.g. it can eliminate
>> the requirement for an application to keep its own copy of dynamic
>> values.
>
> It can indeed, but to do that in a good way it would need to return
> the data in a way that the caller wants it and is unlikely to get wrong.
>
I believe that the API that I suggested does exactly that.  Both the
original and the alternate in the note above.

Timothe Litt
ACM Distinguished Engineer
--------------------------
This communication may not represent the ACM or my employer's views,
if any, on the matters discussed.





-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Received on 2021-03-25