Above and beyond 32 protocols

From: Daniel Stenberg via curl-library <curl-library_at_lists.haxx.se>
Date: Mon, 13 Sep 2021 13:01:38 +0200 (CEST)

Hi team.

We added support for GOPHERS in late 2020. There's a new PR proposing support
for the ManageSieve protocol. We had a PR previously suggesting Gemini support
and the other day ICAP was brought up in a discussion. WebSockets is another
common one discussed.

I don't think it's crazy to imagine that we might add support for more
protocols going forward. Sooner or later.

There are of course a lot of requirements we should have before accepting new
protocols, but ignoring that, we need to make sure that we can handle a
growing number of protocols internally.

I think the libcurl architecure is pretty well designed by now to handle
different and more protocols. There is however at least one bottle neck that
we need to address: the CURLPROTO_ defines.

Let's back up and repeat some basics:

1. curl/libcurl can be and is built on systems without 64 bit (long
    long) support. A question is if this is a target audience we need to
    consider much going forward.

2. On Windows, a 'long' is just 32 bits.

These two points make it hard for us to use anything > 32 bits in a standard
API. We have types that *can* be 64 bit but they aren't necessarily that
everywhere due to (1) and they can't be 'long' due to (2).

Further, most numerical values we set in curl_easy_setopt() accepts a 'long'.
Thus limited to 32 bits to be univerally portable (2).

We offer the options CURLOPT_PROTOCOLS and CURLOPT_REDIR_PROTOCOLS that both
take a 'long' as a bitmask for a protocol combination.

Currently [*], we have 30 different protocol defintions in the public header.
(We "wasted" a few on RTMP variations that probably could've been avoided but
at the time 32 protocols felt ample and we didn't plan better.)

This means that **we only have two spare bits** in that bitmask, while at the
same time there's a distinct possibility that we might want to add more than
two protocols to curl in the future.

# Ways out of this dilemma?

I can think of two ways. Neither is without a cost.

A) If we drop the < 64 bit requirement (1), we can introduce
CURLOPT_PROTOCOLS_LARGE etc that use a (guaranteed) 64 bit type and we double
room for protocols, which should be enough for the forseeable future.

B) If we maintain the < 64 bit requirement (1), we would have to do something
like introducing a *second* protocol option for the next set of protocols. It
would be error-prone and cumbersome for users (it will appear semi-random as
to which protocols that appear in which bitmask etc) and also inconvenient to
handle internally. Perpaps one of the bits in the first bitmask should then
signal that there are bits set in the second one. Or perhaps that's not
necessary.

I think I personally prefer (A). I would imagine that the amount of libcurl
users without a working 64 bit type should be fairly low.

# Feedback

I'm all ears. Especially if you have alternative solutions to suggest or if
you have an opinion on which way to go.

This is not a problem we must solve *right now*, but I would feel better if we
have an idea about how to address it when we get there. Because I'm convinced
we will reach this point eventually.

[*] = https://github.com/curl/curl/blob/master/include/curl/curl.h#L996

-- 
  / daniel.haxx.se
  | Commercial curl support up to 24x7 is available!
  | Private help, bug fixes, support, ports, new features
  | https://curl.se/support.html
-- 
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Received on 2021-09-13