Implementing IPFS support in cURL

From: Mark Gaiser via curl-library <curl-library_at_lists.haxx.se>
Date: Wed, 23 Mar 2022 17:38:50 +0100

Hi,

For reference see [1] from an attempt that tried to implement it in the
library but pulled back.

For a very short summary of how an implementation would look like. IPFS [2]
data can be accessed through gateways [3]. The intention here is to
translate an IPFS url into a http(s) gateway url. So for example:
- ipfs://QmbGtJg23skhvFmu9mJiePVByhfzu5rwo74MEkVDYAmF5T
becomes this format:
- <gateway>/ipfs/QmbGtJg23skhvFmu9mJiePVByhfzu5rwo74MEkVDYAmF5T
with an actual example being:
- https://dlink.web/ipfs/QmbGtJg23skhvFmu9mJiePVByhfzu5rwo74MEkVDYAmF5T

For further reference, when i say <cid> i mean
QmbGtJg23skhvFmu9mJiePVByhfzu5rwo74MEkVDYAmF5T

In cURL only this translation is going to end up and won't require any
additional libraries. Nothing from IPFS itself will be added.

The referenced PR [1] talks about modifying "url_proto" (tool_operate.c
line 662) in one of the last comments to implement it there. But if I look
at the cURL source I see no special conditions there at all. Therefore it
"feels" a bit dirty and hackish to implement IPFS support in that function.

I did some debugging and found that:
- seturl (urlapi.c line 805) doesn't understand "ipfs://<cid>" and
considers <cid> to be the "host".
- parseurl_and_replace calls parseurl (urlapi.c line 1136) and returns
CURLUE_OK with ipfs://, it should probably not but that does mean that a
function like "parseurl" is going to be more complex to let it fail for
"ipfs://".

An approach that could potentially work is:
1. Let parseurl fail with "ipfs://" (actually seturl, but called from
parseurl)
2. The function that calls parseurl (so parseurl_and_replace in urlapi.c on
line 1149) would get an "elseif" for IPFS and IPNS handling. It would do
the transforming of the URL and call parseurl again with the result.
3. Now everything should work

I'm not very familiar with the cURL codebase, this is in fact the first
time I look at it. It would be really helpful to know if:
1. The suggested approach would be acceptable as a PR?
2. If this is the correct place to modify cURL for this purpose. Or should
the library route be taken after all?
3. A potential patch would also include logic to try and detect the IPFS
gateway on a best effort basis which would be able to be set manually too.
See [4] for a more detailed description.

I'm looking forward to hearing your opinion!

Best regards,
Mark Gaiser

[1] https://github.com/curl/curl/pull/8468
[2] https://ipfs.io/
[3] https://docs.ipfs.io/concepts/ipfs-gateway/#overview
[4] https://github.com/curl/curl/pull/8468#issuecomment-1076279407

-- 
Unsubscribe: https://lists.haxx.se/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Received on 2022-03-23