Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Implementing IPFS support in cURL
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Mark Gaiser via curl-library <curl-library_at_lists.haxx.se>
Date: Thu, 24 Mar 2022 13:16:31 +0100
On Thu, Mar 24, 2022 at 10:09 AM Daniel Stenberg <daniel_at_haxx.se> wrote:
> On Thu, 24 Mar 2022, Mark Gaiser wrote:
>
> > I might have been wrong here. My thinking was that "ipfs://<cid>" should
> be
> > handled as:
> > protocol: ipfs
> > path: cid
> >
> > But this is a non issue if it's purely handled on the tool side.
>
> Ah right. If done within the library then it would posssibly have to get
> handled differently.
>
> >> I thought we agreed that this change would be commandline tool only?
> The
> >> tool code is all in src/.
> >
> > I wasn't aware that to be a decision, I viewed it as a mere hint of a
> > possible way to implement it but to investigate further to see if that's
> the
> > best approach.
>
> Ok, I'm sorry for not being clearer.
>
> I think for libcurl to support ipfs:// it should actually support IPFS -
> the
> protocol - and not just do a URL rewrite.
>
Like actually interacting with IPFS components like bitswap.
Yeah, let's park that on the "potential future but not now" list :)
Just an url rewrite with gateway detection for now would be awesome.
>
> In spite of what the subject of the email says, this work is not adding
> support for IPFS the way I see it. It adds support for "IPFS URL rewrite
> to
> HTTPS". I don't think the library needs such rewrite support.
> > So just to get the opinions aligned. Am i correct in assuming that all of
> > the IPFS logic needed for cURL, those being:
> > - Transform ipfs://<cid> to <gateway>/ipfs/<cid>
> > - Detecting which gateway to use
> > should be implemented purely in the tool code?
>
> That matches my view.
>
Great! Then I'll get working on a patch for this and make a (wip) PR on
github once I have it working.
> How is it detecting the gateway?
I've done this in the ffmpeg code too (still pending approval there). How
it works and what i would like to add in cURL too is:
- The cURL code will look at the environment variables:
| - IPFS_GATEWAY
| - IPFS_PATH
- IPFS_PATH by default is considered to be in ~/.ipfs. Whatever its value
is, we look for a file called "gateway" in that folder and use the first
line which will be the gateway.
- IPFS_GATEWAY, when set, will be used.
- When no path can be found and no gateway is set we fallback to dlink.web.
A note on dlink.web. That site is managed by Protocol Labs (the same people
who made IPFS).
For ffmpeg they approved that url to be used with also an "always works"
idea behind it.
I'm assuming they would approve that for cURL too.
The beauty of this, specifically that last fallback to dlink.web, is that
you get an "always works" feeling with IPFS.
Now people obviously should host their own node... but an "always works"
fallback scenario is worth a lot too!
>
> > If it does go in the tooling side then you will get at least one "out of
> > place" looking if where it checks for an ipfs and ipns protocol to do
> > something custom in that case. As far as i can tell there is no tooling
> code
> > currently that handles any other protocol in such a custom way. This is
> why
> > I'm asking for feedback on the approach here.
>
> You're on uncharted waters. We have not implemented any
> protocol-as-a-URL-rewrite before.
>
Awesome :)
>
> --
>
> / daniel.haxx.se
> | Commercial curl support up to 24x7 is available!
> | Private help, bug fixes, support, ports, new features
> | https://curl.se/support.html
>
Date: Thu, 24 Mar 2022 13:16:31 +0100
On Thu, Mar 24, 2022 at 10:09 AM Daniel Stenberg <daniel_at_haxx.se> wrote:
> On Thu, 24 Mar 2022, Mark Gaiser wrote:
>
> > I might have been wrong here. My thinking was that "ipfs://<cid>" should
> be
> > handled as:
> > protocol: ipfs
> > path: cid
> >
> > But this is a non issue if it's purely handled on the tool side.
>
> Ah right. If done within the library then it would posssibly have to get
> handled differently.
>
> >> I thought we agreed that this change would be commandline tool only?
> The
> >> tool code is all in src/.
> >
> > I wasn't aware that to be a decision, I viewed it as a mere hint of a
> > possible way to implement it but to investigate further to see if that's
> the
> > best approach.
>
> Ok, I'm sorry for not being clearer.
>
> I think for libcurl to support ipfs:// it should actually support IPFS -
> the
> protocol - and not just do a URL rewrite.
>
Like actually interacting with IPFS components like bitswap.
Yeah, let's park that on the "potential future but not now" list :)
Just an url rewrite with gateway detection for now would be awesome.
>
> In spite of what the subject of the email says, this work is not adding
> support for IPFS the way I see it. It adds support for "IPFS URL rewrite
> to
> HTTPS". I don't think the library needs such rewrite support.
> > So just to get the opinions aligned. Am i correct in assuming that all of
> > the IPFS logic needed for cURL, those being:
> > - Transform ipfs://<cid> to <gateway>/ipfs/<cid>
> > - Detecting which gateway to use
> > should be implemented purely in the tool code?
>
> That matches my view.
>
Great! Then I'll get working on a patch for this and make a (wip) PR on
github once I have it working.
> How is it detecting the gateway?
I've done this in the ffmpeg code too (still pending approval there). How
it works and what i would like to add in cURL too is:
- The cURL code will look at the environment variables:
| - IPFS_GATEWAY
| - IPFS_PATH
- IPFS_PATH by default is considered to be in ~/.ipfs. Whatever its value
is, we look for a file called "gateway" in that folder and use the first
line which will be the gateway.
- IPFS_GATEWAY, when set, will be used.
- When no path can be found and no gateway is set we fallback to dlink.web.
A note on dlink.web. That site is managed by Protocol Labs (the same people
who made IPFS).
For ffmpeg they approved that url to be used with also an "always works"
idea behind it.
I'm assuming they would approve that for cURL too.
The beauty of this, specifically that last fallback to dlink.web, is that
you get an "always works" feeling with IPFS.
Now people obviously should host their own node... but an "always works"
fallback scenario is worth a lot too!
>
> > If it does go in the tooling side then you will get at least one "out of
> > place" looking if where it checks for an ipfs and ipns protocol to do
> > something custom in that case. As far as i can tell there is no tooling
> code
> > currently that handles any other protocol in such a custom way. This is
> why
> > I'm asking for feedback on the approach here.
>
> You're on uncharted waters. We have not implemented any
> protocol-as-a-URL-rewrite before.
>
Awesome :)
>
> --
>
> / daniel.haxx.se
> | Commercial curl support up to 24x7 is available!
> | Private help, bug fixes, support, ports, new features
> | https://curl.se/support.html
>
-- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.htmlReceived on 2022-03-24