Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: IPFS support in Curl
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Marcin Rataj via curl-library <curl-library_at_lists.haxx.se>
Date: Mon, 21 Feb 2022 15:15:03 +0100
On Mon, Feb 21, 2022 at 10:07 AM Daniel Stenberg via curl-library
<curl-library_at_lists.haxx.se> wrote:
Joining the thread, as I may help with fleshing out the details on
IPFS side from the user agent perspective.
> Using a scheme called 'IPFS' when in reality it is doing HTTPS
> is going to cause confusion. Are we adding IPFS support or are we not?
There are three ways a user agent can support resolution of ipfs:// addresses:
(A) Fetch from a trusted HTTP gateway. Payload integrity check happens
on the gateway. Note: HTTP consumers often run their own gateway on
localhost for better performance and to remove this need for trusting
a third-party gateway.
(B) Fetch from a "trustless" HTTP gateway: data is fetched
block-by-block or as a CAR
(https://ipld.io/specs/transport/car/carv1/), both being opaque byte
streams. Then, the deserialization and integrity verification happens
on the client.
(C) Full P2P node (integrity verification + libp2p stack with DHT/mDNS
and P2P transports).
I believe in case of curl (tool/library) we would be discussing
something between A and B.
See my proposal below.
> > The main purpose of putting it into the curl tool is ergonomics.
>
> I'm much easier to persuade adding this to the tool than to the library. To
> me, the tool seems like a reasonable layer to do "HTTPS URL translations" in
> rather than in the library.
Yes, this is sensible:
"A" is a thin porcelain that could be added to curl (tool). It could
be smart enough to prefer local IPFS Gateway if available, or allow
fetching without deserialization
"B" requires understanding of CAR, IPLD data structures like unixfs,
and choosing which specific codecs and hash functions to support,
which sounds like it belong to libcurl, but also means a feature creep
and way more work and discussions than "A"
Given the complexity of "B", a pragmatic approach is to do "A" in curl
(tool) first.
To add value ton top of regular HTTP GET, enable the user to skip
verification/deserialization made on gateway, and fetch a single raw
Block:
curl ipfs://cid -o file.jpg
curl ipfs://cid --raw-block | ipfs dag put --input-codec=raw --store-codec=raw
or a CAR (support on gateways is landing later this year), enabling
fetch and verification of entire directory trees:
(curl ipfs://cid --car | ipfs dag import) && ipfs get cid
curl ipfs://cid --car | ipfs-car --output path/to/write/to
This follows the philosophy of having one tool for a job: allow users
to download data via curl, and if they need to access or verify the
IPFS metadata, do that by piping raw data to some other tool.
Nice thing about this scope is removing the need for curl to implement
or care about data format: opt-in flag for fetching a raw block or CAR
could be implemented by tweaking CID when making the request, but if
we don't even want to implement CID encoder/decoder, we could add
support for opt-in via Accept (HTTP header).
> I'm also cautious about adding support for protocols where there is no
> established URL syntax defined. [..]
In regard to URL syntax, are you interested in ABNF form (RFC5234), or
something else?
We are working on gateway and addressing specs this year and could
improve things in this area.
Having an example of desired specificity would very useful here.
> Do I understand it correctly that these IPFS URLs are designed to work
> identically independently if the protocol is HTTPS-via-gaweway or the "real"
> IPFS protocol?
Correct. Both gateways and these ipfs:// addresses aim to maximize the
interop with external tools.
Behavior on the gateways informs how ipfs:// works.
Date: Mon, 21 Feb 2022 15:15:03 +0100
On Mon, Feb 21, 2022 at 10:07 AM Daniel Stenberg via curl-library
<curl-library_at_lists.haxx.se> wrote:
Joining the thread, as I may help with fleshing out the details on
IPFS side from the user agent perspective.
> Using a scheme called 'IPFS' when in reality it is doing HTTPS
> is going to cause confusion. Are we adding IPFS support or are we not?
There are three ways a user agent can support resolution of ipfs:// addresses:
(A) Fetch from a trusted HTTP gateway. Payload integrity check happens
on the gateway. Note: HTTP consumers often run their own gateway on
localhost for better performance and to remove this need for trusting
a third-party gateway.
(B) Fetch from a "trustless" HTTP gateway: data is fetched
block-by-block or as a CAR
(https://ipld.io/specs/transport/car/carv1/), both being opaque byte
streams. Then, the deserialization and integrity verification happens
on the client.
(C) Full P2P node (integrity verification + libp2p stack with DHT/mDNS
and P2P transports).
I believe in case of curl (tool/library) we would be discussing
something between A and B.
See my proposal below.
> > The main purpose of putting it into the curl tool is ergonomics.
>
> I'm much easier to persuade adding this to the tool than to the library. To
> me, the tool seems like a reasonable layer to do "HTTPS URL translations" in
> rather than in the library.
Yes, this is sensible:
"A" is a thin porcelain that could be added to curl (tool). It could
be smart enough to prefer local IPFS Gateway if available, or allow
fetching without deserialization
"B" requires understanding of CAR, IPLD data structures like unixfs,
and choosing which specific codecs and hash functions to support,
which sounds like it belong to libcurl, but also means a feature creep
and way more work and discussions than "A"
Given the complexity of "B", a pragmatic approach is to do "A" in curl
(tool) first.
To add value ton top of regular HTTP GET, enable the user to skip
verification/deserialization made on gateway, and fetch a single raw
Block:
curl ipfs://cid -o file.jpg
curl ipfs://cid --raw-block | ipfs dag put --input-codec=raw --store-codec=raw
or a CAR (support on gateways is landing later this year), enabling
fetch and verification of entire directory trees:
(curl ipfs://cid --car | ipfs dag import) && ipfs get cid
curl ipfs://cid --car | ipfs-car --output path/to/write/to
This follows the philosophy of having one tool for a job: allow users
to download data via curl, and if they need to access or verify the
IPFS metadata, do that by piping raw data to some other tool.
Nice thing about this scope is removing the need for curl to implement
or care about data format: opt-in flag for fetching a raw block or CAR
could be implemented by tweaking CID when making the request, but if
we don't even want to implement CID encoder/decoder, we could add
support for opt-in via Accept (HTTP header).
> I'm also cautious about adding support for protocols where there is no
> established URL syntax defined. [..]
In regard to URL syntax, are you interested in ABNF form (RFC5234), or
something else?
We are working on gateway and addressing specs this year and could
improve things in this area.
Having an example of desired specificity would very useful here.
> Do I understand it correctly that these IPFS URLs are designed to work
> identically independently if the protocol is HTTPS-via-gaweway or the "real"
> IPFS protocol?
Correct. Both gateways and these ipfs:// addresses aim to maximize the
interop with external tools.
Behavior on the gateways informs how ipfs:// works.
-- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.htmlReceived on 2022-02-21