Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Fwd: Adding IPFS Trustless Gateway Protocol Questions
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Hugo Valtier via curl-library <curl-library_at_lists.haxx.se>
Date: Fri, 27 Oct 2023 05:35:30 +0200
> my understanding was that there were two
> main ways to get files over IPFS: one is to get them via HTTP from an IPFS
> gateway that knows about IPFS (what curl does now) and the other is to become a
> full-fledged node in the IPFS network and speak the IPFS protocols to the
> world.
We have multiple HTTP gateway specifications, currently curl use the IPFS
path gateway spec. In this spec curl receives the file in their non IPFS form:
Hash Validated
┌──────────────┐
│Host → Gateway│→ Curl
└──────────────┘
What I'll submit will use the IPFS trustless gateway spec, so curl will receive
the IPFS data structure, hash it, validate it and deserialize it:
Hash Validated
┌─────────────────────┐
│Host → Gateway → Curl│
└─────────────────────┘
> What you describe sounds like a third method, where one may somehow
> find a full IPFS node that happens to have your file and talk a subset of the
> IPFS protocol to get that file. Is that an accurate assessment?
Not exactly, in what I'll propose the server is still acting as a gateway,
so if they don't have the file cached they will download it from someone else.
I agree it's not a perfect solution because most public servers do not proxy
requests (only public gateways do), running private gateways require deploying
a new service.
And there is also the option to know OOB which server hosts the file you want,
but it is less convenient.
> If so, is that really a mode that would be used by a significant number of people?
I'm not gonna deny that there is a bubble effect going on but it's
pretty popular
within my own bubble.
Examples of the trustless gateway used in productions between IPFS nodes are:
- https://www.pinata.cloud/ (SaaS IPFS)
- https://saturn.tech/ (P2P cdn)
- https://web3.storage/ (SaaS IPFS)
- (more I don't know about on the top of my head)
Other IPFS implementations also supports it:
- https://github.com/filecoin-project/lassie (go cli, client)
- https://github.com/filecoin-project/boost (go daemon, server)
- https://github.com/ipfs/kubo (go daemon, server, client not yet)
- https://github.com/ipfs/helia (js browser and nodejs, client)
- https://github.com/little-bear-labs/ipfs-chromium (cpp blink runtime, client)
It is used for two big reasons:
1. IP transit costs are lower over HTTP than the same bytes over QUIC
or TCP, or so I have been told. (even if you comprare H3 vs QUIC)
I don't know how much of it is that CDNs had decades to optimize their HTTP
implementations vs how much that pricing models for CDNs are weird.
2. HTTP is also widely implemented everywhere, making developers' lives
easier when implementing IPFS features.
> How do you find an appropriate node for each file, for example?
The implementations I listed above either query IPNI (federated
indexers which map hashes to servers) and/or DHT (Distributed Index
which maps hashes to servers), our DHT is based on Kadamelia.
Having content routing in curl would allow to skip the gateway and talk
to content hosts directly.
I think it make more sense to figure out data validation before
trying to download content from strangers on the internet.
If that is even a thing curl wants to do.
Date: Fri, 27 Oct 2023 05:35:30 +0200
> my understanding was that there were two
> main ways to get files over IPFS: one is to get them via HTTP from an IPFS
> gateway that knows about IPFS (what curl does now) and the other is to become a
> full-fledged node in the IPFS network and speak the IPFS protocols to the
> world.
We have multiple HTTP gateway specifications, currently curl use the IPFS
path gateway spec. In this spec curl receives the file in their non IPFS form:
Hash Validated
┌──────────────┐
│Host → Gateway│→ Curl
└──────────────┘
What I'll submit will use the IPFS trustless gateway spec, so curl will receive
the IPFS data structure, hash it, validate it and deserialize it:
Hash Validated
┌─────────────────────┐
│Host → Gateway → Curl│
└─────────────────────┘
> What you describe sounds like a third method, where one may somehow
> find a full IPFS node that happens to have your file and talk a subset of the
> IPFS protocol to get that file. Is that an accurate assessment?
Not exactly, in what I'll propose the server is still acting as a gateway,
so if they don't have the file cached they will download it from someone else.
I agree it's not a perfect solution because most public servers do not proxy
requests (only public gateways do), running private gateways require deploying
a new service.
And there is also the option to know OOB which server hosts the file you want,
but it is less convenient.
> If so, is that really a mode that would be used by a significant number of people?
I'm not gonna deny that there is a bubble effect going on but it's
pretty popular
within my own bubble.
Examples of the trustless gateway used in productions between IPFS nodes are:
- https://www.pinata.cloud/ (SaaS IPFS)
- https://saturn.tech/ (P2P cdn)
- https://web3.storage/ (SaaS IPFS)
- (more I don't know about on the top of my head)
Other IPFS implementations also supports it:
- https://github.com/filecoin-project/lassie (go cli, client)
- https://github.com/filecoin-project/boost (go daemon, server)
- https://github.com/ipfs/kubo (go daemon, server, client not yet)
- https://github.com/ipfs/helia (js browser and nodejs, client)
- https://github.com/little-bear-labs/ipfs-chromium (cpp blink runtime, client)
It is used for two big reasons:
1. IP transit costs are lower over HTTP than the same bytes over QUIC
or TCP, or so I have been told. (even if you comprare H3 vs QUIC)
I don't know how much of it is that CDNs had decades to optimize their HTTP
implementations vs how much that pricing models for CDNs are weird.
2. HTTP is also widely implemented everywhere, making developers' lives
easier when implementing IPFS features.
> How do you find an appropriate node for each file, for example?
The implementations I listed above either query IPNI (federated
indexers which map hashes to servers) and/or DHT (Distributed Index
which maps hashes to servers), our DHT is based on Kadamelia.
Having content routing in curl would allow to skip the gateway and talk
to content hosts directly.
I think it make more sense to figure out data validation before
trying to download content from strangers on the internet.
If that is even a thing curl wants to do.
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2023-10-27