Octets after `?` in gopher:// URL are interpreted as a query #3369

iamleot · 2018-12-12T13:41:17Z

According RFC 4266 a gopher:// URL is:

[...]
A Gopher URL takes the form:
  gopher://<host>:<port>/<gopher-path>
where is one of:
  <gophertype><selector>
  <gophertype><selector>%09<search>
  <gophertype><selector>%09<search>%09<gopher+_string>
[...]

is the Gopher selector string. In the Gopher protocol,
Gopher selector strings are a sequence of octets that may contain any
octets except 09 hexadecimal (US-ASCII HT or tab), 0A hexadecimal
(US-ASCII character LF), and 0D (US-ASCII character CR).
[...]

However, everything after a `?' is ignored.

I did this

I will denote with a S> the server, and C> the client:

S> % nc -l 7070
C> % curl 'gopher://localhost:7070/1/foo?bar'
S> /foo

S> % nc -l 7070
C> % curl 'gopher://localhost:7070/1/foo?'
S> /foo

I expected the following

S> % nc -l 7070
C> % curl 'gopher://localhost:7070/1/foo?bar'
S> /foo?bar

S> % nc -l 7070
C> % curl 'gopher://localhost:7070/1/foo?'
S> /foo?

No ? and (if any) no octets after the ? should be omitted.

curl/libcurl version

% curl -V
curl 7.63.0 (x86_64--netbsd) libcurl/7.63.0 OpenSSL/1.1.0i zlib/1.2.10 libidn2/2.0.5
Release-Date: 2018-12-12
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy

operating system

NetBSD

PS: Please note that AFAICS we have no way to distunguish between a:
gopher://host:port/path and gopher://host:port/path? and so I think
that gopher_do() in lib/gopher.c could not be adjusted to just take
the concatenation of data->state.up.path and data->state.up.query
(in both cases data->state.up.query seems NULL).
Of course, if you have any idea how to better implement that,
feel free to share it and I'll try to write a possible patch!

Thank you very much!

The text was updated successfully, but these errors were encountered:

iamleot · 2018-12-12T13:49:15Z

And, sorry, forgot to share!

ATM as a possible workaround instead of directly using ? in the URL all ? should be manually encoded to %3f.

bagder · 2018-12-12T14:16:41Z

Probably a regression since 46e1640, but not detected due lack of proper tests.

Do you feel like taking on writing a fix?

iamleot · 2018-12-12T14:31:50Z

Hello Daniel, Daniel Stenberg writes:

Probably a regression since 46e1640, but not detected due lack of proper tests. Do you feel like taking writing up a fix?

Sure, I'm happy to try to fix that! But, before doing that... Any idea how to best address the possible path and query ambiguity that I've described in PS? (or, how to include the query part but also do not omit possible trailng `?' (I think that from lib/gopher.c, at least by using the urlpieces (data.up) we can't distinguish that)). Thank you!

bagder · 2018-12-12T14:47:21Z

If we indeed need to know if '?' was present in the URL then we need to store that information when parsing it, which isn't done now. Probably by adding a boolean to the Curl_URL struct and getting the info for it here:

curl/lib/urlapi.c

Lines 812 to 814 in d8607da

    
           query = strchr(path, '?'); 
        
           if(query) 
        
             *query++ = 0;

Possibly a smoother approach is that we work around it and avoid extending that struct, so that if the URL is a gopher one, we store a zero length query.

infinnovation-dev · 2018-12-12T16:05:45Z

Surely URLs with no query and with an empty query are semantically different? If so, then it may be as simple as changing

if(query && query[0]) {
    u->query = strdup(query);

to

if(query) {
    u->query = strdup(query);

Similarly for fragment.

iamleot · 2018-12-12T16:16:16Z

The Infinnovation team writes:

Surely URLs with no query and with an empty query are semantically different? If so, then it may be as simple as changing ```c if(query && query[0]) { u->query = strdup(query); ``` to ```c if(query) { u->query = strdup(query); ``` Similarly for fragment.

Mostly yes (needs also another tiny further change), I will hopefully share patches that does something like that in the next minutes! (time to cleanly rebuild and rerun all the tests!) Thanks!

iamleot · 2018-12-12T16:57:04Z

Please note that I have not adjusted fragments handling similarly.

In my experience it is less common (and probably pretty rare!) to
appear in the selector and, unlike ?, I am not sure if it violates
RFC 3986 (that is quoted by RFC 4266) where:

[...]
3.5. Fragment
[...]
[...] Fragment identifier semantics are independent of the
URI scheme and thus cannot be redefined by scheme specifications.

[...]

Fragment identifiers have a special role in information retrieval
systems as the primary form of client-side indirect referencing,
allowing an author to specifically identify aspects of an existing
resource that are only indirectly provided by the resource owner. As
such, the fragment identifier is not used in the scheme-specific
processing of a URI; instead, the fragment identifier is separated
from the rest of the URI prior to a dereference, and thus the
identifying information within the fragment itself is dereferenced
solely by the user agent, regardless of the URI scheme. Although
this separate handling is often perceived to be a loss of
information, particularly for accurate redirection of references as
resources move over time, it also serves to prevent information
providers from denying reference authors the right to refer to
information within a resource selectively. Indirect referencing also
provides additional flexibility and extensibility to systems that use
URIs, as new media types are easier to define and deploy than new
schemes of identification.
[...]

IIUC the fragment part - unlike the query - should be not sent to
the server and only consumed solely by the user agent.

After the migration to URL API all octets in the selector after the first `?' were interpreted as query and accidentally discarded and not passed to the server. Add a gopherpath to always concatenate possible path and query URL pieces. Fixes #3369 Closes #3370

bagder added GOPHER URL labels Dec 12, 2018

iamleot mentioned this issue Dec 12, 2018

Permit to semantically distinguish URL with empty query (closes #3369) #3370

Closed

bagder closed this as completed in 305d25e Dec 13, 2018

lock bot locked as resolved and limited conversation to collaborators Mar 13, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Octets after `?` in gopher:// URL are interpreted as a query #3369

Octets after `?` in gopher:// URL are interpreted as a query #3369

iamleot commented Dec 12, 2018 •

edited

Loading

iamleot commented Dec 12, 2018

bagder commented Dec 12, 2018 •

edited

Loading

iamleot commented Dec 12, 2018 via email

bagder commented Dec 12, 2018

infinnovation-dev commented Dec 12, 2018

iamleot commented Dec 12, 2018 via email

iamleot commented Dec 12, 2018

Octets after ? in gopher:// URL are interpreted as a query #3369

Octets after ? in gopher:// URL are interpreted as a query #3369

Comments

iamleot commented Dec 12, 2018 • edited Loading

I did this

I expected the following

curl/libcurl version

operating system

iamleot commented Dec 12, 2018

bagder commented Dec 12, 2018 • edited Loading

iamleot commented Dec 12, 2018 via email

bagder commented Dec 12, 2018

infinnovation-dev commented Dec 12, 2018

iamleot commented Dec 12, 2018 via email

iamleot commented Dec 12, 2018

Octets after `?` in gopher:// URL are interpreted as a query #3369

Octets after `?` in gopher:// URL are interpreted as a query #3369

iamleot commented Dec 12, 2018 •

edited

Loading

bagder commented Dec 12, 2018 •

edited

Loading