Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

aws-sigv4 request signature does not match expected value #11794

Closed
jaw111 opened this issue Sep 4, 2023 · 6 comments
Closed

aws-sigv4 request signature does not match expected value #11794

jaw111 opened this issue Sep 4, 2023 · 6 comments
Labels
authentication HTTP

Comments

@jaw111
Copy link

jaw111 commented Sep 4, 2023

I did this

I'm trying to sign GET requests to the SPARQL endpoint of an Amazon Neptune cluster where query string parameters are used to pass information. If the parameter values contain spaces, forward slashes /, or colons :, then the signature calculated by curl does not match what the server expects.

An example request:

curl "https://${NEPTUNE_HOSTNAME}:8182/sparql/gsp" \
  --url-query "graph=urn:uuid:6e8bc430-9c3a-11d9-9669-0800200c9a66" \
  --aws-sigv4 "aws:amz:${AWS_REGION}:neptune-db" \
  --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
  --header "x-amz-security-token: ${AWS_SESSION_TOKEN}"

Gets a response like this:

{
  "code": "AccessDeniedException",
  "requestId": "96c53027-1a0c-96a1-5a8b-18a20bee9465",
  "detailedMessage": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."
}

I expected the following

The calculated signature should be correct.

curl/libcurl version

curl 8.2.1 (x86_64-w64-mingw32) libcurl/8.2.1 OpenSSL/3.1.2 (Schannel) zlib/1.3 brotli/1.1.0 zstd/1.5.5 WinIDN libssh2/1.11.0 nghttp2/1.55.1 ngtcp2/0.19.0 nghttp3/0.15.0
Release-Date: 2023-07-26
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI threadsafe UnixSockets zstd

operating system

Windows 64-bit
@bagder
Copy link
Member

bagder commented Sep 4, 2023 •
edited

Combined with #7559, it appears as if the list of "bad" characters is now *, : and /. I figure it might need some closer reading of the "specs" to figure out the complete list and how they should be treated correctly.

The 7559 problem is a documented known bug

@bagder
Copy link
Member

bagder commented Sep 5, 2023 •
edited

I suspect this is because the %HH-encoding curl does is using lowercase letters, and not uppercase as the "spec" seems to suggest it should use.

/cc @outscale-mgo

@bagder
Copy link
Member

bagder commented Sep 5, 2023

I'm pretty sure we need to "canonicalize" the URL query to make aws-sigv4 work proper. Because right now a URL like: https://example.com/?foo=-*/ is not even going to get those three last characters URL encoded.

@outscale-mgo
Copy link
Contributor

It's exactly that, the URL need to be encoded manually.

@jaw111
Copy link
Author

jaw111 commented Sep 5, 2023

OK, I tried manually encoding the parameter value and that works:

curl "https://${NEPTUNE_HOSTNAME}:8182/sparql/gsp?graph=urn%3Auuid%3A6e8bc430-9c3a-11d9-9669-0800200c9a66" \
  --aws-sigv4 "aws:amz:${AWS_REGION}:neptune-db" \
  --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
  --header "x-amz-security-token: ${AWS_SESSION_TOKEN}"

Though ideally using --url-query option should take care of the canonicalization/normalization.

RFC 3986 section 2.1, where this encoding is defined, states:

For consistency, URI producers and normalizers should use uppercase hexadecimal digits for all percent-encodings.

bagder added a commit that referenced this issue Sep 6, 2023
@bagder
http_aws_sigv4: canonicalize the query
f2c943a 
Percent encoding needs to be done using uppercase, and most
non-alphanumerical must be percent-encoded.

Fixes #11794
Reported-by: John Walker
@bagder bagder mentioned this issue Sep 6, 2023
Closed
@bagder
Copy link
Member

bagder commented Sep 6, 2023

#11806 is my attempt at a fix

bagder added a commit that referenced this issue Sep 7, 2023
@bagder
http_aws_sigv4: canonicalize the query
ed395e0 
Percent encoding needs to be done using uppercase, and most
non-alphanumerical must be percent-encoded.

Fixes #11794
Reported-by: John Walker
bagder added a commit that referenced this issue Sep 7, 2023
@bagder
http_aws_sigv4: canonicalize the query
99293f7 
Percent encoding needs to be done using uppercase, and most
non-alphanumerical must be percent-encoded.

Fixes #11794
Reported-by: John Walker
bagder added a commit that referenced this issue Sep 7, 2023
@bagder
http_aws_sigv4: canonicalize the query
85239e6 
Percent encoding needs to be done using uppercase, and most
non-alphanumerical must be percent-encoded.

Fixes #11794
Reported-by: John Walker
@bagder bagder closed this as completed in fc76a24 Sep 7, 2023
ptitSeb pushed a commit to wasix-org/curl that referenced this issue Sep 25, 2023
@bagder @ptitSeb
http_aws_sigv4: canonicalize the query
49fbf6d 
Percent encoding needs to be done using uppercase, and most
non-alphanumerical must be percent-encoded.

Fixes curl#11794
Reported-by: John Walker
Closes curl#11806
@jaw111 jaw111 mentioned this issue Mar 6, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
authentication HTTP
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

http_aws_sigv4: canonicalize the query curl/curl
3 participants
@bagder @jaw111 @outscale-mgo