Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

--aws-sigv4 doesn't sign requests with query string parameters with no value correctly #8107

Closed
dzaslavskiy opened this issue Dec 6, 2021 · 6 comments
Closed

--aws-sigv4 doesn't sign requests with query string parameters with no value correctly #8107

dzaslavskiy opened this issue Dec 6, 2021 · 6 comments
Labels
authentication HTTP KNOWN_BUGS material

Comments

@dzaslavskiy
Copy link

dzaslavskiy commented Dec 6, 2021 •
edited

This request gives an error:

curl --user "<key>:<secret>" --aws-sigv4 aws:amz:<region>:es -k -X GET 'https://<es-host>/_cat/indices?v'

{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}

Removing the query string or or adding a value to the query parameter fixes the issue

curl --user "<key>:<secret>" --aws-sigv4 aws:amz:<region>:es -k -X GET 'https://<es-host>/_cat/indices'

green open index-name index-uuid 5 1       16    0 285.4kb 142.7kb

curl --user "<key>:<secret>" --aws-sigv4 aws:amz:<region>:es -k -X GET 'https://<es-host>/_cat/indices?v=true'

green open index-name index-uuid 5 1       16    0 285.4kb 142.7kb

curl/libcurl version

curl 7.79.1 (x86_64-w64-mingw32) libcurl/7.79.1 OpenSSL/1.1.1l (Schannel) zlib/1.2.11 brotli/1.0.9 zstd/1.5.0 libidn2/2.3.1 libssh2/1.10.0 nghttp2/1.45.1
Release-Date: 2021-09-22
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP zstd

operating system

windows 10; git-bash
@dzaslavskiy dzaslavskiy changed the title --aws-sigv4 doesn't sign requests with query strings correctly --aws-sigv4 doesn't sign requests with query string parameters with only a value correctly Dec 6, 2021
@bagder
Copy link
Member

bagder commented Dec 6, 2021

/cc @outscale-mgo

@dzaslavskiy dzaslavskiy changed the title --aws-sigv4 doesn't sign requests with query string parameters with only a value correctly --aws-sigv4 doesn't sign requests with query string parameters with no value correctly Dec 6, 2021
@outscale-mgo
Copy link
Contributor

I'm presently trying to find the bug, out of curiosity, which call exactly did you made (having the full URL could held debug I guess) ?

@dzaslavskiy
Copy link
Author

dzaslavskiy commented Dec 8, 2021 •
edited 

$ ./curl --user "..." --aws-sigv4 aws:amz:us-east-1:es -X GET "https://search-curl-test-yk7vggewbojld4psjkwqnjifry.us-east-1.es.amazonaws.com/_cat/indices?v=true"

health status index     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana_1 Q9Me-vJiQXGZwplpTsTxpg   1   0          1            0        5kb            5kb

$ ./curl --user "..." --aws-sigv4 aws:amz:us-east-1:es -X GET "https://search-curl-test-yk7vggewbojld4psjkwqnjifry.us-east-1.es.amazonaws.com/_cat/indices?v"

{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}

./curl --user "..." --aws-sigv4 aws:amz:us-east-1:es -X GET "https://search-curl-test-yk7vggewbojld4psjkwqnjifry.us-east-1.es.amazonaws.com/_cat/indices?v="

health status index     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana_1 Q9Me-vJiQXGZwplpTsTxpg   1   0          1            0        5kb            5kb
  1. d. in the aws docs mentions this case:

For each parameter, append the URI-encoded parameter name, followed by the equals sign character (=), followed by the URI-encoded parameter value. Use an empty string for parameters that have no value.

Seems like an equals sign is not being appended here.

EDIT: I think there may some other issues here. The query string is not being fully canonicalized. For example, the docs require sorting query parameters by field name in ascii order.

@outscale-mgo
Copy link
Contributor

outscale-mgo commented Dec 9, 2021 •
edited

thanks for the reply, I've made a Pull Request that should improve url encoding here: #7600
Note that presently the PR doesn't fix your issue, but should do so, so I'm going to rework the PR to handle this issue, and sort the parameters.

@outscale-mgo outscale-mgo mentioned this issue Dec 9, 2021
Closed
@outscale-mgo
Copy link
Contributor

just to let you know I've update the PR, would be nice if you test it, and tell us if it work with your use case.

@dzaslavskiy
Copy link
Author

I have done a few tests on that branch and it works great.

@bagder bagder closed this as completed in 8b12f9e Aug 6, 2023
ptitSeb pushed a commit to wasix-org/curl that referenced this issue Sep 25, 2023
@bagder @ptitSeb
TODO: add *5* entries for aws-sigv4
01ed001 
Closes curl#7559
Closes curl#8107
Closes curl#8810
Closes curl#9717
Closes curl#10129
hjmallon added a commit to hjmallon/curl that referenced this issue Nov 1, 2023
@hjmallon
http_aws_sigv4: canonicalise valueless query params
3fe4cb8 
* Fixes curl#8107
@hjmallon hjmallon mentioned this issue Nov 1, 2023
Closed
hjmallon added a commit to hjmallon/curl that referenced this issue Nov 1, 2023
@hjmallon
http_aws_sigv4: canonicalise valueless query params
af97e5f 
* Fixes curl#8107
hjmallon added a commit to hjmallon/curl that referenced this issue Nov 2, 2023
@hjmallon
http_aws_sigv4: canonicalise valueless query params
0dfbb4f 
* Fixes curl#8107
bagder pushed a commit that referenced this issue Nov 5, 2023
@hjmallon @bagder
http_aws_sigv4: canonicalise valueless query params
bbba69d 
Fixes #8107
Closes #12244
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
authentication HTTP KNOWN_BUGS material
Milestone
No milestone
Development

No branches or pull requests
4 participants
@bagder @dzaslavskiy @outscale-mgo and others