SMTP Auth is not using XOAUTH2 even when specified with --login-options 'AUTH=XOAUTH2' \ #10259

ndevln · 2023-01-08T13:09:51Z

I did this

I want to send an E-Mail over SMTP using OAUTH2.

curl --ssl-reqd -v \
--url 'smtp://smtp.gmail.com' \
--user $USERNAME \
--login-options 'AUTH=XOAUTH2' \
--oauth2-bearer $ACCESSTOKEN \
--mail-from from@example.org \
--mail-rcpt to@example.org \
--upload-file mail.txt

For GMail this fails with

< 250-smtp.gmail.com at your service, [2001:9e8:3c3:7c00:8d8:c156:791a:89a]
< 250-SIZE 35882577
< 250-8BITMIME
< 250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH
< 250-ENHANCEDSTATUSCODES
< 250-PIPELINING
< 250-CHUNKING
< 250 SMTPUTF8
} [5 bytes data]
> AUTH OAUTHBEARER
{ [5 bytes data]
< 334
} [5 bytes data]
> TOKEN
{ [5 bytes data]
< 334 eyJzdGF0dXMiOiJpbnZhbGlkX3JlcXVlc3QiLCJzY29wZSI6Imh0dHBzOi8vbWFpbC5nb29nbGUuY29tLyJ9 -> {"status":"invalid_request","scope":"https://mail.google.com/"}
} [5 bytes data]
> AQ==
{ [5 bytes data]
< 535-5.7.8 Username and Password not accepted. Learn more at
< 535 5.7.8  https://support.google.com/mail/?p=BadCredentials b6-20020aa7d486000000b0048447efe3fcsm2477734edr.84 - gsmt
curl: (67) Login denied

The bearer token is sent with AUTH OAUTHBEARER and formatted according to the format on line 72:

curl/lib/vauth/oauth2.c

Lines 68 to 73 in 5a9a5e1

    
           if(port == 0 || port == 80) 
        
             oauth = aprintf("n,a=%s,\1host=%s\1auth=Bearer %s\1\1", user, host, 
        
                             bearer); 
        
           else 
        
             oauth = aprintf("n,a=%s,\1host=%s\1port=%ld\1auth=Bearer %s\1\1", user, 
        
                             host, port, bearer);

Which should be supported according to: < 250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH
But since I specified --login-options 'AUTH=XOAUTH2' this method should be used.

I expected the following

According to Google the bearer token should be sent with AUTH XOAUTH2
https://developers.google.com/gmail/imap/xoauth2-protocol

And this token format should be used:

curl/lib/vauth/oauth2.c

Lines 95 to 100 in 5a9a5e1

    
           CURLcode Curl_auth_create_xoauth_bearer_message(const char *user, 
        
                                                          const char *bearer, 
        
                                                          struct bufref *out) 
        
           { 
        
             /* Generate the message */ 
        
             char *xoauth = aprintf("user=%s\1auth=Bearer %s\1\1", user, bearer);

Microsoft describes the same standard.
https://learn.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth
According to this site these are the main provider using SASL XOAUTH2 as the authentication mechanism:
https://mailtrap.io/blog/smtp-auth/

Since I never got it working, I don't know if this is the reason for the login the problem. But curl should use the specified login mechanism.

Thank you for all your work.

curl/libcurl version

$ curl -V
curl 7.86.0 (x86_64-w64-mingw32) libcurl/7.86.0 OpenSSL/1.1.1s (Schannel) zlib/1.2.13 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.3 libssh2/1.10.0 nghttp2/1.51.0
Release-Date: 2022-10-26
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI threadsafe TLS-SRP UnixSockets zstd

AND

$version
[1] "7.64.1"

$ssl_version
[1] "(OpenSSL/1.1.1k) Schannel"

$libz_version
[1] "1.2.12"

$libssh_version
[1] "libssh2/1.9.0"

$libidn_version
[1] NA

$host
[1] "x86_64-w64-mingw32"

$protocols
 [1] "dict"   "file"   "ftp"    "ftps"   "gopher" "http"   "https"  "imap"   "imaps"  "ldap"   "ldaps"  "pop3"   "pop3s" 
[14] "rtsp"   "scp"    "sftp"   "smtp"   "smtps"  "telnet" "tftp"  

$ipv6
[1] TRUE

$http2
[1] FALSE

$idn
[1] TRUE

operating system

Window 10 22H2 Build 19045.2364

The text was updated successfully, but these errors were encountered:

polarathene · 2024-01-24T01:22:11Z

UPDATE: See my follow-up comment.

Bug introduced in the 7.80.0 release from Nov 2021
Bug is still present in the latest Dec 2023 8.5.0 release.

Compared to the reporter @ndevln they have cited two versions of curl they tested? 7.64.1 and 7.86.0

While 7.81.0 and 7.88.1 aren't working for me (XOAUTH2 is treated as OAUTHBEARER), I have found curl 7.74.0 does work correctly as shown below.

This is a weird one...

I have no issue with XOAUTH2 with curl from Debian 11 Bullseye, but have noticed newer versions of curl mistakenly use OAUTHBEARER on Debian 12 Bookworm and WSL2, however all these versions of curl are newer than was reported here, so something else must be contributing to that? 🤔

You can reproduce a working example I shared here, but use curl within the container.

This is running on Windows 10 23H2 Build 22631.3007, via WSL2 (Ubuntu):

# Start the `compose.yaml` example:
$ docker compose up -d

# Correctly uses XOAUTH2:
$ docker exec -it dms-mail \
  curl -sv --url 'smtp://localhost:587' \
  --login-options 'AUTH=XOAUTH2' --oauth2-bearer DMS_YWNjZXNzX3Rva2Vu --user john.doe@example.test \
  --mail-from john.doe@example.test --mail-rcpt jane.doe@example.test --upload-file - <<< 'Hello Jane!'

*   Trying 127.0.0.1:587...
* Connected to localhost (127.0.0.1) port 587 (#0)
< 220 mail.example.test ESMTP
> EHLO mail
< 250-mail.example.test
< 250-PIPELINING
< 250-SIZE 10240000
< 250-ETRN
< 250-AUTH PLAIN LOGIN OAUTHBEARER XOAUTH2
< 250-AUTH=PLAIN LOGIN OAUTHBEARER XOAUTH2
< 250-ENHANCEDSTATUSCODES
< 250-8BITMIME
< 250-DSN
< 250 CHUNKING
> AUTH XOAUTH2
< 334
> dXNlcj1qb2huLmRvZUBleGFtcGxlLnRlc3QBYXV0aD1CZWFyZXIgRE1TX1lXTmpaWE56WDNSdmEyVnUBAQ==
< 235 2.7.0 Authentication successful
> MAIL FROM:<john.doe@example.test>
< 250 2.1.0 Ok
> RCPT TO:<jane.doe@example.test>
< 250 2.1.5 Ok
> DATA
< 354 End data with <CR><LF>.<CR><LF>
} [12 bytes data]
< 521 5.5.2 mail.example.test Error: bare <LF> received
* Connection #0 to host localhost left intact

# Postfix log:
$ docker logs dms-mail | grep 'postfix/submission/smtpd.*sasl_method'
Jan 24 00:01:40 mail postfix/submission/smtpd[1011]: 03A4B1EAEC: client=localhost[127.0.0.1], sasl_method=XOAUTH2, sasl_username=john.doe@example.test

# Curl version info:
$ curl -V
curl 7.74.0 (x86_64-pc-linux-gnu) libcurl/7.74.0 OpenSSL/1.1.1w zlib/1.2.11 brotli/1.0.9 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.9.0 nghttp2/1.43.0 librtmp/2.3
Release-Date: 2020-12-09, security patched: 7.74.0-1.3+deb11u11
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

Debian 12 / WSL2 Ubuntu curl versions:

# WSL2
$ curl -V
curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.14
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd


# Debian 12:
$ docker run --rm -it debian:12-slim bash
$ apt-get update && apt-get install curl

$ curl -V
curl 7.88.1 (x86_64-pc-linux-gnu) libcurl/7.88.1 OpenSSL/3.0.11 zlib/1.2.13 brotli/1.0.9 zstd/1.5.4 libidn2/2.3.3 libpsl/0.21.2 (+libidn2/2.3.3) libssh2/1.10.0 nghttp2/1.52.0 librtmp/2.3 OpenLDAP/2.5.13
Release-Date: 2023-02-20, security patched: 7.88.1-10+deb12u5
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Same happens with IMAP auth:

$ curl -sv --url 'imap://localhost:143' \
  --login-options 'AUTH=XOAUTH2' --oauth2-bearer DMS_YWNjZXNzX3Rva2Vu --user john.doe@example.test \
  -X LOGOUT

*   Trying 127.0.0.1:143...
* Connected to localhost (127.0.0.1) port 143 (#0)
< * OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN AUTH=LOGIN AUTH=OAUTHBEARER AUTH=XOAUTH2] Dovecot ready.
> A001 CAPABILITY
< * CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN AUTH=LOGIN AUTH=OAUTHBEARER AUTH=XOAUTH2
< A001 OK Pre-login capabilities listed, post-login capabilities have more.
> A002 AUTHENTICATE OAUTHBEARER bixhPWpvaG4uZG9lQGV4YW1wbGUudGVzdCwBaG9zdD1sb2NhbGhvc3QBcG9ydD0xNDMBYXV0aD1CZWFyZXIgRE1TX1lXTmpaWE56WDNSdmEyVnUBAQ==
< * CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE QUOTA
< A002 OK Logged in
> A003 LOGOUT
< * BYE Logging out
< A003 OK Logout completed (0.001 + 0.000 secs).
* Connection #0 to host localhost left intact

# Dovecot log:
$ docker logs dms-mail | grep 'dovecot: imap-login: Login:.*method='
Jan 24 00:39:26 mail dovecot: imap-login: Login: user=<john.doe@example.test>, method=OAUTHBEARER, rip=172.19.0.1, lip=172.19.0.4, mpid=4539, session=<jkaKS6YPQsisEwAB>

There is nothing special about the Debian 11 container environment, I can reproduce the correct behaviour via a plain Debian 11 container with curl installed there too.

polarathene · 2024-01-24T01:22:43Z

Taking Debian out of the mix, there is versioned curl containers available on DockerHub (which saves me from building/installing curl release binaries myself).

# Run curl  via a separate container out of the compose.yaml
# NOTE: It needs to be connected to the network (docker network ls) compose created for connectivity:
$ docker run --rm -it \
  --network the_implicit_compose_network \
  curlimages/curl:7.74.0 \
    -sv --url 'imap://mail.example.test:143' \
    --login-options 'AUTH=XOAUTH2' --oauth2-bearer DMS_YWNjZXNzX3Rva2Vu --user john.doe@example.test  \
    -X LOGOUT

This shows the failure was introduced after 7.79.1 with 7.80.0 release, but quite a lot changed and I don't really want to do a git bisect to build and test for the relevant commit 😓

At a guess, perhaps this change had unintended consequences (no other commits in that range have an oauth / sasl keyword match): #6930

curl/lib/curl_sasl.c

Lines 216 to 217 in 2620aa9

    
           if(auth & CURLAUTH_BEARER) 
        
             sasl->prefmech |= SASL_MECH_OAUTHBEARER | SASL_MECH_XOAUTH2;

curl/lib/curl_sasl.c

Line 363 in 2620aa9

enabledmechs = sasl->authmechs & sasl->prefmech;

Which somehow results in the OAUTHBEARER case being triggered (possibly only because it's first?):

curl/lib/curl_sasl.c

Lines 456 to 477 in 2620aa9

    
           if((enabledmechs & SASL_MECH_OAUTHBEARER) && oauth_bearer) { 
        
             mech = SASL_MECH_STRING_OAUTHBEARER; 
        
             state1 = SASL_OAUTH2; 
        
             state2 = SASL_OAUTH2_RESP; 
        
             sasl->authused = SASL_MECH_OAUTHBEARER; 
        
             if(force_ir || data->set.sasl_ir) 
        
               result = Curl_auth_create_oauth_bearer_message(conn->user, 
        
                                                              hostname, 
        
                                                              port, 
        
                                                              oauth_bearer, 
        
                                                              &resp); 
        
           } 
        
           else if((enabledmechs & SASL_MECH_XOAUTH2) && oauth_bearer) { 
        
             mech = SASL_MECH_STRING_XOAUTH2; 
        
             state1 = SASL_OAUTH2; 
        
             sasl->authused = SASL_MECH_XOAUTH2; 
        
             if(force_ir || data->set.sasl_ir) 
        
               result = Curl_auth_create_xoauth_bearer_message(conn->user, 
        
                                                               oauth_bearer, 
        
                                                               &resp);

curl/lib/curl_sasl.c

Lines 681 to 697 in 2620aa9

    
           case SASL_OAUTH2: 
        
             /* Create the authorization message */ 
        
             if(sasl->authused == SASL_MECH_OAUTHBEARER) { 
        
               result = Curl_auth_create_oauth_bearer_message(conn->user, 
        
                                                              hostname, 
        
                                                              port, 
        
                                                              oauth_bearer, 
        
                                                              &resp); 
        
               /* Failures maybe sent by the server as continuations for OAUTHBEARER */ 
        
               newstate = SASL_OAUTH2_RESP; 
        
             } 
        
             else 
        
               result = Curl_auth_create_xoauth_bearer_message(conn->user, 
        
                                                               oauth_bearer, 
        
                                                               &resp); 
        
             break;

jay · 2024-01-25T08:42:15Z

At a guess, perhaps this change had unintended consequences (no other commits in that range have an oauth / sasl keyword match): #6930

@monnerat

monnerat · 2024-01-25T11:09:00Z

At a guess, perhaps this change had unintended consequences

Yes, I found it: a part of this commit maps given http authentication options to sasl ones and this works fine in the library. However the cli tool sets the http login option CURLAUTH_BEARER as a side-effect of --oauth2-bearer, causing the current problem.

I suggest to not preset anymore the sasl flags with http ones, but rather use them as a default, i.e. use them only if no login options has been specified.

If this solution is satisfactory, I can have a PR for it very soon.

If some are given, sasl maps http authentication options to sasl ones and merges them to the login options. This may cause problems with the cli tool that sets the http login option CURLAUTH_BEARER as a side-effect of --oauth2-bearer, because this flag maps to more than one sasl mechanisms and the latter cannot be cleared individually by the login options string. The solution retained here is to consider http authentication options as a default (overriding the hardcoded default mask for the protocol) that is ignored if a login option string is given. New test 992 checks this. Fixes curl#10259

polarathene · 2024-01-26T00:34:17Z

If this solution is satisfactory, I can have a PR for it very soon.

Below should provide an easy to test reproduction environment.

One command to get it up and running, then test via local curl build like shown below should verify correct functionality? (at least regarding with mail servers?)

Reproduction - Commands

# Start the `compose.yaml` example:
$ docker compose up -d

# Verify correct auth method is chosen:
$ curl -sv --url 'imap://localhost:143' \
    --login-options 'AUTH=XOAUTH2' --oauth2-bearer DMS_YWNjZXNzX3Rva2Vu --user john.doe@example.test  \
    -X LOGOUT \
    | grep 'AUTHENTICATE XOAUTH2'

$ curl -sv --url 'smtp://localhost:587' \
  --login-options 'AUTH=XOAUTH2' --oauth2-bearer DMS_YWNjZXNzX3Rva2Vu --user john.doe@example.test \
  --mail-from john.doe@example.test --mail-rcpt jane.doe@example.test --upload-file - <<< 'Hello Jane! \
   | grep 'AUTH XOAUTH2'

# Repeat for OAUTHBEARER instead of XOAUTH2

Reproduction - Docker Compose config

# Normally this would provide TLS configured for both services, that's been omitted to keep the reproduction simple
services:
  # Quick and easy mailserver setup with Postfix + Dovecot for testing OAuth2 support
  dms:
    image: docker.io/mailserver/docker-mailserver:13.3
    container_name: dms-mail
    hostname: mail.example.test
    environment:
      # Enable the OAuth2 support in Dovecot and configure it for our mocked service (caddy):
      ENABLE_OAUTH2: 1
      OAUTH2_INTROSPECTION_URL: http://auth.example.test/userinfo/
    # Test authentication against these ports:
    ports:
      - "143:143" # IMAP STARTTLS (Dovecot)
      - "587:587" # SMTP STARTTLS (Postfix)
    configs:
      - source: dms-accounts
        target: /tmp/docker-mailserver/postfix-accounts.cf

  # This would normally be a proper auth service, this is sufficient to mock out the required behaviour for testing
  caddy-oauth2:
    image: caddy:2.7
    container_name: dms-oauth2
    # Leverage Docker's internal DNS for the private network bridge it creates between services:
    hostname: auth.example.test
    ports:
      - "80:80"
    configs:
      - source: mock-auth-service
        target: /etc/caddy/Caddyfile

# Using the Docker Compose `configs.content` feature instead of volume mounting separate files.
# NOTE: This feature requires Docker Compose v2.23.1 (Nov 2023) or newer:
# https://github.com/compose-spec/compose-spec/pull/446
configs:
  # Basic Caddyfile example, see a better documented equivalent at:
  # https://github.com/docker-mailserver/docker-mailserver/blob/v13.3.0/test/config/oauth2/Caddyfile
  mock-auth-service:
    content: |
      :80 {
        @auth header Authorization "Bearer DMS_YWNjZXNzX3Rva2Vu"

        handle @auth {
          respond `{ "email": "john.doe@example.test", "email_verified": true }`
        }
        # Otherwise fail when expected auth header and value were not matched:
        respond 401 {
          close
        }
      }

  # DMS expects an account to be configured to run, this config provides one
  # You can add new accounts with `docker compose exec dms setup email add user@example.test bad-password`
  # Login credentials:
  # user: "john.doe@example.test" password: "secret"
  # user: "jane.doe@example.test" password: "secret"
  dms-accounts:
    # NOTE: `$` needed to be repeated to escape it,
    # which opts out of the `compose.yaml` variable interpolation feature.
    content: |
      john.doe@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
      jane.doe@example.test|{SHA512-CRYPT}$$6$$o65y1ZXC4ooOPLwZ$$7TF1nYowEtNJpH6BwJBgdj2pPAxaCvhIKQA6ww5zdHm/AA7aemY9eoHC91DOgYNaKj1HLxSeWNDdvrp6mbtUY.

They reported and investigated #10259 which was fixed by 7b2d98d. Ref: #10259

bagder added the SMTP label Jan 8, 2023

bagder added the KNOWN_BUGS material label Jul 31, 2023

polarathene mentioned this issue Jan 24, 2024

general: update base image to Debian 12 ("Bookworm") docker-mailserver/docker-mailserver#3403

Merged

8 tasks

monnerat mentioned this issue Jan 25, 2024

sasl: use http authentication mechanisms as a default, not a preset #12790

Closed

jay closed this as completed in 7b2d98d Jan 26, 2024

jay removed the KNOWN_BUGS material label Jan 27, 2024

jay added a commit that referenced this issue Jan 27, 2024

THANKS: add Alexander Bartel and Brennan Kinney

81d2b87

They reported and investigated #10259 which was fixed by 7b2d98d. Ref: #10259

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SMTP Auth is not using XOAUTH2 even when specified with --login-options 'AUTH=XOAUTH2' \ #10259

SMTP Auth is not using XOAUTH2 even when specified with --login-options 'AUTH=XOAUTH2' \ #10259

ndevln commented Jan 8, 2023 •

edited

polarathene commented Jan 24, 2024 •

edited

polarathene commented Jan 24, 2024 •

edited

jay commented Jan 25, 2024

monnerat commented Jan 25, 2024

polarathene commented Jan 26, 2024

SMTP Auth is not using XOAUTH2 even when specified with --login-options 'AUTH=XOAUTH2' \ #10259

SMTP Auth is not using XOAUTH2 even when specified with --login-options 'AUTH=XOAUTH2' \ #10259

Comments

ndevln commented Jan 8, 2023 • edited

I did this

I expected the following

curl/libcurl version

operating system

polarathene commented Jan 24, 2024 • edited

polarathene commented Jan 24, 2024 • edited

jay commented Jan 25, 2024

monnerat commented Jan 25, 2024

polarathene commented Jan 26, 2024

Reproduction - Commands

Reproduction - Docker Compose config

ndevln commented Jan 8, 2023 •

edited

polarathene commented Jan 24, 2024 •

edited

polarathene commented Jan 24, 2024 •

edited