Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

curl-7.63.0 not sending POST body when using --negotiate #3384

Closed
cstivers78 opened this issue Dec 17, 2018 · 10 comments
Closed

curl-7.63.0 not sending POST body when using --negotiate #3384

cstivers78 opened this issue Dec 17, 2018 · 10 comments

Comments

@cstivers78
Copy link

I did this

We use kerberos (GSSAPI/SPNEGO) for authenticating against web services. It seems something broke in 7.63.0, where POST bodies are not being transmitted when --negotiate is being used.

Example command use:

curl --negotiate -u ':' -H "Content-Type: application/json" -d '{"audiences":["people"]}' -XPOST https://some.api --trace-ascii trace.log

When I checked the trace.log, I saw:

785: User-Agent: curl/7.63.0
179e: Accept: */*
17ab: Content-Type: application/json
17cb: Content-Length: 0
17de:
=> Send SSL data, 5 bytes (0x5)
0000: ....!
<= Recv SSL data, 5 bytes (0x5)
0000: .....
<= Recv header, 13 bytes (0xd)
0000: HTTP/2 400
<= Recv header, 37 bytes (0x25)
0000: date: Mon, 17 Dec 2018 19:41:10 GMT
<= Recv header, 41 bytes (0x29)
0000: content-type: text/plain; charset=utf-8
<= Recv header, 20 bytes (0x14)
0000: content-length: 49
<= Recv header, 62 bytes (0x3e)
0000: www-authenticate: Negotiate <<OMITTED>>
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 49 bytes (0x31)
0000: {"status":400,"message":"Invalid request body."}.
<= Recv SSL data, 5 bytes (0x5)
0000: ....!

Notice Content-Length: 0 and No Body.

I expected the following

I expected to see the body transmitted, with a computed Content-Length header.

For example, the same command ran in curl-7.54.0 yields:

1761: User-Agent: curl/7.54.0
177a: Accept: */*
1787: Content-Type: application/json
17a7: Content-Length: 24
17bb:
=> Send data, 24 bytes (0x18)
0000: {"audiences":["people"]}
== Info: We are completely uploaded and fine
<= Recv header, 13 bytes (0xd)
0000: HTTP/2 200
<= Recv header, 37 bytes (0x25)
0000: date: Mon, 17 Dec 2018 19:34:06 GMT
<= Recv header, 47 bytes (0x2f)
0000: content-type: application/json; charset=utf-8
<= Recv header, 21 bytes (0x15)
0000: content-length: 750
<= Recv header, 62 bytes (0x3e)
0000: www-authenticate: Negotiate <<OMITTED>>
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 750 bytes (0x2ee)
0000:  <<OMITTED>>
== Info: Connection #0 to host some.api left intact

curl/libcurl version

[curl -V output]

curl --version
curl 7.63.0 (x86_64-pc-linux-gnu) libcurl/7.63.0 OpenSSL/1.1.1a zlib/1.2.11 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.4) libssh2/1.8.0 nghttp2/1.34.0
Release-Date: 2018-12-12
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

operating system

Occurrect on both Linux and Mac

@bagder
Copy link
Member

bagder commented Dec 17, 2018

Does it work if you use --anyauth instead of --negotiate ?

@gavinhungry
Copy link

@bagder,

Using --anyauth in place of --negotiate with 7.63.0 gives me the exact same results as described by @cstivers78 above.

@bagder
Copy link
Member

bagder commented Dec 17, 2018

But surely using --anyauth did send a body in the initial request and not a zero content-length one?

@bagder
Copy link
Member

bagder commented Dec 17, 2018

something broke in 7.63.0

Do you know the last version in which it worked?

@gavinhungry
Copy link

But surely using --anyauth did send a body in the initial request and not a zero content-length one?

@bagder: Yes, I do see a non-zero Content-Length for the initial request, followed by Content-Length: 0 after the server responds.

I'm on Linux, and 7.62.0 works.

@scottgreenup
Copy link

Can confirm with @gavinhungry , works in 7.62.0 on Arch Linux and the brew version of curl on macOS. Both of them break in 7.63.0.

@bagder
Copy link
Member

bagder commented Dec 18, 2018

There's only a single change done to lib/http_negotiate.c since 7.62.0 and that's 07ebaf8, but it seems innocuous enough...

Would you be able to bisect your way to spot/verify the exact change that broke this functionality? See https://github.com/curl/curl/wiki/how-to-git-bisect on how to do it.

@gavinhungry
Copy link

@bagder:

I bisected curl-7_62_0 (good) with curl-7_63_0 (bad) on Arch Linux.

07ebaf837843124ee670e5b8c218b80b92e06e47 is the first bad commit
commit 07ebaf837843124ee670e5b8c218b80b92e06e47
Author: Elia Tufarolo <elia.tufarolo@hcl.com>
Date:   Tue Nov 13 18:30:56 2018 +0100

    http_negotiate: do not close connection until negotiation is completed
    
    Fix HTTP POST using CURLAUTH_NEGOTIATE.
    
    Closes #3275

:040000 040000 84a440504f00b9345a68585a5640ecea5fda9e75 b947fa1d37f0012693c21f1ccd984e7377cea18f M  lib

@michael-o
Copy link
Contributor

michael-o commented Jan 4, 2019

Sigh, I just have been hit by this on FreeBSD.

7.62.0 is working:

osipovmi@blnn719x:~/var/Projekte/ld-docgen/backend-service/webapp (staging *%=)
$  $HOME/curl-7.62.0/bin/curl --version
curl 7.62.0-DEV (i386-unknown-freebsd11.2) libcurl/7.62.0-DEV OpenSSL/1.0.2q zlib/1.2.11 libidn2/2.0.5 nghttp2/1.35.1
Release-Date: [unreleased]
Protocols: file http https smtp smtps
Features: AsynchDNS IDN Largefile GSS-API Kerberos SPNEGO NTLM SSL libz TLS-SRP HTTP2 HTTPS-proxy
osipovmi@blnn719x:~/var/Projekte/ld-docgen/backend-service/webapp (staging *%=)
$ $HOME/curl-7.62.0/bin/curl -k --negotiate -u : --upload-file target/ld-docgen-webapp-0.0.5-SNAPSHOT-backend-dev.war --verbose 'https://blnn719x.ad001.siemens.net:8444/manager/text/deploy?path=/ld-docgen-backend-dev&update=true&version=010'
*   Trying 147.54.64.19...
* TCP_NODELAY set
* Connected to blnn719x.ad001.siemens.net (147.54.64.19) port 8444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=DE; O=Siemens; OU=PD LD AP DW; CN=blnn719x.ad001.siemens.net
*  start date: May 31 13:00:16 2018 GMT
*  expire date: May 31 13:00:16 2019 GMT
*  issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZZZZZB7; OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> PUT /manager/text/deploy?path=/ld-docgen-backend-dev&update=true&version=010 HTTP/1.1
> Host: blnn719x.ad001.siemens.net:8444
> User-Agent: curl/7.62.0-DEV
> Accept: */*
> Content-Length: 6690415
> Expect: 100-continue
>
< HTTP/1.1 401
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 1034
< Date: Fri, 04 Jan 2019 10:14:35 GMT
< Connection: close
<
* Excess found in a non pipelined read: excess = 1034 url = /manager/text/deploy (zero-length body)
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
* Issue another request to this URL: 'https://blnn719x.ad001.siemens.net:8444/manager/text/deploy?path=/ld-docgen-backend-dev&update=true&version=010'
* Hostname blnn719x.ad001.siemens.net was found in DNS cache
*   Trying 147.54.64.19...
* TCP_NODELAY set
* Connected to blnn719x.ad001.siemens.net (147.54.64.19) port 8444 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* SSL re-using session ID
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* old SSL session ID is stale, removing
* Server certificate:
*  subject: C=DE; O=Siemens; OU=PD LD AP DW; CN=blnn719x.ad001.siemens.net
*  start date: May 31 13:00:16 2018 GMT
*  expire date: May 31 13:00:16 2019 GMT
*  issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZZZZZB7; OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Server auth using Negotiate with user ''
> PUT /manager/text/deploy?path=/ld-docgen-backend-dev&update=true&version=010 HTTP/1.1
> Host: blnn719x.ad001.siemens.net:8444
> Authorization: Negotiate YIISIgYGKwYBBQUCoIISFj...
> User-Agent: curl/7.62.0-DEV
> Accept: */*
> Content-Length: 6690415
> Expect: 100-continue
>
< HTTP/1.1 100
* We are completely uploaded and fine
< HTTP/1.1 200
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate oYH1MIH...
< X-Remote-User: osipovmi@AD001.SIEMENS.NET
< X-Remote-AuthType: SPNEGO
< X-Content-Type-Options: nosniff
< Content-Type: text/plain;charset=utf-8
< Transfer-Encoding: chunked
< Date: Fri, 04 Jan 2019 10:14:46 GMT
<
OK - Deployed application at context path [/ld-docgen-backend-dev##010]
* Closing connection 1
* TLSv1.2 (OUT), TLS alert, close notify (256):

and now broken in 7.63.0:

osipovmi@blnn719x:~/var/Projekte/ld-docgen/backend-service/webapp (staging *%=)
$ $HOME/curl-7.63.0/bin/curl --version
curl 7.63.0-DEV (i386-unknown-freebsd11.2) libcurl/7.63.0-DEV OpenSSL/1.0.2q zlib/1.2.11 libidn2/2.0.5 nghttp2/1.35.1
Release-Date: [unreleased]
Protocols: file http https smtp smtps
Features: AsynchDNS IDN Largefile GSS-API Kerberos SPNEGO NTLM SSL libz TLS-SRP HTTP2 HTTPS-proxy
osipovmi@blnn719x:~/var/Projekte/ld-docgen/backend-service/webapp (staging *%=)
$ $HOME/curl-7.63.0/bin/curl -k --negotiate -u : --upload-file target/ld-docgen-webapp-0.0.5-SNAPSHOT-backend-dev.war --verbose 'https://blnn719x.ad001.siemens.net:8444/manager/text/deploy?path=/ld-docgen-backend-dev&update=false&version=010'             *   Trying 147.54.64.19...
* TCP_NODELAY set
* Connected to blnn719x.ad001.siemens.net (147.54.64.19) port 8444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=DE; O=Siemens; OU=PD LD AP DW; CN=blnn719x.ad001.siemens.net
*  start date: May 31 13:00:16 2018 GMT
*  expire date: May 31 13:00:16 2019 GMT
*  issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZZZZZB7; OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> PUT /manager/text/deploy?path=/ld-docgen-backend-dev&update=false&version=010 HTTP/1.1
> Host: blnn719x.ad001.siemens.net:8444
> User-Agent: curl/7.63.0-DEV
> Accept: */*
> Content-Length: 6690415
> Expect: 100-continue
>
< HTTP/1.1 401
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 1034
< Date: Fri, 04 Jan 2019 09:56:45 GMT
< Connection: close
<
* Excess found in a non pipelined read: excess = 1034 url = /manager/text/deploy (zero-length body)
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
* Issue another request to this URL: 'https://blnn719x.ad001.siemens.net:8444/manager/text/deploy?path=/ld-docgen-backend-dev&update=false&version=010'
* Hostname blnn719x.ad001.siemens.net was found in DNS cache
*   Trying 147.54.64.19...
* TCP_NODELAY set
* Connected to blnn719x.ad001.siemens.net (147.54.64.19) port 8444 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=DE; O=Siemens; OU=PD LD AP DW; CN=blnn719x.ad001.siemens.net
*  start date: May 31 13:00:16 2018 GMT
*  expire date: May 31 13:00:16 2019 GMT
*  issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZZZZZB7; OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Server auth using Negotiate with user ''
> PUT /manager/text/deploy?path=/ld-docgen-backend-dev&update=false&version=010 HTTP/1.1
> Host: blnn719x.ad001.siemens.net:8444
> Authorization: Negotiate YIISIgYGKwYBBQU...
> User-Agent: curl/7.63.0-DEV
> Accept: */*
> Content-Length: 0
>
< HTTP/1.1 200
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate oYH1MIHyoAMKAQC...
< X-Remote-User: osipovmi@AD001.SIEMENS.NET
< X-Remote-AuthType: SPNEGO
< X-Content-Type-Options: nosniff
< Content-Type: text/plain;charset=utf-8
< Transfer-Encoding: chunked
< Date: Fri, 04 Jan 2019 09:56:46 GMT
<
* Ignoring the response-body
* Closing connection 1
* TLSv1.2 (OUT), TLS alert, close notify (256):
* Issue another request to this URL: 'https://blnn719x.ad001.siemens.net:8444/manager/text/deploy?path=/ld-docgen-backend-dev&update=false&version=010'
* Hostname blnn719x.ad001.siemens.net was found in DNS cache
*   Trying 147.54.64.19...
* TCP_NODELAY set
* Connected to blnn719x.ad001.siemens.net (147.54.64.19) port 8444 (#2)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=DE; O=Siemens; OU=PD LD AP DW; CN=blnn719x.ad001.siemens.net
*  start date: May 31 13:00:16 2018 GMT
*  expire date: May 31 13:00:16 2019 GMT
*  issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZZZZZB7; OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> PUT /manager/text/deploy?path=/ld-docgen-backend-dev&update=false&version=010 HTTP/1.1
> Host: blnn719x.ad001.siemens.net:8444
> User-Agent: curl/7.63.0-DEV
> Accept: */*
> Content-Length: 6690415
> Expect: 100-continue
>
< HTTP/1.1 401
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< WWW-Authenticate: Negotiate
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 1034
< Date: Fri, 04 Jan 2019 09:56:46 GMT
< Connection: close
<
* Closing connection 2
* TLSv1.2 (OUT), TLS alert, close notify (256):
<!doctype html><html lang="en"><head><title>HTTP Status 401 – Unauthorized</title><style type="text/css">h1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} h2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} h3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;} a {color:black;} a.name {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 401 – Unauthorized</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The request has not been applied because it lacks valid authentication credentials for the target resource.</p><hr class="line" /><h3>Apache Tomcat/8.5.35</h3></body></html>

I bisected down to:

osipovmi@blnn719x:~/var/Projekte/curl ((07ebaf837...)|BISECTING)
$ git bisect bad
07ebaf837843124ee670e5b8c218b80b92e06e47 is the first bad commit
commit 07ebaf837843124ee670e5b8c218b80b92e06e47
Author: Elia Tufarolo <elia.tufarolo@hcl.com>
Date:   Tue Nov 13 18:30:56 2018 +0100

    http_negotiate: do not close connection until negotiation is completed

    Fix HTTP POST using CURLAUTH_NEGOTIATE.

    Closes #3275

:040000 040000 84a440504f00b9345a68585a5640ecea5fda9e75 b947fa1d37f0012693c21f1ccd984e7377cea18f M       lib

@bagder @kdudka
Can we drop this commit for now until the cause has been identified? This is a showstopper for us.

bagder added a commit that referenced this issue Jan 4, 2019
…completed"

This reverts commit 07ebaf8.

This also reopens PR #3275 which brought the change now reverted.

Fixes #3384
bagder added a commit that referenced this issue Jan 5, 2019
…completed"

This reverts commit 07ebaf8.

This also reopens PR #3275 which brought the change now reverted.

Fixes #3384
@bagder bagder closed this as completed in ebe658c Jan 7, 2019
@michael-o
Copy link
Contributor

@bagder thanks, looking forward to 7.64.0.

@lock lock bot locked as resolved and limited conversation to collaborators Apr 7, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Development

No branches or pull requests

5 participants