curl-and-python

RE: Convert client-certificate curl.exe command to pycurl

From: Binney, Peter <Peter.Binney_at_commerzbank.com>
Date: Thu, 21 May 2015 09:49:47 +0200

The verbose output from the script posted yesterday under https://github.com/pycurl/pycurl/issues/244 is below.

Any values set with curl.setopt(SSLxxx) are ignored, and the client certificate most recently added to Internet Explorer's store is always used.

* Rebuilt URL to: https://wiki-uat.ib.internal/
* Hostname was NOT found in DNS cache
* Trying 10.166.120.172...
* Connected to wiki-uat.ib.internal (10.166.120.172) port 443 (#0)
* schannel: SSL/TLS connection with wiki-uat.ib.internal port 443 (step 1/3)
* schannel: disable server certificate revocation checks
* schannel: sending initial handshake data: sending 77 bytes...
* schannel: sent initial handshake data: sent 77 bytes
* schannel: SSL/TLS connection with wiki-uat.ib.internal port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with wiki-uat.ib.internal port 443 (step 2/3)
* schannel: encrypted data buffer: offset 1460 length 4096
* schannel: encrypted data length: 1374
* schannel: encrypted data buffer: offset 1374 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with wiki-uat.ib.internal port 443 (step 2/3)
* schannel: encrypted data buffer: offset 2834 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with wiki-uat.ib.internal port 443 (step 2/3)
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with wiki-uat.ib.internal port 443 (step 2/3)
* schannel: encrypted data buffer: offset 4532 length 8192
* schannel: sending next handshake data: sending 3264 bytes...
* schannel: SSL/TLS connection with wiki-uat.ib.internal port 443 (step 2/3)
* schannel: encrypted data buffer: offset 43 length 8192
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with wiki-uat.ib.internal port 443 (step 3/3)
* schannel: incremented credential handle refcount = 1
* schannel: stored credential handle in session cache
> GET / HTTP/1.1
User-Agent: PycURL/7.19.5 libcurl/7.37.0 WinSSL zlib/1.2.8
Host: wiki-uat.ib.internal
Accept: */*


* schannel: client wants to read 16384 bytes
* schannel: encrypted data buffer: offset 0 length 16384
* schannel: encrypted data got 582
* schannel: encrypted data buffer: offset 582 length 16384
* schannel: decrypted data length: 538
* schannel: decrypted data added: 538
* schannel: decrypted data cached: offset 538 length 16384
* schannel: encrypted data length: 23
* schannel: encrypted data cached: offset 23 length 16384
* schannel: decrypted data buffer: offset 538 length 16384
* schannel: decrypted data returned 538
* schannel: decrypted data buffer: offset 0 length 16384
< HTTP/1.1 302 Moved Temporarily
< Date: Thu, 21 May 2015 07:40:42 GMT
* Server Apache/2.2.15 (Red Hat) mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_jk/1.2.40 is not blacklisted
< Server: Apache/2.2.15 (Red Hat) mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_jk/1.2.40
< X-Confluence-Request-Time: 1432194042446
* Added cookie JSESSIONID="9FA82120F7CE91E5AE40C7603A1DDA5B" for domain wiki-uat.ib.internal, path /, expire 0
< Set-Cookie: JSESSIONID=9FA82120F7CE91E5AE40C7603A1DDA5B; Path=/; Secure
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Location: https://wiki-uat.ib.internal/login.action;jsessionid=9FA82120F7CE91E5AE40C7603A1DDA5B?os_destination=%2Findex.action
< Content-Length: 0
< Connection: close
< Content-Type: text/html;charset=UTF-8
<
* Closing connection 0
* schannel: shutting down SSL/TLS connection with wiki-uat.ib.internal port 443
* schannel: clear security context handle
* schannel: decremented credential handle refcount = 0
* Issue another request to this URL: 'https://wiki-uat.ib.internal/login.action;jsessionid=9FA82120F7CE91E5AE40C7603A1DDA5B?os_destination=%2Findex.action'
* Hostname was found in DNS cache
* Trying 10.166.120.172...
* Connected to wiki-uat.ib.internal (10.166.120.172) port 443 (#1)
* schannel: SSL/TLS connection with wiki-uat.ib.internal port 443 (step 1/3)
* schannel: re-using existing credential handle
* schannel: sending initial handshake data: sending 109 bytes...
* schannel: sent initial handshake data: sent 109 bytes
* schannel: SSL/TLS connection with wiki-uat.ib.internal port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with wiki-uat.ib.internal port 443 (step 2/3)
* schannel: encrypted data buffer: offset 129 length 4096
* schannel: sending next handshake data: sending 43 bytes...
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with wiki-uat.ib.internal port 443 (step 3/3)
* schannel: incremented credential handle refcount = 1
> GET /login.action;jsessionid=9FA82120F7CE91E5AE40C7603A1DDA5B?os_destination=%2Findex.action HTTP/1.1
User-Agent: PycURL/7.19.5 libcurl/7.37.0 WinSSL zlib/1.2.8
Host: wiki-uat.ib.internal
Accept: */*
Cookie: JSESSIONID=9FA82120F7CE91E5AE40C7603A1DDA5B


* schannel: client wants to read 16384 bytes
* schannel: encrypted data buffer: offset 0 length 16384
* schannel: encrypted data got 1460
* schannel: encrypted data buffer: offset 1460 length 16384
* schannel: decrypted data length: 371
* schannel: decrypted data added: 371
* schannel: decrypted data cached: offset 371 length 16384
* schannel: encrypted data length: 1068
* schannel: encrypted data cached: offset 1068 length 16384
* schannel: decrypted data length: 6
* schannel: decrypted data added: 6
* schannel: decrypted data cached: offset 377 length 16384
* schannel: encrypted data length: 1041
* schannel: encrypted data cached: offset 1041 length 16384
* schannel: failed to decrypt data, need more data
* schannel: client wants to read 16384 bytes
* schannel: encrypted data buffer: offset 1041 length 16384
* schannel: encrypted data got 1460
* schannel: encrypted data buffer: offset 2501 length 16384
* schannel: failed to decrypt data, need more data
* schannel: client wants to read 16384 bytes
* schannel: encrypted data buffer: offset 2501 length 16384
* schannel: encrypted data got 1460
* schannel: encrypted data buffer: offset 3961 length 16384
* schannel: failed to decrypt data, need more data
* schannel: client wants to read 16384 bytes
* schannel: encrypted data buffer: offset 3961 length 16384
* schannel: encrypted data got 1460
* schannel: encrypted data buffer: offset 5421 length 16384
* schannel: failed to decrypt data, need more data
* schannel: client wants to read 16384 bytes
* schannel: encrypted data buffer: offset 5421 length 16384
* schannel: encrypted data got 1460
* schannel: encrypted data buffer: offset 6881 length 16384
* schannel: failed to decrypt data, need more data
* schannel: client wants to read 16384 bytes
* schannel: encrypted data buffer: offset 6881 length 16384
* schannel: encrypted data got 1324
* schannel: encrypted data buffer: offset 8205 length 16384
* schannel: decrypted data length: 8184
* schannel: decrypted data added: 8184
* schannel: decrypted data cached: offset 8561 length 16384
* schannel: decrypted data buffer: offset 8561 length 16384
* schannel: decrypted data returned 8561
* schannel: decrypted data buffer: offset 0 length 16384
< HTTP/1.1 200 OK
< Date: Thu, 21 May 2015 07:40:42 GMT
* Server Apache/2.2.15 (Red Hat) mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_jk/1.2.40 is not blacklisted
< Server: Apache/2.2.15 (Red Hat) mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_jk/1.2.40
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< X-Confluence-Request-Time: 1432194042612
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: text/html;charset=UTF-8
<
* schannel: client wants to read 16384 bytes
* schannel: encrypted data buffer: offset 0 length 16384
* schannel: encrypted data got 2920
* schannel: encrypted data buffer: offset 2920 length 16384
* schannel: decrypted data length: 2
* schannel: decrypted data added: 2
* schannel: decrypted data cached: offset 2 length 16384
* schannel: encrypted data length: 2897
* schannel: encrypted data cached: offset 2897 length 16384
* schannel: decrypted data length: 6
* schannel: decrypted data added: 6
* schannel: decrypted data cached: offset 8 length 16384
* schannel: encrypted data length: 2870
* schannel: encrypted data cached: offset 2870 length 16384
* schannel: failed to decrypt data, need more data
* schannel: client wants to read 16384 bytes
* schannel: encrypted data buffer: offset 2870 length 16384
* schannel: encrypted data got 1460
* schannel: encrypted data buffer: offset 4330 length 16384
* schannel: failed to decrypt data, need more data
* schannel: client wants to read 16384 bytes
* schannel: encrypted data buffer: offset 4330 length 16384
* schannel: encrypted data got 1460
* schannel: encrypted data buffer: offset 5790 length 16384
* schannel: failed to decrypt data, need more data
* schannel: client wants to read 16384 bytes
* schannel: encrypted data buffer: offset 5790 length 16384
* schannel: encrypted data got 1460
* schannel: encrypted data buffer: offset 7250 length 16384
* schannel: failed to decrypt data, need more data
* schannel: client wants to read 16384 bytes
* schannel: encrypted data buffer: offset 7250 length 16384
* schannel: encrypted data got 955
* schannel: encrypted data buffer: offset 8205 length 16384
* schannel: decrypted data length: 8184
* schannel: decrypted data added: 8184
* schannel: decrypted data cached: offset 8192 length 16384
* schannel: decrypted data buffer: offset 8192 length 16384
* schannel: decrypted data returned 8192
* schannel: decrypted data buffer: offset 0 length 16384
* schannel: client wants to read 16384 bytes
* schannel: encrypted data buffer: offset 0 length 16384
* schannel: encrypted data got 1460
* schannel: encrypted data buffer: offset 1460 length 16384
* schannel: decrypted data length: 2
* schannel: decrypted data added: 2
* schannel: decrypted data cached: offset 2 length 16384
* schannel: encrypted data length: 1437
* schannel: encrypted data cached: offset 1437 length 16384
* schannel: decrypted data length: 6
* schannel: decrypted data added: 6
* schannel: decrypted data cached: offset 8 length 16384
* schannel: encrypted data length: 1410
* schannel: encrypted data cached: offset 1410 length 16384
* schannel: failed to decrypt data, need more data
* schannel: client wants to read 16384 bytes
* schannel: encrypted data buffer: offset 1410 length 16384
* schannel: encrypted data got 2920
* schannel: encrypted data buffer: offset 4330 length 16384
* schannel: failed to decrypt data, need more data
* schannel: client wants to read 16384 bytes
* schannel: encrypted data buffer: offset 4330 length 16384
* schannel: encrypted data got 1460
* schannel: encrypted data buffer: offset 5790 length 16384
* schannel: failed to decrypt data, need more data
* schannel: client wants to read 16384 bytes
* schannel: encrypted data buffer: offset 5790 length 16384
* schannel: encrypted data got 756
* schannel: encrypted data buffer: offset 6546 length 16384
* schannel: decrypted data length: 6453
* schannel: decrypted data added: 6453
* schannel: decrypted data cached: offset 6461 length 16384
* schannel: encrypted data length: 72
* schannel: encrypted data cached: offset 72 length 16384
* schannel: decrypted data length: 2
* schannel: decrypted data added: 2
* schannel: decrypted data cached: offset 6463 length 16384
* schannel: encrypted data length: 49
* schannel: encrypted data cached: offset 49 length 16384
* schannel: decrypted data length: 5
* schannel: decrypted data added: 5
* schannel: decrypted data cached: offset 6468 length 16384
* schannel: encrypted data length: 23
* schannel: encrypted data cached: offset 23 length 16384
* schannel: decrypted data buffer: offset 6468 length 16384
* schannel: decrypted data returned 6468
* schannel: decrypted data buffer: offset 0 length 16384
* Closing connection 1
* schannel: shutting down SSL/TLS connection with wiki-uat.ib.internal port 443
* schannel: clear security context handle
* schannel: decremented credential handle refcount = 0

From: curl-and-python [mailto:curl-and-python-bounces_at_cool.haxx.se] On Behalf Of Dima Tisnek
Sent: Wednesday, May 13, 2015 3:31 PM
To: curl with python
Subject: RE: Convert client-certificate curl.exe command to pycurl


There's a good chance winssl prefers different format for certs.
On May 13, 2015 4:13 PM, "Binney, Peter" <Peter.Binney_at_commerzbank.com<mailto:Peter.Binney_at_commerzbank.com>> wrote:
I just download curl, I don't compile it. curl -V says:

curl 7.34.0 (i386-pc-win32) libcurl/7.34.0 OpenSSL/1.0.0k zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtsp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate Largefile NTLM SSL SSPI libz

-----Original Message-----
From: curl-and-python [mailto:curl-and-python-bounces_at_cool.haxx.se<mailto:curl-and-python-bounces_at_cool.haxx.se>] On Behalf Of Dima Tisnek
Sent: Wednesday, May 13, 2015 3:00 PM
To: curl with python
Subject: Re: Convert client-certificate curl.exe command to pycurl

Different library versions are a cause to investigate further.
Is your command-line curl also compiled against WinSSL?

On 13 May 2015 at 15:55, Binney, Peter <Peter.Binney_at_commerzbank.com<mailto:Peter.Binney_at_commerzbank.com>> wrote:
> Windows XP.
> Command line curl is using libcurl.dll 7.34 the pycurl (python 2) uses
> 7.37
>
> -----Original Message-----
> From: curl-and-python [mailto:curl-and-python-bounces_at_cool.haxx.se<mailto:curl-and-python-bounces_at_cool.haxx.se>] On
> Behalf Of Dima Tisnek
> Sent: Wednesday, May 13, 2015 2:51 PM
> To: curl with python
> Subject: Re: Convert client-certificate curl.exe command to pycurl
>
> Also please report your OS and what libraries are actually loaded.
>
> On linux you'd do
> ldd `which curl`
> ldd path-to/site-packages/pycurl.so
>
>
>
> On 13 May 2015 at 15:48, Dima Tisnek <dimaqq_at_gmail.com<mailto:dimaqq_at_gmail.com>> wrote:
>> Peter, please enable debug (verbose output) and I think you will be
>> able to track what went wrong.
>>
>> On 13 May 2015 at 15:31, Binney, Peter <Peter.Binney_at_commerzbank.com<mailto:Peter.Binney_at_commerzbank.com>> wrote:
>>> I am using a command-line call to fetch pages from a client-certificated site using a passworded .pem certificate. eg:
>>>
>>> curl -s --cert ./ResMon.pem:Password --cacert ./rootAndCA.pem
>>> --location --cookie-jar ./cookies.tmp https://wiki.ib.internal
>>>
>>> When I try to do this with pycurl the site is not presented with the client certificate and so returns a "you must login" page.
>>> I have tried numerous variations of the under-documented setopt options. My current version is:
>>>
>>> import pycurl
>>> import cStringIO
>>>
>>> fPointer = cStringIO.StringIO()
>>> curl = pycurl.Curl()
>>> curl.setopt(curl.URL, "https://wiki.ib.internal")
>>> curl.setopt(pycurl.WRITEFUNCTION, fPointer.write)
>>> curl.setopt(curl.CAINFO, CERTS + "./rootAndCA.pem")
>>> curl.setopt(curl.SSLCERT, CERTS + "./ResMon.pem")
>>> curl.setopt(curl.SSLCERTPASSWD, " Password ")
>>> curl.setopt(pycurl.FOLLOWLOCATION, 1) ## cf: --location
>>> curl.setopt(pycurl.COOKIEFILE, '') ## cf: --cookie-jar
>>> curl.perform()
>>>
>>> print("Response code: " + str(curl.getinfo(pycurl.RESPONSE_CODE)))
>>> print(fPointer.getvalue())
>>>
>>> Basically, the SSLCERTxxx information is not being used.
>>> What am I missing please?
>>>
>>> Thanks,
>>> Peter
>>>
>>> _______________________________________________
>>> http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
> _______________________________________________
> http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
>
> _______________________________________________
> http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python

_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
Received on 2015-05-21