Hi,

[[For the sake of the archives, this is a little verbose... it took
  me a lot of searching and trial and error to get this far, so
  hopefully having this in the list archive will save someone else
  the same trouble... please skip to the end for the actual questions]]

I'm writing a multi-protocol file download application using pycurl
7.19 built with libcurl from the curl-7.19.6 release. I want to use my
own progress bar, so I need to know the file size in advance of
launching the download proper...

BACKGROUND FOR THE ARCHIVES
===========================

Here is a test script to get the size of a file over HTTP:

  import cStringIO
  buf = cStringIO.StringIO()

  options = {
    pycurl.URL: 'http://example.com/a_26MB_binary_file',
    pycurl.HEADER: 1,
    pycurl.NOBODY: 1,
    pycurl.WRITEFUNCTION: buf.write,
    pycurl.NOPROGRESS: 1,
  }

  import pycurl
  c = pycurl.Curl()
  for (k, v) in options.items():
    c.setopt(k, v)
  c.perform()
  print ':OUTPUT:\n' + buf.getvalue() + ':END:'

which produces some output that con be parsed:

  :OUTPUT:
  HTTP/1.1 200 OK
  Date: Thu, 17 Sep 2009 23:34:09 GMT
  Server: Apache
  Last-Modified: Mon, 14 Sep 2009 03:18:55 GMT
  ETag: "e120d90-1a22aa2-47381203285c0"
  Accept-Ranges: bytes
  Content-Length: 27404962
  MS-Author-Via: DAV
  Content-Type: text/plain
  :END:

Similarly, if I replace the pycurl.URL line in the program above
with these lines:

  pycurl.URL: 'ftp://example.com/a_26MB_binary_file',
  pycurl.USERPWD: 'ftp:gary_at_example.com',

I can get the file size over FTP from this output:

  :OUTPUT:
  Content-Length: 27404962
  Accept-ranges: bytes
  :END:

I've found that I can fetch file *contents* over SFTP by replacing
the pycurl.URL above with the following:

  pycurl.URL: 'sftp://example.com/~/a_26MB_binary_file',
  pycurl.USERPWD: 'gary:',
  pycurl.SSH_PRIVATE_KEYFILE: '/home/gary/dsa_nopass',
  pycurl.SSH_PUBLIC_KEYFILE: '/home/gary/dsa_nopass' + '.pub',

QUESTION 1
==========

The above works only if I use a key with no-passphrase.
Is it possible to use a key that takes a passphrase?

I tried this:

  pycurl.URL: 'sftp://example.com/~/a_26MB_binary_file',
  pycurl.USERPWD: 'gary:key pass-phrase'
  pycurl.SSH_PRIVATE_KEYFILE: '/home/gary/dsa_pass',
  pycurl.SSH_PUBLIC_KEYFILE: '/home/gary/dsa_pass' + '.pub',

but it doesn't work (those keys do work with ssh proper):

  * About to connect() to example.com port 22 (#0)
  * Trying 123.45.67.89... * connected
  * Connected to example.com (123.45.67.89) port 22 (#0)
  * SSH authentication methods available: publickey,password
  * Using ssh public key file /home/gary/dsa_pass.pub
  * Using ssh private key file /home/gary/dsa_pass
  * SSH public key authentication failed: Unable to initialize private key from file
  * Authentication failure
  * Closing connection #0
  Traceback (most recent call last):
    File "./pycurl", line 32, in <module>
      c.perform()
  pycurl.error: (67, 'Authentication failure')

Is that just because I have a server with publickey,password?
Certainly if I change pycurl.USERPWD to use my login password,
rather than the privatekey passphrase, the download works...
but only because the publickey authentication fails and libcurl
falls back to password authentication.

QUESTION 2
==========

Setting aside how to get an SFTP connection using a private key
with a passphrase, is it possible to get just the filesize over
SFTP using pycurl?

I tried replacing the pycurl.URL setting in my program at the top
of this post with the following:

  pycurl.URL: 'sftp://example.com/~/a_26MB_binary_file',
  pycurl.USERPWD: 'gary:',
  pycurl.SSH_PRIVATE_KEYFILE: '/home/gary/dsa_nopass',
  pycurl.SSH_PUBLIC_KEYFILE: '/home/gary/dsa_nopass' + '.pub',

But even using a private key with no passphrase, to bypass the
'Unable to initialize private key form file' error, I get back an
empty response:

  * About to connect() to example.com port 22 (#0)
  * Trying 123.45.67.89... * connected
  * Connected to example.com (123.45.67.89) port 22 (#0)
  * SSH authentication methods available: publickey,password
  * Using ssh public key file /home/gary/dsa_nopass.pub
  * Using ssh private key file /home/gary/dsa_nopass
  * Initialized SSH public key authentication
  * Authentication complete
  * Connection #0 to host example.com left intact
  :OUTPUT:
  :END:
  * Closing connection #0

I know SFTP support is very new, but even if these are known
limitations in 7.19.6 I'd be happy to know that it's not something
dumb on my part that is stopping this from working.
 
Any help very much appreciated. Please let me know if I can provide
any additional details or otherwise help debug these problems.

Cheers,
    Gary

-- 
Gary V. Vaughan (gary_at_thewrittenword.com)