Hey,

I am using libcurl's multi APIs to upload (push) a file to a sharepoint
server with NTLM authentication. Everything is fine and dandy when the file
size is >= 2000 bytes, but hangs when we are < 2000 bytes. I'll try to
frame the problem, my tests, a solution and questions as concisely as
possible.

*My setup*
////////////////////////////////////////////////////////////////////////////////////////////////////
*// general setup*
CURLM *multi_handle = curl_multi_init();
CURL *curl = curl_easy_init();
curl_easy_setopt(curl, CURLOPT_URL, pRequest->getUrl());
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L);
*// stream setup*
curl_off_t fileSize = (curl_off_t)pFileStream->size();
curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, fileSize);
curl_easy_setopt(curl, CURLOPT_READFUNCTION, &readFromStream);
curl_easy_setopt(curl, CURLOPT_READDATA, &readStreamState);
curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, &writeToStream);
curl_easy_setopt(curl, CURLOPT_HEADERDATA, &headerStream);
*// set up cookies / headers*
curl_easy_setopt(curl, CURLOPT_COOKIE, cookies.c_str());
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);

*// do our upload*
bool isDone = false;
int itemsStillRunning, numMessages;
curl_multi_add_handle(multi_handle, curl);
while(!this->wasCanceled()){
   CURLMcode code = curl_multi_perform(multi_handle, &itemsStillRunning);
   assert(code == CURLM_OK);
   while(CURLMsg *msg = curl_multi_info_read(multi_handle, &numMessages)){
      if(msg->msg == CURLMSG_DONE){
         isDone = true;
         break;
      }
   }
   if(isDone){
      break;
   }
   this->flush();
}

*// cleanup*
curl_slist_free_all(headers);
curl_multi_remove_handle(multi_handle, curl);
curl_easy_cleanup(curl);
curl_multi_cleanup(multi_handle);
////////////////////////////////////////////////////////////////////////////////////////////////////

*Things I've tried*
1. running the same upload with the easy APIs (which will also hang)
2. tried using libcurl 7.30.0 (which also has the same problem)
3. turned on verbose to see that NTLM is getting hanging after a few 401s
3.1 enabling CURLOPT_FORBID_REUSE to see if internal state has anything to
do with this issue. However, CURLOPT_FORBID_REUSE paired with NTLM is a
no-no, as NTLM requires a keep alive to authenticate.
3.2 I see various verbose output for "Rewind stream after send", so I
thought maybe libcurl is trying to rewind my streams. Installing handlers
for CURLOPT_SEEKFUNCTION and CURLOPT_IOCTLFUNCTION does nothing for this
issue.

*Looking into the source*
There seems to be some code that is specific to NTLM and uploads of < 2000
bytes (located at http.c line~386).

////////////////////////////////////////////////////////////////////////////////////////////////////
if((*(expectsend - bytessent) < 2000)* || (conn->ntlm.state !=
NTLMSTATE_NONE) || (conn->proxyntlm.state != NTLMSTATE_NONE)) {
   /* The NTLM-negotiation has started *OR* there is just a little (<2K)
data left to send, keep on sending. */
   /* rewind data when completely done sending! */
   if(!conn->bits.authneg){
      conn->bits.rewindaftersend = TRUE;
      infof(data, "Rewind stream after send\n");
   }
   return CURLE_OK;
}
////////////////////////////////////////////////////////////////////////////////////////////////////

I've only been using libcurl for the past few months, and have only been
looking at the source these past few days, so I definitely don't have any
insights on some of the black magic / tricks / legacy code that may exist
within libcurl.

*A Solution*
2000 seemed like an odd number. Through my naivety, I decided to remove the
restriction on < 2000 bytes and see what happens. And lo and behold, small
file uploads work.

*Questions*
+ If we are below 2000 bytes and working with NTLM, we set rewindaftersend
= true. What is the purpose of this? The change above means rewindaftersend
does not get set, and our NTLM authentication continues, eventually
authenticates and uploads our file.
+ Why 2000 bytes?
+ I'd like to test this against some test cases - Is the test framework
functioning for libcurl 7.30.0? I've tried running it, but I get various
lexical errors in Makefile.am (eg. line 69 perhcheck -> perlcheck).
+ Does anyone have any insights on alternative solutions to upload small
files (<2000 bytes) using NTLM authentication?

Thank you so much for your help.

Vom

-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2013-06-01