Skip to content

problems sending large data on macOS over unix domain sockets #4919

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
bagder opened this issue Feb 12, 2020 · 3 comments
Closed

problems sending large data on macOS over unix domain sockets #4919

bagder opened this issue Feb 12, 2020 · 3 comments
Assignees
@bagder

Comments

@bagder
Copy link
Member

bagder commented Feb 12, 2020

(I received this bug report by a user who shall remain anonymous)

curl 7.68.0 seems to have problems sending large data on macOS over unix domain sockets. You can reproduce by using netcat as a server like this:

rm -rf /tmp/foofoo.sock && nc -U -l /tmp/foofoo.sock

And then invoke curl:

curl -v --unix-sock /tmp/foofoo.sock -H "$(python -c 'print "x"*90000'): OK" http://localhost:8888/

(This has curl send a very long single header that ends with OK)

If you look at the output from curl, you’ll notice that it never gets around to sending the OK part of the string. This seems to be specific to recent curls and to macOS

Taking a look at dtrace it seems like curl stops asking select whether the socket is writable

$ sudo dtrace -n 'syscall::sendto:entry / execname == "curl" && arg0 == 5 / { printf("%s(%d): %d, %s, %d", execname, pid, arg0, copyinstr(arg1), arg2); } syscall::sendto:return / execname == "curl" / { printf("return %d %d", arg1, errno); } syscall::select:entry / execname == "curl" / { printf("select(%d, %p, %p, %p)\n", arg0, arg1, arg2, arg3); printf("READ:\n"); tracemem(copyin(arg1, 128), 16); printf("WRITE:\n"); tracemem(copyin(arg2, 128), 128);   } syscall::select:return / execname=="curl" / { printf("%s return: %d %d", probefunc, arg1, errno); }'
Password:
dtrace: description 'syscall::sendto:entry ' matched 4 probes
CPU     ID                    FUNCTION:NAME
  2    347                     select:entry select(4, 7ffeebbdf230, 7ffeebbdf2b0, 7ffeebbdf330)
READ:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
WRITE:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  2    348                    select:return select return: 0 0
  2    347                     select:entry select(6, 7ffeebbdf170, 7ffeebbdf0f0, 7ffeebbdf1f0)
READ:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
WRITE:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ...............
        10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  2    348                    select:return select return: 1 0
  2    427                     sendto:entry curl(4116): 5, GET / HTTP/1.1
Host: localhost:8888
User-Agent: curl/7.68.0
Accept: */*
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx, 90084
  2    428                    sendto:return return 8192 0
  6    347                     select:entry select(6, 7ffeebbdf230, 7ffeebbdf1b0, 7ffeebbdf2b0)
READ:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ...............
WRITE:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  6    348                    select:return select return: 1 0

(This is a printout of the send calls and then the fdsets passed to select, the relevant FD is 5 so 0x20)
@bagder
Copy link
Member Author

bagder commented Apr 5, 2020 •
edited
Loading

What puzzles me the most about this bug is that very little internals even know what kind of socket it is working with and most of the code is just agnostic and work with "a socket" independent of what kind of connection it is. It might imply that unix domain sockets somehow behaves differently than SOCK_STREAM ones do - on macos. Research is required.

The bug triggers if the custom header is larger than 8100 bytes something. Then only the first piece of the request is sent and nothing more is ever sent. The socket just doesn't seem to signal itself as writable anymore...

@bagder
Copy link
Member Author

bagder commented Apr 7, 2020

The bug triggers on macOS if the custom header is larger than 8100 bytes something. Then only the first piece of the request is sent and nothing more is ever sent. Clearly the logic for handling a "split" request send is broken, and I believe it is broken on all platforms.

This bug also reproduces on Linux but then at around ~220K so the custom header needs to be made much larger there.

@bagder
Copy link
Member Author

bagder commented Apr 7, 2020 •
edited
Loading

I arrived at this little patch that lets me write a simple test case for this problem using the regular test suite, which makes it easier to work on a fix:

diff --git a/lib/http.c b/lib/http.c
index 324dd7252..814d69fd2 100644
--- a/lib/http.c
+++ b/lib/http.c
@@ -1227,12 +1227,25 @@ CURLcode Curl_add_buffer_send(Curl_send_buffer **inp,
       return result;
     }
     memcpy(data->state.ulbuf, ptr, sendsize);
     ptr = data->state.ulbuf;
   }
-  else
+  else {
+#ifdef CURLDEBUG
+    /* Allow debug builds override this logic to force short initial sends */
+    char *p = getenv("CURL_SMALLREQSEND");
+    if(p) {
+      size_t altsize = (size_t)strtoul(p, NULL, 10);
+      if(altsize)
+        sendsize = CURLMIN(size, altsize);
+      else
+        sendsize = size;
+    }
+    else
+#endif
     sendsize = size;
+  }
 
   result = Curl_write(conn, sockfd, ptr, sendsize, &amount);
 
   if(!result) {
     /*

@bagder bagder self-assigned this Apr 7, 2020
bagder added a commit that referenced this issue Apr 7, 2020
@bagder
http: don't consider upload done if the request isn't completely sent…

Verified

This commit was signed with the committer’s verified signature.
bagder Daniel Stenberg
GPG key ID: 5CC908FDB71E12C2
Verified
Learn about vigilant mode
02d03c2 
… off

Fixes #4919
@bagder bagder linked a pull request Apr 7, 2020 that will close this issue
Fix split http GET request #5197
Closed
@bagder bagder closed this as completed in 0ef54ab Apr 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bagder bagder

Labels
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix split http GET request curl/curl
1 participant
@bagder