curl on Windows is slow when output destination is on SMB #12750
Comments
This branch will measure how long fwrite, fflush and fclose take to write data to output files. To override the default FILE buffer size (which I think internally in Microsoft's CRT is probably 4k) use CURL_OUTFILE_BUFSIZE. Example: REM create a sparse 1GB file REM fsutil file createnew c:\temp\input 1073741824 fsutil sparse setflag c:\temp\input fsutil sparse setrange c:\temp\input 0 1073741824 REM write from input to output using curl with default FILE buffer size REM for me the total fwrite time is about 600k microseconds on average REM set CURL_OUTFILE_BUFSIZE= curld -o output file://c:/temp/input REM write from input to output using curl with 1MB FILE buffer size REM for me the total fwrite time is about 350k microseconds on average REM set CURL_OUTFILE_BUFSIZE=1048576 curld -o output file://c:/temp/input Ref: curl#12750
This branch will measure how long fwrite, fflush and fclose take to write data to output files. To override the default FILE buffer size (which I think internally in Microsoft's CRT is probably 4k) use CURL_OUTFILE_BUFSIZE. Example: REM create a sparse 1GB file REM fsutil file createnew c:\temp\input 1073741824 fsutil sparse setflag c:\temp\input fsutil sparse setrange c:\temp\input 0 1073741824 REM write from input to output using curl with default FILE buffer size REM for me the total fwrite time is about 600k microseconds on average REM set CURL_OUTFILE_BUFSIZE= curld -o output file://c:/temp/input REM write from input to output using curl with 1MB FILE buffer size REM for me the total fwrite time is about 350k microseconds on average REM set CURL_OUTFILE_BUFSIZE=1048576 curld -o output file://c:/temp/input Ref: curl#12750
|
Maybe the fwrite default buffer size is too small? IIRC it's 4k in Windows. However I suspect in most cases microsoft's CRT functions are calling Windows API functions which have their own buffering and I'm sure there would be another layer of buffering for SMB so it didn't try to write 4k chunks repeatedly.... I made a test branch that monitors the cumulative time it takes to fwrite/fflush/fclose the output files in Windows, get it here: https://github.com/curl/curl/compare/master...jay:curl:win32_measure_fwrite_time?expand=1 Try outputting to a file via SMB with different values (as seen in the example in the commit message) I used file:// input above to rule out a server issue retrieving the input |
|
I applied the patch on top of 8.5.0, but could not compile it, because in my VS 2019 the internals of the FILE type are completely obscured; the definition is only: and therefore I get compile errors about |
This branch will measure how long fwrite, fflush and fclose take to write data to output files. To override the default FILE buffer size (which I think internally in Microsoft's CRT is probably 4k) use CURL_OUTFILE_BUFSIZE. Example: REM create a sparse 1GB file REM fsutil file createnew c:\temp\input 1073741824 fsutil sparse setflag c:\temp\input fsutil sparse setrange c:\temp\input 0 1073741824 REM write from input to output using curl with default FILE buffer size REM for me the total fwrite time is about 600k microseconds on average REM set CURL_OUTFILE_BUFSIZE= curld -o output file://c:/temp/input REM write from input to output using curl with 1MB FILE buffer size REM for me the total fwrite time is about 350k microseconds on average REM set CURL_OUTFILE_BUFSIZE=1048576 curld -o output file://c:/temp/input Ref: curl#12750
|
Thanks for testing. I have amended the commit to remove the _bufsiz check.
That's quite a difference. I tested with a 1GB file and I also see a difference. 38 seconds vs 11 seconds if I set a 10MB FILE buffer. I also monitored the Windows API calls to see what the Microsoft CRT is actually calling: The file is opened as I would expect with CreateFile: Then it's written:
edit: actually I guess that amount of calls would be expected however I'm surprised Windows doesn't bundle small writes with some larger intermediate buffer before sending over the network. |
|
Can anyone else in Windows reproduce this? |
|
I did a few tests with the official version of the curl. The results are as follows: First Method (Using --upload-file): Second Method (using --output): Third Method (Using internal Copy command): The initial results make sense, and I think the SMB protocol version, buffer size, caching, etc. play some role. However, a more in-depth analysis (User+Kernel mode) is required to narrow down the root causes. Note: The First Method uses SMBv1 (presumably curl project implementation), whereas the other Methods use SMBv2/3 (depending on the OS). |
|
Thanks for testing. I think we can say that this is a Windows issue not a curl issue. We could mitigate the effect of Windows' smaller SMB buffer by writing larger blocks for network files. However, I'm hesitant to do that because even if we can reliably detect HANDLEs as network files there may be unwanted performance effects on other protocols. Also, it would use a lot more memory per file which would be a problem if many files were open in parallel mode. |
|
I'm not sure what's wrong with the new build (8.6.0), but the write speed is ~3.x times slower than the 8.5.0 or 8.4.0. I monitored the ReadFile and WriteFile API calls, and the buffer size is indeed different. The read buffer is 8192 bytes vs 102400 in previous versions, and write buffer is 4096 bytes vs 16384. |
@icing could this be from your changes to file protocol |
|
Probably mostly because of a6c9a33. |
|
But I don't see us reverting this change just because SMB on Windows is silly. |
yes but why are we doing that with the smaller read buffer |
|
Because we no longer use the download buffer - it is now stack based. Because it does not need to be big to do fast transfers everywhere else. And of course because we did not anticipate this particular thing to do even worse. |
|
I was going to suggest adding an option for changing the buffer size. Basically what @jay implemented earlier for testing (via CURL_OUTFILE_BUFSIZE), only now officially, with a command line option. This way the users who need downloading onto SMB would be able to get full speed, without affecting others. But if you decided to switch buffer located on stack, that may not be possible anymore... |
|
@bagder we could replace the stack buffers with allocated ones, using |
Since file:// transfers are also transfers, we could also make use of the download buffer for this... |
- refs curl#12750 - borrow multi handle's transfer buffer for up- and downloads of file: urls. - restores the buffer size behaviour of curl 8.5.0
|
I made #12932 which uses the multi's transfer buffer for file: operations. This should restore the buffer sizes and behaviour exactly as it was in curl 8.5.0. |
- For file:// transfers use the multi handle's transfer buffer for up- and downloads. Prior to this change a6c9a33 (precedes 8.6.0) changed the file:// transfers to use a smaller stack based buffer, and that caused a significant performance decrease in Windows. Bug: #12750 (comment) Reported-by: edmcln@users.noreply.github.com Closes #12932
|
Fixed via #12932 |
I did this
Make sure you have a file URL with high download speed; a local disk, and a network SMB resource with high upload speed (either mapped as a disk letter, or accessible via network share URI).
First, try to download the file with
--outputoption pointing onto a file located on the local disk, and mark the speed (might need a few seconds for it to stabilize).Second, try the same, but with
--outputpointing to the SMB share, and compare the speed.In my example, I set up a local HTTP server using
python -mhttp.serverfor a directory with some Linux ISO distributions on my local hard drive (max speed for downloading is 250-300 Mbytes/s); and for the destination I used an SMB share on my locally running CentOS 7 virtual machine (transfer speed varies, but is about 100-250 Mbytes/s). This way I made sure I'm not throttled by my ISP, or the upstream server. But my friend found this issue when downloading a file from Internet via 500 Mbps connection onto his NAS, so it doesn't have to be all local.My results:
In other words, the speed is over 10 times lower when saving onto SMB. To confirm that the SMB itself is not slow, here is the measurement of direct copy of the same file to the same SMB location:
which gives us about 122 Mbytes/s. (And it's not because of caching; I monitored for network activity, and it stopped as soon as the file copying finished.)
I expected the following
The download speed onto SMB should be roughly 100-200 Mbytes/s, as the hardware permits.
curl/libcurl version
Reproduced on several different builds:
operating system
Windows 10 22H2 Pro x64 (10.0.19045.3570)
Windows 11 23H2 Pro x64 (10.0.22631.3007)
The text was updated successfully, but these errors were encountered: