RE: [EXTERNAL] Re: libcurl 8.0.1 and mem leaks reported on callback - windows x64 build

From: Parham, Edward C via curl-library <curl-library_at_lists.haxx.se>
Date: Sun, 9 Apr 2023 10:25:07 +0000

Hi everyone,

So…before I went to bed, I modified Tyler’s program. I took his main and did:

//--------modified main()----------
CURL* curl;

//global init
curl_global_init(CURL_GLOBAL_ALL);
curl = curl_easy_init();

while(true)
{
    Sleep(1000);
    PerformCurlWork(curl);
}

//Easy Cleanup
curl_easy_cleanup(curl);

//Global Cleanup
curl_global_cleanup();


//------------modified PerformCurlWork with CURL pointer parameter----------
void PerformCurlWork(CURL* curlHandle)
{
 
    CURLcode res;
    curl_slist* headers = {};
    ...
    curl_easy_reset(curlHandle);


--------------------------------------Results--------------------------------------
Tyler’s old program over five hours sleeping every three seconds:
Start: 1740k private bytes
Hour Five: 8310 private bytes

With my modifications running once a second:
Start: 1740k private bytes
Hour Six: 1916k private bytes
(private bytes fluctuate up 1884-1900s)
--------------------------------------------------------------------------------------

You’ll see that by initializing the handle you’re going to repeatedly use once and minimizing the easy_init and easy_cleanup, you’ll have substantial performance benefits on the Windows OS. The reset is a nice way to keep things tidy IMHO. So definitely not a memory leak...just 'Windows being Windows" as Tyler puts it LOL. But this does serve as a good reminder about optimization and efficiency of resources. Doing Tyler's original program on Linux is no sweat for the OS. But on Windows, you get dramatically different performance results.

Hope this helps solve the mystery!

Thanks,
Ed


Ed Parham
Software Architect (EPMS)
Eaton Digital Office / Data Center Segment, Eaton
edwardcparham_at_eaton.com





From: curl-library <curl-library-bounces_at_lists.haxx.se> On Behalf Of Parham, Edward C via curl-library
Sent: Saturday, April 8, 2023 11:26 PM
To: libcurl development <curl-library_at_lists.haxx.se>
Cc: Parham, Edward C <EdwardCParham_at_Eaton.com>
Subject: Re: [EXTERNAL] Re: libcurl 8.0.1 and mem leaks reported on callback - windows x64 build

The one thing to keep in mind about Windows - it's a very "expensive" to constantly create and destroy these handles to curl repetitively.  Curl is beautifully lightweight and flexible. The OS...I cannot say the same for. 

I'm facing a similar issue as Tyler but am working on an object pool to simply create the handles and just calls a handle when I need it.  

I don't think this is a memory leak per-se with what is being observed.  Windows consuming and allocating memory each time a new curl handle is created/destroyed, created/destroyed, etc. is just an expensive process.

I'll chime in more tomorrow once I have some results.  I do see the same allocation he is using his code bin sample.  But...I'm in agreement about the earlier comment about the CRT debugger not seeing the string reallocate before it gets the results. 

Thanks,
Ed




________________________________________
From: curl-library <mailto:curl-library-bounces_at_lists.haxx.se> on behalf of Henrik Holst via curl-library <mailto:curl-library_at_lists.haxx.se>
Sent: Saturday, April 8, 2023 11:05:43 PM
To: libcurl development <mailto:curl-library_at_lists.haxx.se>
Cc: Henrik Holst <mailto:henrik.holst_at_millistream.com>
Subject: [EXTERNAL] Re: libcurl 8.0.1 and mem leaks reported on callback - windows x64 build
 
yeah it's hard to say what the problem is here, I hope some one with more Windows experience can step in with some knowledge.

I have run the same test on Linux, and thus gcc, for the last 42 minutes and memory usage have been stable :

henrik_at_Sineya:~$ top -p 115405

top - 05:03:47 up  9:50,  1 user,  load average: 0,64, 0,56, 0,49
Uppgifter:   1 totalt,   0 körande,   1 sovande,   0 stoppade,   0 zombie
%Cpu/er:  2,2 an,  0,4 sy,  0,0 ni, 97,3 in,  0,0 vä,  0,0 ha,  0,1 ma,  0,0 st
MiB Minn :  32007,3 totalt,   2159,8 fritt,   4024,9 anv.,  25822,6 buff/cache
MiB Växl:   8192,0 totalt,   8189,7 fritt,      2,3 anv.,  27438,8 tillg Minn

    PID ANVÄNDAR  PR  NI    VIRT    RES   DELT S  %CPU  %MIN      TID+ KOMMANDO                                                                                                                                    
 115405 henrik    20   0   93,9m  11,5m  10,0m S   0,0   0,0   0:00.77 testapp                                                                                                                                        

Now 11.5M is a lot of memory for a program this small but defauilt libcurl on Linux is compiled with a lot of stuff. I also changed the code slightly just to see that the readBuffer didn't do something strange but it remains sane:

The response code is: 200
the curl return code is: 0
size: 3816 capacity: 6240

The response code is: 200
the curl return code is: 0
size: 3816 capacity: 5816

The response code is: 200
the curl return code is: 0
size: 3816 capacity: 5382

The response code is: 200
the curl return code is: 0
size: 3816 capacity: 5382

The response code is: 200
the curl return code is: 0
size: 3816 capacity: 3816

/HH

Den sön 9 apr. 2023 kl 04:38 skrev Tyler Wilson via curl-library <mailto:curl-library_at_lists.haxx.se>:
That is correct.

I'm going to let it run overnight, but I recompiled the program and
changed the project properties to the following:

Windows SDK - Visual Studio 2022 (v143)
C++ Language Standard - ISO C++ 20 Standard
C Language Standard - ISO C17 (2018 standard).

I'm not seeing the type of memory growth that I was earlier at the
rate that I was.  The only thing I'm not 100% sure of is whether this
is "Windows being Windows" and being a pig with memory allocation, or
if there is truly a difference in memory consumption based on
configurations of C++ and C standard.  Admittedly, I wonder at times
whether the Legacy MSVC has some issues.

I'll keep you posted.

Thanks again for all the feedback.  I'm glad this is such an active
community.  And gosh...libcurl is so easy to stand-up and make it do
what I want it to do with minimal coding.

Thanks,
Tyler

On Sat, Apr 8, 2023 at 10:21 PM Henrik Holst
<mailto:henrik.holst_at_millistream.com> wrote:
>
> is the while in main() basically doing:
>
> while (true) {
>   PerformCurlWork();
>   Sleep(3000);
> }
>
> ?
>
> /HH
>
> Den sön 9 apr. 2023 kl 03:34 skrev Tyler Wilson via curl-library <mailto:curl-library_at_lists.haxx.se>:
>>
>> Thank you for the feedback so far everyone.  It wouldn't surprise me
>> if the CRT tools in Visual Studio may be providing false positives.
>> I'm kinda on a limb here trying to understand why I'm continuously
>> allocating memory though.
>>
>> At the moment, I modified the code slightly to run in a while loop
>> with 3000ms sleeps in between each post.  When I started the program,
>> I was around 3MB but now I'm up to close to 20MB with no signs of
>> letting down.  And this has been going on for five six hours now.  As
>> to the question about which line it's coming from, the CRT tools don't
>> seem to provide me with that much detail.  The only reason I know it's
>> the callback is because the bytes match up exactly with the response
>> I'm getting from Mockbin.
>>
>> Is there anyone in the community that deploys this in Windows
>> environments in addition to Linux?  I realize that most of the samples
>> I've seen here are Linux based.  I've tried debug mode, release mode,
>> etc..  I'm at a complete loss here as to why this isn't working.   If
>> it helps at all from the Visual Studio side of things:
>> Windows SDK Version =  10.0 (latest installed).
>> Platform Toolset =  Visual Studio 2022 (v143)
>> C++ Language Standard = Default ISO C++ 14 Standard
>> C Language Standard - Default (Legacy MSVC)
>>
>> I am going to try and recompile this and switch from Legacy MSVC to
>> either C11 or C17 and see if that makes a difference - those are the
>> two options I see in Visual Studio.  I'll admit I'm more C++ than C,
>> but is there a preference to the version of the C standard that my
>> program should be compiled with?
>>
>> Thanks,
>> Tyler
>>
>> On Sat, Apr 8, 2023 at 6:25 PM Henrik Holst
>> <mailto:henrik.holst_at_millistream.com> wrote:
>> >
>> > btw this could also be due to (and bear with me because it was ages ago that I did any programming on Windows) you are using one CRT (non debug) of the curl library and another CRT (debug) of your application so when you call curl_easy_perform() then Windows switches to the CRT of the libcurl DLL and then inside that CRT libcurl calls your WriteCallback() function where it then does a realloc on std::string.
>> >
>> > Could be that this switching of CRT:s is what is confusing the memory leak function of the CRT of your application or (and here I show how little I know about C++) that readBuffer() is not destroyed when PerformCurlWork() ends but instead when the libcurl DLL is unloaded which happens after your call to _CrtDumpMemoryLeaks().
>> >
>> > In the page for the function Microsoft writes this:
>> >
>> > False positives
>> >
>> > _CrtDumpMemoryLeaks can give false indications of memory leaks if a library marks internal allocations as normal blocks instead of CRT blocks or client blocks. In that case, _CrtDumpMemoryLeaks is unable to tell the difference between user allocations and internal library allocations. If the global destructors for the library allocations run after the point where you call _CrtDumpMemoryLeaks, every internal library allocation is reported as a memory leak. Versions of the Standard Template Library earlier than Visual Studio .NET may cause _CrtDumpMemoryLeaks to report such false positives.
>> >
>> >
>> > Unsure if any of this applies here due to me not knowing squat about C++ nor Windows anymore.
>> >
>> > /HH
>> >
>> > Den sön 9 apr. 2023 kl 00:03 skrev Henrik Holst <mailto:henrik.holst_at_millistream.com>:
>> >>
>> >> sounds like the VS2022 CRT debug tools don't unwind the stack before the check so it doesn't call the std:string destructor or something like that. I compiled your code on Linux and run it using Valgrind which is the #1 when it comes to memleak detection and it found none:
>> >>
>> >> henrik_at_Sineya:~$ valgrind ./memleaktest
>> >> ==62452== Memcheck, a memory error detector
>> >> ==62452== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
>> >> ==62452== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
>> >> ==62452== Command: ./hej
>> >> ==62452==
>> >>
>> >> The response code is: 200
>> >> the curl return code is: 0
>> >> ==62452==
>> >> ==62452== HEAP SUMMARY:
>> >> ==62452==     in use at exit: 0 bytes in 0 blocks
>> >> ==62452==   total heap usage: 4,633 allocs, 4,633 frees, 444,149 bytes allocated
>> >> ==62452==
>> >> ==62452== All heap blocks were freed -- no leaks are possible
>> >> ==62452==
>> >> ==62452== For lists of detected and suppressed errors, rerun with: -s
>> >> ==62452== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
>> >>
>> >> /HH
>> >>
>> >> Den lör 8 apr. 2023 kl 19:11 skrev Tyler Wilson via curl-library <mailto:curl-library_at_lists.haxx.se>:
>> >>>
>> >>> Hi everyone,
>> >>>
>> >>> I'm still learning, but I'm hoping you can help.
>> >>>
>> >>> I have libcurl up and running and love it.  But, I'm seeing memory leaks and not sure if it's me or something else.
>> >>>
>> >>> Stats:
>> >>> -  Windows platform x64, with Visual Studio 2022.
>> >>> -  Downloaded source code from curl website as a .gz file.
>> >>> -  Building according to win instructions using Native Tools:
>> >>>
>> >>> nmake /f Makefile.vc mode=dll MACHINE=x64 WITH_SSL=no DEBUG=no
>> >>>
>> >>>
>> >>> I have a very simple program that sends data to Mockbin.  It responds with the payload I sent plus a whole lot more.
>> >>>
>> >>> When my program is done though, VS2022 CRT debug tools claim that there is a memory leak.  Looking at the debug output, it's coming from the response that I'm getting from Mockbin.
>> >>>
>> >>> 'curlmemleakexample.exe' (Win32): Unloaded 'C:\Windows\System32\FWPUCLNT.DLL'
>> >>> Detected memory leaks!
>> >>> Dumping objects ->
>> >>> {160} normal block at 0x000002181100D260, 2496 bytes long.
>> >>>  Data: <         {      > 00 20 20 20 20 20 20 20 20 7B 0A 20 20 20 20 20
>> >>> {159} normal block at 0x00000218110062C0, 16 bytes long.
>> >>>  Data: <_at_d              > 40 64 80 A5 F6 7F 00 00 00 00 00 00 00 00 00 00
>> >>> Object dump complete.
>> >>>
>> >>> The data above is the response from Mockbin based on the length of the bytes and what I saw come back from Wireshark over HTTP.  I am calling global_init before my program starts, and calling global_free when I'm done.   I have pasted my sample of code at the following link:
>> >>>
>> >>> https://www.codebin.cc/code/304ead33e4dd78b7bb1eeb36460eed6a9a4fe85506b6f4185329a6e861e00f6e
>> >>>
>> >>> Why would I still be getting reported memory leaks on the information from the callback?  Is it because my callback is a global function?  Am I maybe not understanding something about the API and maybe it requires the function to do something different?
>> >>>
>> >>> Many thanks in advance for your help and assistance.  I hope I was able to give you enough details.
>> >>>
>> >>> Thanks....Tyler!
>> >>> --
>> >>> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
>> >>> Etiquette:   https://curl.se/mail/etiquette.html
>> --
>> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
>> Etiquette:   https://curl.se/mail/etiquette.html
--
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2023-04-09