Re: libcurl 8.0.1 and mem leaks reported on callback - windows x64 build

From: Henrik Holst via curl-library <curl-library_at_lists.haxx.se>
Date: Sun, 9 Apr 2023 05:05:43 +0200

yeah it's hard to say what the problem is here, I hope some one with more
Windows experience can step in with some knowledge.

I have run the same test on Linux, and thus gcc, for the last 42 minutes
and memory usage have been stable :

henrik_at_Sineya:~$ top -p 115405

top - 05:03:47 up 9:50, 1 user, load average: 0,64, 0,56, 0,49
Uppgifter: 1 totalt, 0 körande, 1 sovande, 0 stoppade, 0 zombie
%Cpu/er: 2,2 an, 0,4 sy, 0,0 ni, 97,3 in, 0,0 vä, 0,0 ha, 0,1 ma,
 0,0 st
MiB Minn : 32007,3 totalt, 2159,8 fritt, 4024,9 anv., 25822,6
buff/cache
MiB Växl: 8192,0 totalt, 8189,7 fritt, 2,3 anv., 27438,8 tillg
Minn

    PID ANVÄNDAR PR NI VIRT RES DELT S %CPU %MIN TID+
KOMMANDO

 115405 henrik 20 0 93,9m 11,5m 10,0m S 0,0 0,0 0:00.77
testapp


Now 11.5M is a lot of memory for a program this small but defauilt libcurl
on Linux is compiled with a lot of stuff. I also changed the code slightly
just to see that the readBuffer didn't do something strange but it remains
sane:

The response code is: 200
the curl return code is: 0
size: 3816 capacity: 6240

The response code is: 200
the curl return code is: 0
size: 3816 capacity: 5816

The response code is: 200
the curl return code is: 0
size: 3816 capacity: 5382

The response code is: 200
the curl return code is: 0
size: 3816 capacity: 5382

The response code is: 200
the curl return code is: 0
size: 3816 capacity: 3816

/HH

Den sön 9 apr. 2023 kl 04:38 skrev Tyler Wilson via curl-library <
curl-library_at_lists.haxx.se>:

> That is correct.
>
> I'm going to let it run overnight, but I recompiled the program and
> changed the project properties to the following:
>
> Windows SDK - Visual Studio 2022 (v143)
> C++ Language Standard - ISO C++ 20 Standard
> C Language Standard - ISO C17 (2018 standard).
>
> I'm not seeing the type of memory growth that I was earlier at the
> rate that I was. The only thing I'm not 100% sure of is whether this
> is "Windows being Windows" and being a pig with memory allocation, or
> if there is truly a difference in memory consumption based on
> configurations of C++ and C standard. Admittedly, I wonder at times
> whether the Legacy MSVC has some issues.
>
> I'll keep you posted.
>
> Thanks again for all the feedback. I'm glad this is such an active
> community. And gosh...libcurl is so easy to stand-up and make it do
> what I want it to do with minimal coding.
>
> Thanks,
> Tyler
>
> On Sat, Apr 8, 2023 at 10:21 PM Henrik Holst
> <henrik.holst_at_millistream.com> wrote:
> >
> > is the while in main() basically doing:
> >
> > while (true) {
> > PerformCurlWork();
> > Sleep(3000);
> > }
> >
> > ?
> >
> > /HH
> >
> > Den sön 9 apr. 2023 kl 03:34 skrev Tyler Wilson via curl-library <
> curl-library_at_lists.haxx.se>:
> >>
> >> Thank you for the feedback so far everyone. It wouldn't surprise me
> >> if the CRT tools in Visual Studio may be providing false positives.
> >> I'm kinda on a limb here trying to understand why I'm continuously
> >> allocating memory though.
> >>
> >> At the moment, I modified the code slightly to run in a while loop
> >> with 3000ms sleeps in between each post. When I started the program,
> >> I was around 3MB but now I'm up to close to 20MB with no signs of
> >> letting down. And this has been going on for five six hours now. As
> >> to the question about which line it's coming from, the CRT tools don't
> >> seem to provide me with that much detail. The only reason I know it's
> >> the callback is because the bytes match up exactly with the response
> >> I'm getting from Mockbin.
> >>
> >> Is there anyone in the community that deploys this in Windows
> >> environments in addition to Linux? I realize that most of the samples
> >> I've seen here are Linux based. I've tried debug mode, release mode,
> >> etc.. I'm at a complete loss here as to why this isn't working. If
> >> it helps at all from the Visual Studio side of things:
> >> Windows SDK Version = 10.0 (latest installed).
> >> Platform Toolset = Visual Studio 2022 (v143)
> >> C++ Language Standard = Default ISO C++ 14 Standard
> >> C Language Standard - Default (Legacy MSVC)
> >>
> >> I am going to try and recompile this and switch from Legacy MSVC to
> >> either C11 or C17 and see if that makes a difference - those are the
> >> two options I see in Visual Studio. I'll admit I'm more C++ than C,
> >> but is there a preference to the version of the C standard that my
> >> program should be compiled with?
> >>
> >> Thanks,
> >> Tyler
> >>
> >> On Sat, Apr 8, 2023 at 6:25 PM Henrik Holst
> >> <henrik.holst_at_millistream.com> wrote:
> >> >
> >> > btw this could also be due to (and bear with me because it was ages
> ago that I did any programming on Windows) you are using one CRT (non
> debug) of the curl library and another CRT (debug) of your application so
> when you call curl_easy_perform() then Windows switches to the CRT of the
> libcurl DLL and then inside that CRT libcurl calls your WriteCallback()
> function where it then does a realloc on std::string.
> >> >
> >> > Could be that this switching of CRT:s is what is confusing the memory
> leak function of the CRT of your application or (and here I show how little
> I know about C++) that readBuffer() is not destroyed when PerformCurlWork()
> ends but instead when the libcurl DLL is unloaded which happens after your
> call to _CrtDumpMemoryLeaks().
> >> >
> >> > In the page for the function Microsoft writes this:
> >> >
> >> > False positives
> >> >
> >> > _CrtDumpMemoryLeaks can give false indications of memory leaks if a
> library marks internal allocations as normal blocks instead of CRT blocks
> or client blocks. In that case, _CrtDumpMemoryLeaks is unable to tell the
> difference between user allocations and internal library allocations. If
> the global destructors for the library allocations run after the point
> where you call _CrtDumpMemoryLeaks, every internal library allocation is
> reported as a memory leak. Versions of the Standard Template Library
> earlier than Visual Studio .NET may cause _CrtDumpMemoryLeaks to report
> such false positives.
> >> >
> >> >
> >> > Unsure if any of this applies here due to me not knowing squat about
> C++ nor Windows anymore.
> >> >
> >> > /HH
> >> >
> >> > Den sön 9 apr. 2023 kl 00:03 skrev Henrik Holst <
> henrik.holst_at_millistream.com>:
> >> >>
> >> >> sounds like the VS2022 CRT debug tools don't unwind the stack before
> the check so it doesn't call the std:string destructor or something like
> that. I compiled your code on Linux and run it using Valgrind which is the
> #1 when it comes to memleak detection and it found none:
> >> >>
> >> >> henrik_at_Sineya:~$ valgrind ./memleaktest
> >> >> ==62452== Memcheck, a memory error detector
> >> >> ==62452== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward
> et al.
> >> >> ==62452== Using Valgrind-3.18.1 and LibVEX; rerun with -h for
> copyright info
> >> >> ==62452== Command: ./hej
> >> >> ==62452==
> >> >>
> >> >> The response code is: 200
> >> >> the curl return code is: 0
> >> >> ==62452==
> >> >> ==62452== HEAP SUMMARY:
> >> >> ==62452== in use at exit: 0 bytes in 0 blocks
> >> >> ==62452== total heap usage: 4,633 allocs, 4,633 frees, 444,149
> bytes allocated
> >> >> ==62452==
> >> >> ==62452== All heap blocks were freed -- no leaks are possible
> >> >> ==62452==
> >> >> ==62452== For lists of detected and suppressed errors, rerun with: -s
> >> >> ==62452== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0
> from 0)
> >> >>
> >> >> /HH
> >> >>
> >> >> Den lör 8 apr. 2023 kl 19:11 skrev Tyler Wilson via curl-library <
> curl-library_at_lists.haxx.se>:
> >> >>>
> >> >>> Hi everyone,
> >> >>>
> >> >>> I'm still learning, but I'm hoping you can help.
> >> >>>
> >> >>> I have libcurl up and running and love it. But, I'm seeing memory
> leaks and not sure if it's me or something else.
> >> >>>
> >> >>> Stats:
> >> >>> - Windows platform x64, with Visual Studio 2022.
> >> >>> - Downloaded source code from curl website as a .gz file.
> >> >>> - Building according to win instructions using Native Tools:
> >> >>>
> >> >>> nmake /f Makefile.vc mode=dll MACHINE=x64 WITH_SSL=no DEBUG=no
> >> >>>
> >> >>>
> >> >>> I have a very simple program that sends data to Mockbin. It
> responds with the payload I sent plus a whole lot more.
> >> >>>
> >> >>> When my program is done though, VS2022 CRT debug tools claim that
> there is a memory leak. Looking at the debug output, it's coming from the
> response that I'm getting from Mockbin.
> >> >>>
> >> >>> 'curlmemleakexample.exe' (Win32): Unloaded
> 'C:\Windows\System32\FWPUCLNT.DLL'
> >> >>> Detected memory leaks!
> >> >>> Dumping objects ->
> >> >>> {160} normal block at 0x000002181100D260, 2496 bytes long.
> >> >>> Data: < { > 00 20 20 20 20 20 20 20 20 7B 0A 20 20 20
> 20 20
> >> >>> {159} normal block at 0x00000218110062C0, 16 bytes long.
> >> >>> Data: <_at_d > 40 64 80 A5 F6 7F 00 00 00 00 00 00 00 00
> 00 00
> >> >>> Object dump complete.
> >> >>>
> >> >>> The data above is the response from Mockbin based on the length of
> the bytes and what I saw come back from Wireshark over HTTP. I am calling
> global_init before my program starts, and calling global_free when I'm
> done. I have pasted my sample of code at the following link:
> >> >>>
> >> >>>
> https://www.codebin.cc/code/304ead33e4dd78b7bb1eeb36460eed6a9a4fe85506b6f4185329a6e861e00f6e
> >> >>>
> >> >>> Why would I still be getting reported memory leaks on the
> information from the callback? Is it because my callback is a global
> function? Am I maybe not understanding something about the API and maybe
> it requires the function to do something different?
> >> >>>
> >> >>> Many thanks in advance for your help and assistance. I hope I was
> able to give you enough details.
> >> >>>
> >> >>> Thanks....Tyler!
> >> >>> --
> >> >>> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> >> >>> Etiquette: https://curl.se/mail/etiquette.html
> >> --
> >> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> >> Etiquette: https://curl.se/mail/etiquette.html
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
>

-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2023-04-09