curl-library
Re: segfault with gzip encoding support
Date: Tue, 23 Dec 2003 19:05:35 -0800 (PST)
oops. the code is attached:) thanks,
Vikram
On Tue, 23 Dec 2003, Vikram wrote:
>
> Hi Dan,
>
> Thanks for your reply. The patch applied successfully; the code still
> segfualts though.
>
> I decided to remove pycurl, python and my original code from the picture
> in order to narrow down the problem and make it easier to solve. To that
> effect I modifed the simple example called multithread.c in the libcurl
> website and ran it. It segfaults with the backtrace posted below. I have
> attached the code to this mail. The test website I try is www.google.com
> which has "accept-encoding: deflate,gzip"
>
> (gdb) bt
> #0 0x88085915 in Curl_unencode_gzip_write (data=0x8079000, k=0x8084358,
> nread=1293) at content_encoding.c:210
> #1 0x88082300 in Curl_readwrite (conn=0x8084000, done=0xbfafeef3 "") at
> transfer.c:990
> #2 0x88082e04 in Transfer (conn=0x8084000) at transfer.c:1401
> #3 0x88083563 in Curl_perform (data=0x8079000) at transfer.c:1875
> #4 0x880838d9 in curl_easy_perform (curl=0x8079000) at easy.c:261
> #5 0x80487b4 in pull_one_url (url=0x8048957) at multithread.c:39
> #6 0x880e50a8 in _thread_start () from /usr/lib/libc_r.so.4
> #7 0x0 in ?? ()
>
> Also as far as the zmemcpy goes I made zlib use the systems memcpy and get
> the same segfault so it doesn't look like a zlib's zmemcpy problem.
>
> Thanks for your help.
> Vikram
>
>
>
> On Mon, 22 Dec 2003, Dan Fandrich wrote:
>
> > On Mon, Dec 22, 2003 at 04:54:43PM -0800, Vikram wrote:
> > > We have been having problems with using multiple threads and downloading
> > > websites with 'gzip' content-encoding support. A backtrace from gdb is
> > > pasted below. This happens on a FreeBSD machine [4.8-release, 2gb ram].
> > >
> > > I then wrote a simple python program which does nothing else except create
> > > threads and each thread uses pycurl which uses curl-7.10.8 and does a
> > > simple get with encoding set to 'gzip'. The program segfaults. (zlib
> > > version 1.1.4)
> > >
> > > I am not sure whether the problem is with libcurl or with pycurl. Have
> > > there been any other problems reported with gzip encoding support and
> > > multiple threads (esp on freebsd machines)? Any suggestions ? Thanks.
> >
> > It's unlikely to be a problem with pycurl--the backtrace goes deep into the
> > bowels of libcurl.
> >
> > >
> > > Vikram
> > >
> > > -----------------------------------------------------------------
> > > first backtrace:
> > >
> > > #0 0x884eb302 in zmemcpy (
> > > dest=0xbfaedd6c ' ' <repeats 13 times>, "<html> <head>
> > > <title>Welcome to AdWords</title> <link href=\"AdsLater.css\"
> > > rel=\"stylesheet\"> <style type=\"text/css\"><!--\n\n
> > > div.buttonborder {\n width: auto;\n border: 2"...,
> > > source=0x8503000 ' ' <repeats 13 times>, "<html> <head>
> > > <title>Welcome to AdWords</title> <link href=\"AdsLater.css\"
> > > rel=\"stylesheet\"> <style type=\"text/css\"><!--\n\n
> > > div.buttonborder {\n width: auto;\n border: 2"..., len=5074)
> > > at zutil.c:68
> > > #1 0x884ee5cd in inflate_flush (s=0x8471900, z=0x83c93bc, r=0) at
> > > infutil.c:50
> > > #2 0x884ee46a in inflate_codes (s=0x8471900, z=0x83c93bc, r=0) at
> > > infcodes.c:237
> > > #3 0x884ecd55 in inflate_blocks (s=0x8471900, z=0x83c93bc, r=0) at
> > > infblock.c:340
> > > #4 0x884eb908 in inflate (z=0x83c93bc, f=2) at inflate.c:221
> > > #5 0x883beb2b in Curl_unencode_gzip_write (data=0x8356000, k=0x83c9358,
> > > nread=1572) at content_encoding.c:326
> > > #6 0x883bb300 in Curl_readwrite (conn=0x83c9000, done=0xbfafdf23 "") at
> > > transfer.c:990
> > > #7 0x883bbe04 in Transfer (conn=0x83c9000) at transfer.c:1401
> > > #8 0x883bc563 in Curl_perform (data=0x8356000) at transfer.c:1875
> > > #9 0x883bc8d9 in curl_easy_perform (curl=0x8356000) at easy.c:261
> > > #10 0x88397953 in do_curl_perform (self=0x833c40c, args=0x815f02c) at
> > > pycurl.c:651
> > > #11 0x80f7344 in PyCFunction_Call ()
> > > #12 0x80b2cb2 in call_function ()
> > > #13 0x80b017c in eval_frame ()
> > > #14 0x80b16f1 in PyEval_EvalCodeEx ()
> >
> > This first backtrace is the interesting one, although it doesn't completely
> > make sense to me at first glance. It appears that the zmemcpy is overwriting
> > an internal libcurl buffer which is allocated on the stack. My first guess
> > would be that your thread's stack isn't large enough; that buffer is 64 KiB
> > long, so your thread's stack has to be considerably larger than that.
> >
> > That buffer started out at 8 KiB in an earlier version if libcurl, which
> > is a reasonable size to allocate on the stack, but had to be increased
> > to 64 KiB when 8 proved to be inadequate. It should probably be changed
> > to be allocated on the heap, but it's difficult to avoid a memory leak
> > when an error occurs that way.
> >
> > I also noticed that a few zlib fields aren't being explicitly initialized
> > within libcurl. They're probably zeroed out elsewhere, but it's safer
> > to do it anyway; the attached patch does so. There's a slight but
> > non-zero chance this might be related to your problem.
> >
> >
> > > -----------------------------------------------------------------------------
> > > second backtrace:
> > >
> > >
> > > #0 0x88383915 in Curl_unencode_gzip_write (data=0x82d8000, k=0x8284b58,
> > > nread=1294) at content_encoding.c:210
> >
> > This line doesn't have any associated code. It may be an anomaly due
> > to a corrupted stack.
> >
> > > #1 0x88380300 in Curl_readwrite (conn=0x8284800, done=0xbfafe843 "") at
> > > transfer.c:990
> > > #2 0x88380e04 in Transfer (conn=0x8284800) at transfer.c:1401
> > > #3 0x88381563 in Curl_perform (data=0x82d8000) at transfer.c:1875
> > > #4 0x883818d9 in curl_easy_perform (curl=0x82d8000) at easy.c:261
> > > #5 0x8835c953 in do_curl_perform (self=0x82cfc0c, args=0x815f02c) at
> > > pycurl.c:651
> > > #6 0x80f7344 in PyCFunction_Call ()
> > > #7 0x80b2cb2 in call_function ()
> > > #8 0x80b017c in eval_frame ()
> > > #9 0x80b16f1 in PyEval_EvalCodeEx ()
> > > #10 0x80b2fcd in fast_function ()
> > > #11 0x80b2d85 in call_function ()
> > > #12 0x80b017c in eval_frame ()
> > > #13 0x80b16f1 in PyEval_EvalCodeEx ()
> > > #14 0x80b2fcd in fast_function ()
> > > #15 0x80b2d85 in call_function ()
> > > #16 0x80b017c in eval_frame ()
> > > #17 0x80b16f1 in PyEval_EvalCodeEx ()
> > > #18 0x80f6c5e in function_call ()
> > > #19 0x805ace7 in PyObject_Call ()
> > > #20 0x80b2950 in PyEval_CallObjectWithKeywords ()
> > > #21 0x80daf37 in t_bootstrap ()
> > > #22 0x8816a0a8 in _thread_start () from /usr/lib/libc_r.so.4
> >
> > >>> Dan
> > --
> > http://www.MoveAnnouncer.com The web change of address service
> > Let webmasters know that your web site has moved
> >
>
>
-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
- TEXT/PLAIN attachment: multithread.c