Bugs item #2667901, was opened at 2009-03-06 03:06
Message generated for change (Comment added) made by dfandrich
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=2667901&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: libcurl
Group: wrong behaviour
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Constantin Slednev (unk2k)
Assigned to: Nobody/Anonymous (nobody)
Summary: libcurl 7.19.4

Initial Comment:
libcurl 7.19.4
pycurl 7.19.0
python 2.4.4
bind9
Debian 2.6.30

Python fetch function code:
[code]
def fetch(pool,func,conn = 300):

    num_processed = 0

    num_pool = len(pool)

    ret=[]

    conn = min(num_pool,conn)

    m = CurlMulti()

    m.handles = []

    for i in range(conn):

        c = Curl()

        c.setopt(FOLLOWLOCATION, 1)

        c.setopt(MAXREDIRS, 2)

        c.setopt(NOSIGNAL, 0)

        c.setopt(CONNECTTIMEOUT, 3)

        c.setopt(TIMEOUT, 5)

        c.setopt(HTTPHEADER, ['User-Agent: Mozilla/5.0 (rv:1.8.1.12) Gecko/20080129 Firefox/2.0.0.14 Windows; U; Windows NT 6.0; en-US; rv:1.8.1.14'])

        m.handles.append(c)

    worker = m.handles[:]

    while num_pool > num_processed:

        while worker and pool:

            url = pool.pop()

            while url in pool:

                pool.remove(url)

                num_pool=num_pool-1

            c= worker.pop()

            c.res = cStringIO.StringIO()

            c.setopt(URL,url)

            c.setopt(WRITEFUNCTION,c.res.write)

            m.add_handle(c)

            c.url=url

        while 1:

            ret,num_h = m.perform()

            if ret != E_CALL_MULTI_PERFORM: break

        while 1:

            num_q, ok_list, err_list = m.info_read()

            for c in ok_list:

                m.remove_handle(c)

                c.data=c.res.getvalue()

                worker.append(c)

                print c.url

                break

                for mask in func:

                    if re.findall(mask[1],c.url):

                        mask[0](c)

            for c, errno, errmsg in err_list:

                m.remove_handle(c)

                c.data=c.res.getvalue()

                worker.append(c)

            num_processed = num_processed + len(ok_list)

            if num_q == 0:

                break

        try:

            m.select(1.0)

        except:

            pass

    return
[/code]

When I run the function fetch formed 300 threads, these threads install curl and the next step is to collapse dns server. The problem is that if the dns server that is (gethostbyname ()) does not answer, then my script stops at X time, until you reply. This delat curlMulti bad.

STOP IN THE:
ret,num_h = m.perform()

Strace give the:
proxy:~#strace -p11221
***************cut***************
close(6) = 0
munmap(0xb7a1f000, 4096) = 0
stat64("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=25, ...}) = 0
gettimeofday({1236187885, 415306}, NULL) = 0
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 6
connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("194.87.37.56")}, 28) = 0
fcntl64(6, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(6, F_SETFL, O_RDWR|O_NONBLOCK) = 0
gettimeofday({1236187885, 415451}, NULL) = 0
poll([{fd=6, events=POLLOUT}], 1, 0) = 1 ([{fd=6, revents=POLLOUT}])
send(6, "+\243\1\0\0\1\0\0\0\0\0\0\3www\16free-advertise\3n"..., 40, MSG_NOSIGNAL) = 40
poll([{fd=6, events=POLLIN}], 1, 5000^C <unfinished ...>
Process 14778 detached
proxy:~#

log BIND9 Say the:
too many timeouts resolving 'free-advertise.net/A' (in 'free-advertise.NET'?): disabling EDNS

        
Well, if we can not resolve 1 of the 299 then will pass rather than wait

My email: northcustoms_AT_ya.ru
Please fix the bug and report me.
Thank you very much

----------------------------------------------------------------------

>Comment By: Dan Fandrich (dfandrich)
Date: 2009-03-06 20:09

Message:
Have you compiled libcurl with c-ares?

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=2667901&group_id=976
Received on 2009-03-07