I just tried running curl's regression test suite with gcc's test coverage
feature again to see how much the test suite has improved since the last
time I tried it back in March (http://curl.haxx.se/mail/lib-2007-03/0028.html).
The enabled curl features are the same as last time, except that this
time I enabled debug mode:

curl 7.16.3 (i686-pc-linux-gnu) libcurl/7.16.3 OpenSSL/0.9.7a zlib/1.2.1.2 libidn/0.5.6
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: Debug IDN IPv6 Largefile NTLM SSL libz

Again, SSH and C-Ares are the notable omissions. This combination of
features allowed 355 of 373 tests to run.

Since March, 568 lines of code and 22 new functions have been added
(in the tested configuration--debug mode might be the cause of some of
them), and 45 test cases have been added to the test suite.

After the last report, I looked at some of the detailed coverage data
and added some tests to fill some of the gaps, and others have added
new tests as well. The code coverage figures have all increased except
for "functions completely executed". The increases have been modest,
however; e.g. blocks executed went from 62.5% to 64.4%, and branches
taken went from 48.6% to 50.7%. I'm a bit surprised the figures didn't
go up any more, but it means that the low-hanging fruit has been plucked.

Since March the torture tests have also been fixed and I'm now running
them as a daily autobuild. The torture tests simulate low memory conditions
by running curl many times and forcing a different malloc to fail each time,
allowing many of the internal error recovery code paths to execute.
Including the torture tests, the coverage figures look like this (from
ggcov):

Summary (all files)
=======
  67.2951% blocks executed (9994 of 14851)
     (0 blocks suppressed)
  84.6429% functions executed (474 of 560)
  35.8929% functions completely executed (201 of 560)
     (0 functions suppressed)
  70.913% lines executed (10905 of 15378)
  69.3328% lines completely executed (10662 of 15378)
     (0 lines suppressed)
  54.6% branches taken (4881 of 8941)

That figure of 70.9% is what is currently (as of 7.16.3) being achieved,
which compares to the 85% coverage goal I mentioned in March.

The larger blocks still left uncovered are the telnet, dict and ldap
protocols. After running a couple of manual tests on all three protocols
(plus a few other simple tests) these are the figures I get:

Summary (all files)
=======
  69.7125% blocks executed (10353 of 14851)
     (0 blocks suppressed)
  88.5714% functions executed (496 of 560)
  36.7857% functions completely executed (206 of 560)
     (0 functions suppressed)
  73.8913% lines executed (11363 of 15378)
  72.142% lines completely executed (11094 of 15378)
     (0 lines suppressed)

So creating three new test servers to handle three remaining protocols
gains less than 3% additional code coverage. That implies that the
remaining 28% of the code is spread throughout the library and is going
to take lots of effort to create exactly the right test conditions
to exercise. Some code is going to be impossible to exercise using the
public libcurl API because it's error handling code that won't ever be
called upon.

So, I think we're close the point of diminishing returns when it comes to
code coverage. But that's not intended to dissuade anyone from adding
those three new test servers, of course!

>>> Dan

-- 
http://www.MoveAnnouncer.com              The web change of address service
          Let webmasters know that your web site has moved