To bring this thread to a logical semi-conclusion of sorts:

On Sun, 3 Mar 2013 23:27:50 +0100 (CET)
Daniel Stenberg <daniel_at_haxx.se> wrote:

> curl has a complete test system and setup. You run tests by their
> numbers, and you use a start script to do so. That script reads test
> information from an XML(-like) file and then starts the necessary
> servers etc and then also passes on the necessary instructions to the
> server so that it will serve the test as intended. The script then
> runs the command as instructed in the test file and then it compares
> outputs, protocol dumps, checks for memory leaks and more.
>
> To see how the most simple FTP LIST test case looks like, see
> https://github.com/bagder/curl/blob/master/tests/data/test100
>
> The runtests.pl man page is here:
> http://curl.haxx.se/dev/runtests.html, the test case file format is
> documented here:
> https://github.com/bagder/curl/blob/master/tests/FILEFORMAT

I investigated that some more and the biggest question presently is
this:

It looks like the test runner launches both the test server and the
test client (which the test runner assumes to be curl). Is it possible
to launch just the server component?

>
> > Do you have any suggestions for how pycurl->libcurl translation
> > should be checked?
>
> My thinking is that you'd write a test stub library that provides
> libcurl's API (like perhaps called testpycurl), but for all function
> calls the lib would only output its given arguments in some kind of
> sensible manner and return valid return codes.
>
> The test code would then make sure that the pycurl code would equal a
> correct sequence of libcurl calls that "makes sense" according to the
> docs and that has been proven to work (perhaps by curl's own test
> suite). So as long as all tests makes "legal" libcurl sequences, they
> should be working and there should be corresponding test cases for
> them among the curl tests...
>
> It was only an idea. Maybe it doesn't work.
>

While I am not opposed to this idea, I am also not very eager to
implement it right now, for the following reasons:

1. Going through the entire pycurl test suite, there are tests which
require the operations to actually happen. For example, anything
involving multithreading. A case that I found while doing python 3
porting was trying to perform buffer (bytes) operations on a (unicode)
string. These cannot be tested by examining libcurl calls alone.

2. pycurl provides quite a bit of functionality on python level. For
example, it is legitimate to have tests checking that pycurl invokes
callbacks with appropriate arguments.

3. More generally, having a test that checks whether pycurl is able to
fetch a URL and save it to a file inspires a lot more confidence in the
library than a test that checks that the same operation issues a known
sequence of libcurl calls. The point here is that end to end
("acceptance") tests are useful and will likely remain around forever.

4. Most of the infrastructure I implemented so far I actually copied
from my other recent projects, which means I spent very little time on
it and I also have high confidence that it works.

My priorities right now are finishing python 2.5 compatibility, then
python 3 compatibility which still requires a nontrivial amount of work
and probably research as there are things that just don't work and I
have close to no idea why, then going through remaining sourceforge
patches and then through open tickets. Rewriting tests that I know to
work is not on the roadmap. That said, if anyone wants to investigate
using libcurl's tools (or any other, really) instead of what I wrote to
date, please feel free and let's discuss it afterwards.

Oleg
_______________________________________________
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-and-python
Received on 2013-03-07