The Java and C++ have fairly extensive system tests (the Java has less extensive unit tests, I'm not sure about the C++ coverage). The .Net has significanly less, but they tend to transliterations of the Java tests. There's also a bunch of interop tests which are reimplemented in each of the languages, but has patchy coverage (Java and C++ implement most, the .Net implements less, I don't think Python implements any). While attempting to automate running these, it was pointed out that the obvious model for the interop tests is that the coordinator sends a test case to the clients describing what they should do rather than a "run test 1" message.
Proposal:
Implement a generic system for turning test definitions into test code.
Outline:
Given a document like this:
<test>
<create type="queue" name="queue" exhange="amq.direct" routing-key="queue">
<create type="consumer" name="consumer" destination="queue">
<create type="producer" name="producer" destination="queue">
<send number="10" exchange="amq.direct" routing-key="queue" size="1024" producer="producer">
<recieve number="10" consumer="consumer">
</test>
the test would send 10 messages through the broker and read them back.
This would allow for easy sharing of test cases throughout the clients and would mean that the interop test co-ordinator could send an xml document to the clients to allow for easy extension of the interop tests without having clients lagging behind.
Problems:
Well, writing such a beast shouldn't be too difficult, although a couple of questions immediately raise their heads with regard to onMessage vs receieve senders and threading that would need to be addressed. Also the test grammar would need to be defined and agreed quite closely for this to really work.
The other question is whether the test runners should interpret the XML directly, which would allow for consistency of code between the test suite and the interop testing, or if they should generate FooUnit test cases which would allow for easier debugging but a longer build cycle.
Comments and implementations gratefully received.