Testing frameworks and suites
In Derby land, the word "framework" is usually used for specifying which JDBC driver to use when running a test, or a suite of multiple tests. There are currently three such frameworks that are recognized by the Derby test harness and being used in the developer community:
Embedded - Using the Derby Embedded JDBC driver
DerbyNetClient - Using the Derby Client JDBC driver
DerbyNet - Using the IBM Universal JDBC driver (also known as JCC)
In the DerbyNet and DerbyNetClient frameworks the Derby Network Server is accessed using one of the client drivers. In the Embedded framework the Network Server is not used.
Specifying test framework
Which framework to use can usually be specified on the command line by using the framework system property, e.g.:
> java -Dframework=DerbyNetClient org.apache.derbyTesting.functionTests.harness.RunTest lang/wisconsin.java
The default framework is the Embedded framework. Framework names are case sensitive. It is also possible to specify a specific framework when running suites, but keep in mind that not all tests are suitable for all frameworks - more on that below.
Test suites and frameworks
The Derby regression test suites contain multiple tests. The tests are arranged into suites for ease of use in running multiple related tests. A test suite can consist of both individual tests and other test suites.
It can be quite confusing for a new developer to figure out in which framework a specific test or test suite is being run while running "the mother of all test suites", derbyall, or some of its subsuites.
Some suites are set up for specific frameworks:
derbynetmats: basic network server tests using the DerbyNet framework. Note that this suite is also a subsuite of the derbynetclientmats suite.
derbynetclientmats: basic network server tests using the DerbyNetClient framework. Note that this suite is a superset of the derbynetmats suite.
(mats is a widely used testing abbreviation for "minimal acceptance test suite")
This means that during a derbyall run using default framework settings, most of the tests in the derbynetmats suite are run at least twice: Once in the DerbyNet framework and once in the DerbyNetClient framework.
The fact that individual tests may be excluded from being run in a certain framework by using .exclude files (in the "old" test harness only) makes this even more complicated.
The general rule for test authors is that it is best if a test can run in all three frameworks (embedded, and the two client configurations). For tests that verify pure SQL behavior, the framework shouldn't matter. Many of the derbyLang tests are meant to test just SQL behavior, so many just run in embedded framework only. Running these tests in other frameworks might generate different outputs so you may have to generate different masters.
How to find out the suites or frameworks in which a test will run?
To see in which suite(s) a particular test runs, you can examine the 'suites' directory of the code repository (e.g. here).
All non-JUnit suites are defined by a .runall file in the suites directory. This file lists all the individual tests in the suite, for example (from derbynetclientmats.runall):
junitTests/derbyNet/CompatibilityTest.java jdbcapi/xaSimplePositive.sql jdbcapi/xaStateTran.sql jdbcapi/lobStreams.java jdbcapi/XATest.java jdbcapi/checkDataSource.java derbynet/ByteArrayCombinerStreamTest.junit derbynet/SqlExceptionTest.junit lang/grantRevoke.java
In addition, any subsuites to be included in a suite are specified in a .properties file, also in the suites directory. For example, the suites/derbynetclientmats.properties file includes the following line:
suites= encodingTests derbynetclientmats derbynetmats jdbc40
You can also invoke the test or a suite without actually running the test(s), and then see if your test gets picked up in the framework of your interest. The "old" test harness has a listOnly option (-DlistOnly=true).
The JUnit tests are configured a little differently, see DerbyJUnitTesting.
There is a lot more information in the file java/testing/README.htm in the source tree.
Other helpful tips may be found via the DerbyTesting wiki page.