Differences between revisions 38 and 39
Revision 38 as of 2013-06-12 21:49:07
Size: 9573
Comment:
Revision 39 as of 2013-09-26 16:29:27
Size: 9583
Comment:
Deletions are marked like this. Additions are marked like this.
Line 72: Line 72:
|| '''Suite''' || '''Initial Test Count''' || '''Remaining Test Count (2007/02/15)''' || '''2013/06/11''' || '''Comments''' || || '''Suite''' || '''Initial Test Count''' || '''Remaining Test Count (2007/02/15)''' || '''2013/09/26''' || '''Comments''' ||
Line 89: Line 89:
|| jdbc20 ||10 || 5 || 1 || JdbcapiJunitTestConversion || || jdbc20 ||10 || 5 || 0 (removed) || JdbcapiJunitTestConversion ||
Line 108: Line 108:
|| '''Total''' || || 344 test runs with 306 unique tests || 135 test runs || || || '''Total''' || || 344 test runs with 306 unique tests || 134 test runs || ||

Replacing functionality of the Derby test harness

Goals

  • Derby's use of JUnit fits the standard pattern
    • All tests pass when run as individual tests
    • No setup required on the command line, can run tests directly from JUnit's TestRunners

    • Standard mechanism to add suites (use of TestSuite class directly)

  • Tests can be run from an ant target
  • Tests/Suites can be added into other project's test suites with no additional knowledge or requirements

Conversion or writing JUnit tests FAQ

SQL tests

Harness based script tests (.sql scripts) can be converted in three ways:

  • Continue running the test as a script using the ScriptTestCase class, see ConvertSqlScriptTestsToJunit

  • Look at using the utility org.apache.derbyTesting.functionTests.util.SQLToJUnit attached to convert the old test automatically. More info is in DERBY-2151.

  • Re-write the test by hand using standard JDBC calls and the utility methods and decorators available. The method org.apache.derbyTesting.junit.Utilities.showResultSet(ResultSet rs) can be used to print resultsets out in two dimensional array format for use by assertFullResultSet.

Decorators

A number of useful decorators and convenience methods that return decorators exist, this section shows which to use for certain situations. Remember that a decorator can wrap a set of test fixtures:

  return new CleanDatabaseTestSetup(suite);

or a single test fixture

 suite.addTest(new CleanDatabaseTestSetup(new MyJDBCTest("testJDBCStuff")));

Decorators can also be nested, some thought may be required to ensure the nesting results in the correct execution order:

  return new CleanDatabaseTestSetup(
             DatabasePropertyTestSetup.setLockTimeouts(suite, 2, 4));

Question

Answer

How to set system properties for a test?

Setting properties on a database level is preferred as setting properties at a system level may interfere with other tests that could be run in parallel. However the SystemPropertyTestSetup decorator does exist to allow setting system properties, passing in the set of properties to set.

How to set database properties for a test?

Use the DatabasePropertyTestSetup decorator passing in the set of properties to set.

How to reduce the lock timeout values for a test?

Use the conveniencedecorator returned by DatabasePropertyTestSetup.setLockTimeouts or DatabasePropertyTestSetup directly.

How to set BUILTIN authentication with a set of users

Use the convenience decorator returned by DatabasePropertyTestSetup.builtinAuthentication, use a unique password token

How to run as a different user?

Use the decorator returned by TestConfiguration.changeUserDecorator

How to run tests with Derby's client driver?

Any test class's suite() method should return a suite that runs fixtures in embedded and client/server mode as required. TestConfiguration.defaultSuite() takes a Class object and returns a suite that runs all the test methods starting with test in both embedded and client server. When more control is needed over wich tests are run in emebdded and client/server TestConfiguration provides these methods: clientServerDecorator, clientServerSuite & embeddedSuite. In addition the NetworkServerTestSetup decorator can be used directly.

Since the database is re-used when running tests how do I ensure the test starts out with a clean database?

Use the CleanDatabaseTestSetup decorator.

How to execute DDL at setUp that is dropped automatically at tearDown?

Provide a class that extends CleanDatabaseTestSetup and implement the decorateSQL method. This can be achieved as an inner class, search for references to CleanDatabaseTestSetup for examples.

How to run a test in SQL authorization mode?

Use the utility method TestConfiguration.sqlAuthorizationDecorator. The simple DatabasePropertyTestSetup cannot be used directly since setting the SQL authorization mode cannot be undone. The returned decorator changes the default database to a different one that has SQL authorization mode set. This database, like the default database, is re-used by each call to TestConfiguration.sqlAuthorizationDecorator.

Harness features

The test decorator pattern, a class extending TestSetup or the Derby specific BaseJDBCTestSetup seems the natural way to replace most of the functionality of the test harness.

  • Feature

    Decorator

    JIRA

    Comments

    Create & remove database

    yes

    DERBY-1557

    Setup SecurityManager and policy file

    no & yes

    Messing around with this it seems the best way is to have BaseTestCase install a security manager with the default policy and have decorators to remove the security manager or change the policy file for individual tests. Then a large number of decorators are not needed to run the tests and the exception cases (no security manager, separate policy file) will stand out. Adding a class functionTests.util.SecurityManagerSetup

    Cleanup database objects

    yes

    DERBY-1556

    new - allows databases to be shared across tests to decrease test running time

    Determine JDK environment

    no

    DERBY-1638 for JDBC level

    E.g. isFoundation() etc., is JDK 1.3

    Boot & shutdown network server

    yes

    DERBY-1712

    At least within the same JVM can be a decorator

    Setup & remove extin, extout, extout folders

    yes

    (see DERBY-1656 )

    Copy & remove support files

    yes

    In some cases files can be accessed directly as resources instead of copying them. See the test resource methods in BaseTestCase

    Set & clear derby properties

    yes

    Replacement for the <test>_derby.properties, system properties implemented in class SystemPropertyTestSetup

    Shutdown database/server

    no

    DERBY-1826

    Support running junit tests directly with a remote server

    no

    DERBY-1973

Folder Policy

It would be good to have some up-front policy about how folders are used, to provide consistency and help with good security manager testing. See:

Old Harness Suite status

Progress on removing tests from the old suites, with the assumption they are converted to JUnit tests.

Suite

Initial Test Count

Remaining Test Count (2007/02/15)

2013/09/26

Comments

demo

2

2

0

see DERBY-2903 and DERBY-3294

derbylang

148

91 (~70% .sql tests)

37 (34 .sql)

LangJunitTestConversion

derbynetclientmats

8

6

4 ( 1 junit running in old harness)

ClientJunitTestConversion (derbynet/* only)

derbynetmats

45

42

0

ClientJunitTestConversion (derbynet/* only)

derbytools

15

15

2

encodingTests

4

4

1

encryption

3

3

2

encryptionAES

2

2

0

encryptionAll

8

8

5

DERBY-1001, DERBY-3711. For an encryption Decorator (among other things), see DERBY-1952.

encryptionBlowfish

3

3

1

encryptionCFB

3

3

1

encryptionDES

3

3

1

encryptionECB

3

3

1

encryptionOFB

3

3

1

i18nTest

7

7

0 (removed)

j9derbynetmats

30

0(ignore)

old suite for running on a specific jvm

jdbc20

10

5

0 (removed)

JdbcapiJunitTestConversion

jdbc40

8

4

0(removed)

jdbcapi

22

15

0 (removed)

JdbcapiJunitTestConversion

jdbcxa40

2

0 (removed)

jdk14

9

7

0 (removed)

JdbcapiJunitTestConversion

largeDataTests

1

1

0 (removed)

Not in derbyall

multi

1

1

0 (removed)

DERBY-1764

nist

125

0 (removed)

DERBY-2007

propertyinfo

1

1

1

simpledemo

1

1

1

storemats

6

6

1

storemore

38

38

16

storerecovery

13

13

9

storetests

8

8

5

storeunit

34

34

34

unit

4

4

4

upgrade

1

1

0 (removed)

DERBY-2217

xa

11

11

5

JdbcapiJunitTestConversion

xmlTests

2

0 (removed)

DERBY-1758

Total

344 test runs with 306 unique tests

134 test runs

Test Conversions

ConvertLobTestsToJunit

ConvertSqlScriptTestsToJunit

KillDerbyTestHarness (last edited 2013-09-26 16:29:27 by MyrnavanLunteren)