SecurityManager testing

Update

Most of this document is written before the work was started, now most of the tests run under the SecurityManager. Further down in this document you can see the state per test suite.

Overview

Currently any test that runs the network server as a separate java executable uses the security manager and a policy file (nwsvr.policy) for the network server's JVM. This is a good step but I (Dan Debrunner) have been looking to improve the situation to run most tests under the security manager (by default). Discussion is under this thread http://mail-archives.apache.org/mod_mbox/db-derby-dev/200510.mbox/%3c434946C8.6000706@debrunners.com%3e

Goal

My goal is to ensure that running derbyall tests all Derby's functionality works with a security manager and a correctly, minimally configured policy file. By minimally I mean just the fewset set of permissions required, hopefully in-line with the documentation. E.g. a policy file that allowed all permissions would work but would not be a good test of Derby.

Current Behaviour

The harness determins a code base from the class path and sets this as the property csinfo.codebase for the policy file. This code base will correspond to either the classes directory or the directory containing derby jar files. The policy file (nwsvr.policy) then has a set of permissions that are granted to the code base, which is the entire derby code.

Issue with the current behaviour

Granting permissions to a single code base that includes all the derby code can lead to hidden bugs, especially due to the fact the test harness does not need to be secure and is not designed that way, whereas the other derby components need to be secure. For example, the test harness needs to read and modify system properties so that permission is granted, now the engine should not be needing that permission but due to the single code base in the policy file, it has that permission and now silently could start to depend on it.

Proposed change

I have a more specific properties file (derby_tests.policy) that has a section for each derby jar file with code, and grants only the required (and reasonable) permissions for each jar. E.g. derby.jar is not granted any socket related permissions and derbynet.jar is not granted any access to the database files. With this file incorrect permissions that need to be granted are obvious and bugs can be entered against them.

In addition a section in the policy file will exist for the classes directory with a superset of the permissions. This is for when the tests are run directly out of the classes directory.

There is a chance that the tests will pass under the classes and fail with the jars with a contribution or change. The risk is small (and most likely would point to a bug). Comments can be added in the policy file indicating if changes are made to the classes section that similar changes might be needed to the jar sections and tests should also be run using the jars.

Justification

I strongly believe that the single code base approach today is not sufficent for Derby's security testing, due to the potential for hidden bugs. In switching to this new style I think I've found three bugs so far against Derby related to permissions, including one potentially serious one where a create index fails due to no access to a temp file. I need to look at that one more. I think the number of bugs (so far) shows the change is a good one.

Status

Jira Entry

Comments

DERBY-615

Entry for running all tests under Security Manager (this effort)

DERBY-537

Found by user

DERBY-538

Work item for 537

DERBY-446

Found by ad-hoc testing

DERBY-616

Fixed. Found by this effort

DERBY-622

Found by this effort

DERBY-623

Fixed. Found by this effort

DERBY-624

Fixed. Found by this effort

DERBY-613

Fixed by changes to DERBY-375 . Found by this effort (not a security issue)

DERBY-626

Fixed. Found by this effort, regression hidden by bug DERBY-622 and lack of testing with Security Manager

DERBY-647

Fixed. Found by this effort (not a security issue)

DERBY-656

Fixed. Found by user

DERBY-709

Fixed. Found by this effort

DERBY-811

Found by this effort

DERBY-857

Found by this effort

DERBY-858

Found by this effort (not a security issue)

Table of tests running under the SecurityManager by suite.

Values in the columns represent the number of tests running using the SecurityManager out of the total for the suite. Counts may be approximate.

Test Suite

Test JVM

Network Server JVM

Comments

derbylang

147/150

n/a

derbynetclientmats

77/90

88/88

propertyinfo

1/1

n/a

storeall

90/96

n/a

xa

11/11

n/a

unit

4/4

n/a

jdbcapi

24/26

n/a

jdbc20

10/10

n/a

jdk14

10/10

n/a

demo

3/3

n/a

nist

126/126

n/a

useProcess=false

encryptionAll

29/29

n/a

multi

1/1

n/a

derbytools

11/13

n/a

i18nTest

7/7

n/a

Total (start)

95.4% (551/577)

100%

derbynetmats

0/73

73/73

DB2 JCC tests - enable later, not testing Derby functionality

Progress Table

Trunk Revision

10.1 Revision

Test JVM

Network Server JVM

Comment

Start

n/a

0%

100%

Initial numbers using nwsvr.policy

2005/10/13
320762

326221

0%

100%

Switch to new policy file derby_tests.policy

2005/10/14
321124

326282

0%

100%

Fix DERBY-624

2005/10/19
326727

330110

0%

100%

Fix DERBY-626

2005/10/19
326731

-

0%

100%

Disable number of tests from security manager, preparing for 326736

2005/10/19
326733

-

0%

100%

Minor cleanup in test harness

2005/10/19
326736

-

58%

100%

Enable security manager for most tests, except noSecurityManager=true, JCC and nist (useProcess=true)

2005/10/20
327406

-

0%

100%

Disabled security manager for most tests, due to failures in testing

2005/10/25
328515

-

0%

100%

Fixed DERBY-647 incorrect output file location for stress.multi

2005/10/28
329310

-

0%

100%

Change unit tests to use System.setProperty

2005/10/31
326736

-

58%

100%

Second attempt, enable security manager for most tests, except noSecurityManager=true, JCC and nist (useProcess=true)

2005/11/15
344313

-

58%

100%

Fix encrpytion tests failing in JDK 1.3

2005/11/23
348501

-

58%

100%

Enable two tests under security manager

2005/11/23
348532

-

58%

100%

Enable two i18n tests under security manager

2005/12/27

-

66%

100%

Don't count running DB2 driver under SecurityManager since that is not Derby code

2006/01/03
365776

-

89%

100%

Run useprocess=false tests under a SecurityManager (just the nist suite)

2006/01/23 various

-

91.6%

100%

Various small commits and re-count the tests running under SecurityManager

2006/02/27 various

-

93.6%

100%

Mainly fixing DERBY-709 to enable backup tests under SecurityManager

2006/04/07 various

-

95.4%

100%

Mainly fixing DERBY-616

SecurityManagerTesting (last edited 2009-09-20 22:11:20 by localhost)