Differences between revisions 25 and 26
Revision 25 as of 2006-04-07 20:27:00
Size: 10454
Editor: DanDebrunner
Comment:
Revision 26 as of 2009-09-20 22:11:20
Size: 10527
Editor: localhost
Comment: converted to 1.6 markup
Deletions are marked like this. Additions are marked like this.
Line 80: Line 80:
||<#FFFF00> [http://issues.apache.org/jira/browse/DERBY-615 DERBY-615] || Entry for running all tests under Security Manager (this effort) || ||<#FFFF00> [[http://issues.apache.org/jira/browse/DERBY-615|DERBY-615]] || Entry for running all tests under Security Manager (this effort) ||
Line 82: Line 82:
||<#FF0000> [http://issues.apache.org/jira/browse/DERBY-537 DERBY-537] || Found by user ||
||<#FFFF00> [http://issues.apache.org/jira/browse/DERBY-538 DERBY-538] || Work item for 537 ||
||<#FF0000> [http://issues.apache.org/jira/browse/DERBY-446 DERBY-446] || Found by ad-hoc testing ||
||<#00FF00> [http://issues.apache.org/jira/browse/DERBY-616 DERBY-616] || Fixed. Found by this effort ||
||<#FF0000> [http://issues.apache.org/jira/browse/DERBY-622 DERBY-622] || Found by this effort ||
||<#00FF00> [http://issues.apache.org/jira/browse/DERBY-623 DERBY-623] || Fixed. Found by this effort ||
||<#00FF00> [http://issues.apache.org/jira/browse/DERBY-624 DERBY-624] || Fixed. Found by this effort ||
||<#00FF00> [http://issues.apache.org/jira/browse/DERBY-613 DERBY-613] || Fixed by changes to [http://issues.apache.org/jira/browse/DERBY-375 DERBY-375] . Found by this effort (not a security issue) ||
||<#00FF00> [http://issues.apache.org/jira/browse/DERBY-626 DERBY-626] || Fixed. Found by this effort, regression hidden by bug DERBY-622 and lack of testing with Security Manager ||
||<#00FF00> [http://issues.apache.org/jira/browse/DERBY-647 DERBY-647] || Fixed. Found by this effort (not a security issue) ||
||<#00FF00> [http://issues.apache.org/jira/browse/DERBY-656 DERBY-656] || Fixed. Found by user ||
||<#00FF00> [http://issues.apache.org/jira/browse/DERBY-709 DERBY-709] || Fixed. Found by this effort ||
||<#FF0000> [http://issues.apache.org/jira/browse/DERBY-811 DERBY-811] || Found by this effort ||
||<#FF0000> [http://issues.apache.org/jira/browse/DERBY-857 DERBY-857] || Found by this effort ||
||<#FF0000> [http://issues.apache.org/jira/browse/DERBY-858 DERBY-858] || Found by this effort (not a security issue) ||
||<#FF0000> [[http://issues.apache.org/jira/browse/DERBY-537|DERBY-537]] || Found by user ||
||<#FFFF00> [[http://issues.apache.org/jira/browse/DERBY-538|DERBY-538]] || Work item for 537 ||
||<#FF0000> [[http://issues.apache.org/jira/browse/DERBY-446|DERBY-446]] || Found by ad-hoc testing ||
||<#00FF00> [[http://issues.apache.org/jira/browse/DERBY-616|DERBY-616]] || Fixed. Found by this effort ||
||<#FF0000> [[http://issues.apache.org/jira/browse/DERBY-622|DERBY-622]] || Found by this effort ||
||<#00FF00> [[http://issues.apache.org/jira/browse/DERBY-623|DERBY-623]] || Fixed. Found by this effort ||
||<#00FF00> [[http://issues.apache.org/jira/browse/DERBY-624|DERBY-624]] || Fixed. Found by this effort ||
||<#00FF00> [[http://issues.apache.org/jira/browse/DERBY-613|DERBY-613]] || Fixed by changes to [[http://issues.apache.org/jira/browse/DERBY-375|DERBY-375]] . Found by this effort (not a security issue) ||
||<#00FF00> [[http://issues.apache.org/jira/browse/DERBY-626|DERBY-626]] || Fixed. Found by this effort, regression hidden by bug DERBY-622 and lack of testing with Security Manager ||
||<#00FF00> [[http://issues.apache.org/jira/browse/DERBY-647|DERBY-647]] || Fixed. Found by this effort (not a security issue) ||
||<#00FF00> [[http://issues.apache.org/jira/browse/DERBY-656|DERBY-656]] || Fixed. Found by user ||
||<#00FF00> [[http://issues.apache.org/jira/browse/DERBY-709|DERBY-709]] || Fixed. Found by this effort ||
||<#FF0000> [[http://issues.apache.org/jira/browse/DERBY-811|DERBY-811]] || Found by this effort ||
||<#FF0000> [[http://issues.apache.org/jira/browse/DERBY-857|DERBY-857]] || Found by this effort ||
||<#FF0000> [[http://issues.apache.org/jira/browse/DERBY-858|DERBY-858]] || Found by this effort (not a security issue) ||
Line 130: Line 130:
|| 2005/10/13 [[BR]] [http://svn.apache.org/viewcvs?view=rev&rev=320762 320762 ] || [http://svn.apache.org/viewcvs?view=rev&rev=326221 326221] ||<#FF0000> 0% ||<#00FF00> 100% || Switch to new policy file derby_tests.policy ||
|| 2005/10/14 [[BR]] [http://svn.apache.org/viewcvs?view=rev&rev=321124 321124] || [http://svn.apache.org/viewcvs?view=rev&rev=326282 326282] ||<#FF0000> 0% ||<#00FF00> 100% || Fix [http://issues.apache.org/jira/browse/DERBY-624 DERBY-624] ||
|| 2005/10/19 [[BR]] [http://svn.apache.org/viewcvs?view=rev&rev=326727 326727] || [http://svn.apache.org/viewcvs?view=rev&rev=330110 330110] ||<#FF0000> 0% ||<#00FF00> 100% || Fix [http://issues.apache.org/jira/browse/DERBY-626 DERBY-626] ||
|| 2005/10/19 [[BR]] [http://svn.apache.org/viewcvs?view=rev&rev=326731 326731] || - ||<#FF0000> 0% ||<#00FF00> 100% || Disable number of tests from security manager, preparing for 326736 ||
|| 2005/10/19 [[BR]] [http://svn.apache.org/viewcvs?view=rev&rev=326733 326733] || - ||<#FF0000> 0% ||<#00FF00> 100% || Minor cleanup in test harness ||
|| 2005/10/19 [[BR]] [http://svn.apache.org/viewcvs?view=rev&rev=326736 326736] || - ||<#FF0000> 58% ||<#00FF00> 100% || Enable security manager for most tests, except noSecurityManager=true, JCC and nist (useProcess=true) ||
|| 2005/10/20 [[BR]] [http://svn.apache.org/viewcvs?view=rev&rev=327406 327406] || - ||<#FF0000> 0% ||<#00FF00> 100% || Disabled security manager for most tests, due to failures in testing ||
|| 2005/10/25 [[BR]] [http://svn.apache.org/viewcvs?view=rev&rev=328515 328515] || - ||<#FF0000> 0% ||<#00FF00> 100% || Fixed [http://issues.apache.org/jira/browse/DERBY-647 DERBY-647] incorrect output file location for stress.multi ||
|| 2005/10/28 [[BR]] [http://svn.apache.org/viewcvs?view=rev&rev=329310 329310] || - ||<#FF0000> 0% ||<#00FF00> 100% || Change unit tests to use System.setProperty ||
|| 2005/10/31 [[BR]] [http://svn.apache.org/viewcvs?view=rev&rev=326736  326736] || - ||<#FF0000> 58% ||<#00FF00> 100% || Second attempt, enable security manager for most tests, except noSecurityManager=true, JCC and nist (useProcess=true) ||
|| 2005/11/15 [[BR]] [http://svn.apache.org/viewcvs?view=rev&rev=344313 344313 ] || - ||<#FF0000> 58% ||<#00FF00> 100% || Fix encrpytion tests failing in JDK 1.3 ||
|| 2005/11/23 [[BR]] [http://svn.apache.org/viewcvs?view=rev&rev=348501 348501 ] || - ||<#FF0000> 58% ||<#00FF00> 100% || Enable two tests under security manager ||
|| 2005/11/23 [[BR]] [http://svn.apache.org/viewcvs?view=rev&rev=348532 348532 ] || - ||<#FF0000> 58% ||<#00FF00> 100% || Enable two i18n tests under security manager ||
|| 2005/10/13 <<BR>> [[http://svn.apache.org/viewcvs?view=rev&rev=320762|320762 ]] || [[http://svn.apache.org/viewcvs?view=rev&rev=326221|326221]] ||<#FF0000> 0% ||<#00FF00> 100% || Switch to new policy file derby_tests.policy ||
|| 2005/10/14 <<BR>> [[http://svn.apache.org/viewcvs?view=rev&rev=321124|321124]] || [[http://svn.apache.org/viewcvs?view=rev&rev=326282|326282]] ||<#FF0000> 0% ||<#00FF00> 100% || Fix [[http://issues.apache.org/jira/browse/DERBY-624|DERBY-624]] ||
|| 2005/10/19 <<BR>> [[http://svn.apache.org/viewcvs?view=rev&rev=326727|326727]] || [[http://svn.apache.org/viewcvs?view=rev&rev=330110|330110]] ||<#FF0000> 0% ||<#00FF00> 100% || Fix [[http://issues.apache.org/jira/browse/DERBY-626|DERBY-626]] ||
|| 2005/10/19 <<BR>> [[http://svn.apache.org/viewcvs?view=rev&rev=326731|326731]] || - ||<#FF0000> 0% ||<#00FF00> 100% || Disable number of tests from security manager, preparing for 326736 ||
|| 2005/10/19 <<BR>> [[http://svn.apache.org/viewcvs?view=rev&rev=326733|326733]] || - ||<#FF0000> 0% ||<#00FF00> 100% || Minor cleanup in test harness ||
|| 2005/10/19 <<BR>> [[http://svn.apache.org/viewcvs?view=rev&rev=326736|326736]] || - ||<#FF0000> 58% ||<#00FF00> 100% || Enable security manager for most tests, except noSecurityManager=true, JCC and nist (useProcess=true) ||
|| 2005/10/20 <<BR>> [[http://svn.apache.org/viewcvs?view=rev&rev=327406|327406]] || - ||<#FF0000> 0% ||<#00FF00> 100% || Disabled security manager for most tests, due to failures in testing ||
|| 2005/10/25 <<BR>> [[http://svn.apache.org/viewcvs?view=rev&rev=328515|328515]] || - ||<#FF0000> 0% ||<#00FF00> 100% || Fixed [[http://issues.apache.org/jira/browse/DERBY-647|DERBY-647]] incorrect output file location for stress.multi ||
|| 2005/10/28 <<BR>> [[http://svn.apache.org/viewcvs?view=rev&rev=329310|329310]] || - ||<#FF0000> 0% ||<#00FF00> 100% || Change unit tests to use System.setProperty ||
|| 2005/10/31 <<BR>> [[http://svn.apache.org/viewcvs?view=rev&rev=326736|326736]] || - ||<#FF0000> 58% ||<#00FF00> 100% || Second attempt, enable security manager for most tests, except noSecurityManager=true, JCC and nist (useProcess=true) ||
|| 2005/11/15 <<BR>> [[http://svn.apache.org/viewcvs?view=rev&rev=344313|344313 ]] || - ||<#FF0000> 58% ||<#00FF00> 100% || Fix encrpytion tests failing in JDK 1.3 ||
|| 2005/11/23 <<BR>> [[http://svn.apache.org/viewcvs?view=rev&rev=348501|348501 ]] || - ||<#FF0000> 58% ||<#00FF00> 100% || Enable two tests under security manager ||
|| 2005/11/23 <<BR>> [[http://svn.apache.org/viewcvs?view=rev&rev=348532|348532 ]] || - ||<#FF0000> 58% ||<#00FF00> 100% || Enable two i18n tests under security manager ||
Line 144: Line 144:
|| 2006/01/03 [[BR]] [http://svn.apache.org/viewcvs?view=rev&rev=365776 365776 ] || - ||<#FFFF00> 89% ||<#00FF00> 100% || Run useprocess=false tests under a !SecurityManager (just the nist suite) || || 2006/01/03 <<BR>> [[http://svn.apache.org/viewcvs?view=rev&rev=365776|365776 ]] || - ||<#FFFF00> 89% ||<#00FF00> 100% || Run useprocess=false tests under a !SecurityManager (just the nist suite) ||

SecurityManager testing

Update

Most of this document is written before the work was started, now most of the tests run under the SecurityManager. Further down in this document you can see the state per test suite.

Overview

Currently any test that runs the network server as a separate java executable uses the security manager and a policy file (nwsvr.policy) for the network server's JVM. This is a good step but I (Dan Debrunner) have been looking to improve the situation to run most tests under the security manager (by default). Discussion is under this thread http://mail-archives.apache.org/mod_mbox/db-derby-dev/200510.mbox/%3c434946C8.6000706@debrunners.com%3e

Goal

My goal is to ensure that running derbyall tests all Derby's functionality works with a security manager and a correctly, minimally configured policy file. By minimally I mean just the fewset set of permissions required, hopefully in-line with the documentation. E.g. a policy file that allowed all permissions would work but would not be a good test of Derby.

Current Behaviour

The harness determins a code base from the class path and sets this as the property csinfo.codebase for the policy file. This code base will correspond to either the classes directory or the directory containing derby jar files. The policy file (nwsvr.policy) then has a set of permissions that are granted to the code base, which is the entire derby code.

Issue with the current behaviour

Granting permissions to a single code base that includes all the derby code can lead to hidden bugs, especially due to the fact the test harness does not need to be secure and is not designed that way, whereas the other derby components need to be secure. For example, the test harness needs to read and modify system properties so that permission is granted, now the engine should not be needing that permission but due to the single code base in the policy file, it has that permission and now silently could start to depend on it.

Proposed change

I have a more specific properties file (derby_tests.policy) that has a section for each derby jar file with code, and grants only the required (and reasonable) permissions for each jar. E.g. derby.jar is not granted any socket related permissions and derbynet.jar is not granted any access to the database files. With this file incorrect permissions that need to be granted are obvious and bugs can be entered against them.

In addition a section in the policy file will exist for the classes directory with a superset of the permissions. This is for when the tests are run directly out of the classes directory.

There is a chance that the tests will pass under the classes and fail with the jars with a contribution or change. The risk is small (and most likely would point to a bug). Comments can be added in the policy file indicating if changes are made to the classes section that similar changes might be needed to the jar sections and tests should also be run using the jars.

Justification

I strongly believe that the single code base approach today is not sufficent for Derby's security testing, due to the potential for hidden bugs. In switching to this new style I think I've found three bugs so far against Derby related to permissions, including one potentially serious one where a create index fails due to no access to a temp file. I need to look at that one more. I think the number of bugs (so far) shows the change is a good one.

Status

Jira Entry

Comments

DERBY-615

Entry for running all tests under Security Manager (this effort)

DERBY-537

Found by user

DERBY-538

Work item for 537

DERBY-446

Found by ad-hoc testing

DERBY-616

Fixed. Found by this effort

DERBY-622

Found by this effort

DERBY-623

Fixed. Found by this effort

DERBY-624

Fixed. Found by this effort

DERBY-613

Fixed by changes to DERBY-375 . Found by this effort (not a security issue)

DERBY-626

Fixed. Found by this effort, regression hidden by bug DERBY-622 and lack of testing with Security Manager

DERBY-647

Fixed. Found by this effort (not a security issue)

DERBY-656

Fixed. Found by user

DERBY-709

Fixed. Found by this effort

DERBY-811

Found by this effort

DERBY-857

Found by this effort

DERBY-858

Found by this effort (not a security issue)

Table of tests running under the SecurityManager by suite.

  • Test JVM - JVM running the JDBC/ij test
  • Network Server JVM - JVM running the network server when the client is in a separate JVM.

Values in the columns represent the number of tests running using the SecurityManager out of the total for the suite. Counts may be approximate.

Test Suite

Test JVM

Network Server JVM

Comments

derbylang

147/150

n/a

derbynetclientmats

77/90

88/88

propertyinfo

1/1

n/a

storeall

90/96

n/a

xa

11/11

n/a

unit

4/4

n/a

jdbcapi

24/26

n/a

jdbc20

10/10

n/a

jdk14

10/10

n/a

demo

3/3

n/a

nist

126/126

n/a

useProcess=false

encryptionAll

29/29

n/a

multi

1/1

n/a

derbytools

11/13

n/a

i18nTest

7/7

n/a

Total (start)

95.4% (551/577)

100%

derbynetmats

0/73

73/73

DB2 JCC tests - enable later, not testing Derby functionality

Progress Table

Trunk Revision

10.1 Revision

Test JVM

Network Server JVM

Comment

Start

n/a

0%

100%

Initial numbers using nwsvr.policy

2005/10/13
320762

326221

0%

100%

Switch to new policy file derby_tests.policy

2005/10/14
321124

326282

0%

100%

Fix DERBY-624

2005/10/19
326727

330110

0%

100%

Fix DERBY-626

2005/10/19
326731

-

0%

100%

Disable number of tests from security manager, preparing for 326736

2005/10/19
326733

-

0%

100%

Minor cleanup in test harness

2005/10/19
326736

-

58%

100%

Enable security manager for most tests, except noSecurityManager=true, JCC and nist (useProcess=true)

2005/10/20
327406

-

0%

100%

Disabled security manager for most tests, due to failures in testing

2005/10/25
328515

-

0%

100%

Fixed DERBY-647 incorrect output file location for stress.multi

2005/10/28
329310

-

0%

100%

Change unit tests to use System.setProperty

2005/10/31
326736

-

58%

100%

Second attempt, enable security manager for most tests, except noSecurityManager=true, JCC and nist (useProcess=true)

2005/11/15
344313

-

58%

100%

Fix encrpytion tests failing in JDK 1.3

2005/11/23
348501

-

58%

100%

Enable two tests under security manager

2005/11/23
348532

-

58%

100%

Enable two i18n tests under security manager

2005/12/27

-

66%

100%

Don't count running DB2 driver under SecurityManager since that is not Derby code

2006/01/03
365776

-

89%

100%

Run useprocess=false tests under a SecurityManager (just the nist suite)

2006/01/23 various

-

91.6%

100%

Various small commits and re-count the tests running under SecurityManager

2006/02/27 various

-

93.6%

100%

Mainly fixing DERBY-709 to enable backup tests under SecurityManager

2006/04/07 various

-

95.4%

100%

Mainly fixing DERBY-616

SecurityManagerTesting (last edited 2009-09-20 22:11:20 by localhost)