Derby Public Interfaces & Forward Compatibility

WORK IN PROGESS

Initial set of public interfaces from an e-mail discussion on the derby-dev list. The list is subject to change.

Goal

The goal is to allow any application written against the public interfaces of an older version of Derby to run, without any changes, against a newer version of Derby.

Detailed information on Derby's versioning scheme.

Exceptions

There will be occasions where the Derby community does introduce regressions or what might be considered regressions by some. Examples of such exceptions include fixing incorrect implementations of a standard or a correctness bug (for example a query returning the wrong results). Such changes may break existing applications. We will strive to avoid such changes but they may occur. Any such change should be a deliberate choice by the community, not by accident. If at all possible such changes should be deferred to minor release boundaries.

Specific examples:

• Disabling ResultSet.getXXXStream to be called twice on the same column in the same row in 10.2. Previously 10.1 and 10.1 allowed this but returned the incorrect data. It seems a reasonable choice as it stops incorrect behaviour and is in-line with the JDBC spec.
  
• Dropping support of JDK 1.3/JDBC2.1. Technically a regression but it seems unlikely that by the time the community does this that there will be any demand for JDK 1.3. (We've voted that it's ok to do this in 10.3, but someone has to scratch that itch)
  

• The Derby community voted that the client driver should match embedded for implementation defined behaviour.
  

We will mark any fixes or product changes that intentionally change product behaviour and might affect existing applications by checking the Existing Application Impact checkbox. Also the "Release Note Needed" box will be checked as well and user impact described in ReleaseNoteFormat.

Users of Derby have several options under the hopefully rare occasion that a new release breaks their application:

• Fix their application
• Use the old release (i.e., don't upgrade)
• Pay someone who is supporting Derby to fix their problem.
• Get involved in the Derby community and suggest/provide a fix to their problem.
• Patch the source in their local copy of the Derby source code by reverting the fix that causes the problem for them.
  

What is an Interface?

An interface is any "interaction point" of the software, something that customers may touch and may therefore depend upon. Each interface is assigned a specific stability level which communicates the guarantee, or the contract, to how and when these interfaces will change in a non-compatible way. Here are some examples of interfaces:

• Application Programmatic Interfaces (APIs)
• Command Line Interfaces (CLIs). A CLI is actually often split up into sub-interfaces
  • CLI arguments
  • stdin
  • stdout
  • error codes
• File formats
• File names
• Network port numbers
• Network protocols
• Environment variables
• Directory locations
• Jar file names
• Class names
  

Stability Levels

To help describe the types of stability guarantees Derby provides for a given interface, we identify the following stability levels. These definitions are inspired by the OpenSolaris interface taxonomy

Stable

• Incompatible change allowed in major release (X.0)
  

This is an interface which we expect others to depend upon to remain stable enough for them to build their own products or systems on top of. Most of our public interfaces should be given the stability level of Stable. Stable interfaces should never change between minor releases, and even between major releases changes should be extremely rare.

Note that an interface must be documented before it can be classified Stable or Unstable. Any undocumented interface can only be given a Private or Private Stable stability level. New features can not be considered Stable or Unstable until they are documented.

It is important to note that having a major release does not mean that the release will have incompatible changes to a Stable interface. It is just that a major release is the only time where an incompatibility can be introduced on a Stable interface

Any changes to Stable interfaces should be documented in the release notes.

Standard

• Incompatible change allowed in major release (X.0)
  

These are interfaces that we make available which are produced by a standards body, or which is an industry standard that we feel is unlikely to change in incompatible ways.

Any changes to Standard interfaces should be documented in the release notes.

Unstable

• Incompatible change allowed in minor release (x.Y)
  

This is usually for new functionality that we feel is experimental or unstable. Documentation should make it very clear that this interface may change between minor releases. Changes to unstable interfaces should be documented in the release notes.

Note that an interface must be documented before it can be classified Stable or Unstable. Any undocumented interface can only be given a Private or Private Stable stability level. New features can not be considered Stable or Unstable until they are documented.

Private

• Incompatible change allowed at any time
  

Private interfaces are interfaces which are not documented for external users, and generally users should not be depending upon, and which can change at any time.

Private Stable

• Incompatible changes allowed at major release (X.0)
  

Some private interfaces require stability even though they are not documented in the user documentation and not supported for direct use by the users of Derby, in order for us to guarantee compatibility across releases. For instance, the on-disk format for Derby databases must remain stable across minor releases and in general changes to this format should be rare, because of the impact on upgrade.

Deprecated

• Incompatible change allowed in minor release (x.Y)
  

An interface can be downgraded from some other classification (such as Stable) to Deprecated to encourage users to move to a newer interface that replaces this interface. An interface marked as deprecated must continue to be supported until the next major release. An interface can be reclassified as deprecated at any minor release boundary, but requires consensus approval within the Derby community. The interface must be clearly documented as deprecated in the documentation and/or release notes, and if possible warnings should be issued when an interface is used that it is deprecated (with the ability to turn these warnings off).

Interfaces that have been reclassified to Deprecated should be documented in the release notes.

A deprecated interface may be removed from the project after four minor and/or major releases.

What is an Incompatible Change?

What constitutes an incompatible change depends upon the type of interface

Changes to Java Classes

We are going to follow the definitions for what types of changes are considered acceptable for what types of releases as defined by the Apache Jakarta Commons project. See

http://jakarta.apache.org/commons/releases/versioning.html

Changes to Stored Procedures

System procedures and metadata procedures are both Stable interfaces, so it is important to understand what entails a compatible change for stored procedures.

• Incompatible:
  • Changing the name of the stored procedure
  • Changing the signature (input params, output params, and return value)
  • Changing the name, type or order of existing columns in the result set
  • Changing any documented side effects of the stored procedure
    
• Compatible:
  • Adding new allowable values to the input parameters
  • Adding new allowable values to the return code
  • Adding columns to the result set
  • Adding new allowable values to the existing columns of the result set
    

In general, if you need to make a change for new functionality that is incompatible, it is recommended that you create a new stored procedure and have the newer version of the code use that new stored procedure.

Changes to Database Tables

Documented system tables are a Stable interface, so care must be taken when considering changing these

• Incompatible:
  • Changing the name of the table
  • Changing the names, types, or order of existing columns in the table
  • Changing the semantics of the table - what it represents
    
• Compatible:
  • Adding new allowable values to an existing column
  • Adding new columns to the table
  • Adding new indexes to the table
  • Changing the primary key for the table
    

Changes to the system tables cannot occur for soft upgrade. Associated upgrade code and tests will be added for compatible changes.

Incompatible Changes for Data File Formats

I need help with this one

Changes to Configuration Properties

• Incompatible:
  • Changing the name of the property
  • Removing a property
  • Changing the default value of a property
  • Changing the valid values for a property
  • Changing the meaning of a property
    
• Compatible
  • Adding a new property
  • Adding support for new valid values for a property
    

Changes to Command Line Interfaces (CLI)

These are both scripts and Java classes that implement a command-line interface

A CLI is subdvided into multiple interfaces: its name, its arguments, its return code, stdin, stdout, environment variables it uses, its error messages. Each of these may have a different stability level. I describe an incompatible change for each of these

• Name: Cannot change the name of the CLI (script name or class name)
• Arguments: Cannot change names, valid values or semantics of existing arguments. Cannot remove arguments. Can add new arguments.
• Return code: Cannot change semantics of or remove existing return codes. Can add new return codes.
• Stdin: Changing format or expected values from stdin is an incompatible change
• Stdout: Changing format of stdout is an incompatible change. Usually this is not a Stable interface.
• Environment variables: can not remove use of or change interpretation of existing environment variables. Can add support for new environment variables.
• Error messages: can not change text of existing error messages. Can add new error messages.
  

Changes to a Network Protocol

DRDA is the current protocol we use, and is the underpinning of client/server compatibility

Generally the network protocol needs to be versioned and the client and server need to negotiate to a given version level. The types of changes described below are changes to an existing version of the protocol.

Compatible Change

• Add unimplemented DRDA support to client or server with appropriate logic to trigger it only for Derby versions that support it.
  

Incompatible Change

• Add unimplemented DRDA Support to client or server that might impact compatiblity with earlier releases of client or server.
  

Other Miscellaneous Interfaces

Here we describe what an incompatible change means for other miscellaneous interfaces

• Port Number : Cannot change the default port number

message. Can add new messages.

• Miscellaneous file or string formats : Cannot change the semantics or format in any way and remain compatible. Normally this is handled by versioning the format.
  

Interface Table

This table describes each of the interfaces Derby provides, and the stability guarantee we provide for that interface. This is the interface table for the 10.1 release. This is subject to change (in a compatible way, meaning new interfaces may be added or old interfaces may be upgraded to a more stable level) at each minor release boundary.

NOTE: This table is still a draft form and is subject to change

NOTE: Any interface that is in the public documentation is considered a Stable interface unless explicitly called out as otherwise in the table below

JVM Support and Version Interoperability

• Platform support
  • J2SE 1.3.1/1.4/1.5 support (Note: 10.2 is the last supported release for 1.3.1)
  • J2ME/CDC/Foundation support
  • Pure Java to enable run-anywhere
    
• Derby's versioning of its on-disk database format is described here:
  *http://db.apache.org/derby/papers/versionupgrade.html
  
• Derby will at a minimum negotiate down to the lower interface revision level:
  • When different versions of Derby client and server are used together (in the same or different JVM's)
  • When different jvm versions are used on client and server.
    

FAQ

Would this set of rules drive the decision of when to declare a major release versus a minor release?

They would be one of the deciding factors, likely a major deciding factor. Other factors will likely be involved. It is more than likely that we may want to declare a new major release even though we have not introduced any incompatibilities.

We currently don't have any rules about when we would change to 11.0 from the current 10.x pattern.

Making an incompatible change to a Stable or Standard interface would require us to change to 11.0.

Would it be because someone changed some "Standard" interface in an incompatible way that we move to 11.0?

If a standards body makes an incompatible change to a standard interface that we rely on, and we make the choice to upgrade to the new, incompatible revision of that standard, then, yes, we should move to a new major release. A standards body making an incompatible change to a standard should be fairly rare, however.

If no-one made such a change, would be be having 10.14 in a number of years?

Yes, possibly, unless the community has other reasons to upgrade the version.