Apache Derby

Requirement Specifications for Monitoring and Management Extensions using JMX

Document Contents

1. Revision History
2. Disclaimer and Legal Information
3. About this document

• 3.1 Purpose
  3.2 Intended Audience
  3.3 Notations and Conventions
  3.4 Structure of this document
  3.5 A note about ranking
  

4. Assumptions and Dependencies
5. Requirement Analysis

• 5.1 Overview
  5.2 Current Scenario
  

  • 5.2.1 Embedded Instance
    5.2.2 Embedded in Server Framework
    5.2.3 Network Server
    

  5.3 Conclusion
  

6. Requirements Identified

• 6.1 Overview
  6.2 Features Identified
  6.3 Remote vs. Local
  6.4 Conclusion
  

7. References

1. Revision History

2. Disclaimer and Legal Information

Copyright 2005 The Apache Software Foundation or its licensors, as applicable.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

3. About this Document

3.1. Purpose

• This document identifies the requirements and design goals for adding Management extensions to Apache Derby as Google Summer of Code 2006 project. Consequently the document identifies a set of requirements, categorizes them and attempts to assign priority to the identified requirements for their implementation. However, the priorities assigned reflect my own personal views and are left open for the community to decide. The final priorities will be decided by the community as a whole. This document does not describe any high level architecture or implementation details. These will be described in a subsequent documents.

3.2. Intended Audience

• The intended audience includes a wide community of those interested in developing monitoring and management applications for the Apache Derby. It also includes the end users of the database who will ultimately use these features and the developers who might use these features for fine tuning their applications.

3.3. Notations and Conventions

• The documentation set uses the following conventions.

• Convention	Explanation	Example
  Monospace	Filename	MyFile.java
  	Directory Names and Path Names	\MyDir\MyDir2
  	Commands and Command line options	ecl-02
  	Method names, Classes, Interfaces in running text	Use the  createService()  method to...
  	Parameters or other placeholders	doSomething(int val1, int val2)
  Monospace bold	User input	[C:] dir
  Italics	Emphasis; Introducing or defining terms	The term access takes as its subject
  [ ]	Optional Items	-Fa[c]  Indicates these command-line options:  -Fa  and  -Fac
  {|}	Choice of one item from a selection of two or more items.	-aX{K | W | P} Indicates these command-line options:
  		-aXK
  		-aXW
  		-aXP

3.4. Structure of this document

• This section is divided into seven sections. Section one to four provide basic information about legal stuff and notational conventions used in this document. Section five describes what features currently exist in Derby to monitor and tune the performance of the system and why we need a better monitoring and management system. Section six then tries to identify features that might be useful to developers and administrators. While this section enumerates a whole bunch of features, this section also tries to assign a priority to their implementation. Section seven enlists the sources of information for this document.

3.5. A Note about Ranking

• Developers and Derby users who read this document are requested to assign their priorities to the features enlisted in section six. I have provided an initial version of priorities based on my personal opinion. Developers and Users are requested add a column to the table, add their names as coloumn heading and assign priorities to the features. This will help us in deciding which features have to be implemented and in which order.

4. Assumptions and Dependencies

• This document follows Java Coding conventions defined at http://java.sun.com/docs/codeconv/html/CodeConvTOC.doc.html

5. Requirement Analysis

5.1. Overview

• Apache Derby is a relational, light weight, embedded database developed entirely in Java. Being an embedded database it is very flexible in its use and can be used either embedded inside an application sharing the same JVM as the application or can be embedded inside a server framework (such as Geronimo) or can be deployed as a standalone server (with the Network server framework) in a traditional client server environment. However, currently derby does not support any mature monitoring and management features which lets a developer or an administrator view statistics about the health of a system and fine tune system parameters via a management console or application. Such functionality is indispensable for any database system of any scale for a multitude of reasons. This section tries to identify the requirements for such a monitoring and management framework.

5.2. Current Scenario

• The following section identifies the what is currently supported in derby with reference to:

• 1. Monitoring the state the system
  

  • 
    

  2. Tuning system parameters
  

  • 
    

• Although current features can not be termed as monitoring or management features, but they do provide some primitive functionality. The following section describes these in details with respect to the three ways Derby can be used - embedded and as network server.

5.2.1. Embedded Instance

• When Derby runs as an embedded database engine sharing the same JVM there are quite a few ways for setting the properties for a Derby instance, both system wide and database. Such properties can be set either in the derby.properties file or can be at run time by:
  

  • 
    

  • Using parameters while starting application e.g. java -Dderby.xxx=something YourApplication.
    
    

  • Using API's provided.
    
    

• While these features let an application read or update system properties, there is no interactive way of doing it. Further, these can not be termed as monitoring and management features. What is needed is a functionality which a developer can use to tune the system while examining its state from outside the application. A typical usage scenario would be when a user runs his application and then starts the management console to view the current state of system or to debug his application. Without writing any code in his application he is able to change the system state and view how the system responds. Such functionality should be available to all applications without any coding. The developer/administrator should be able to tune the system at runtime without changing his code.

5.2.2. Embedded inside a server framework

• 
  Essentially, this is same as 5.2.1 where the application server or some other server framework becomes client for Derby. Most application servers come with a management console that lets administrators and developers configure various system wide settings locally or remotely. Since Derby does not provide any management framework that integrates into application server frameworks, developers of such systems using Derby have to write wrappers around Derby API calls to provide any management functionality, if at all.

• These servers usually generate a lot of runtime statistics and building monitoring extensions right into Derby will make integration much more transparent. The need for a standard management framework is accentuated in this case. Currently, the only way a developer can change system settings is via API calls or derby.properties file.

5.2.3. Network Server

• In this mode Derby runs in embedded mode, wrapped inside a server framework that multiple remote clients connect to. Derby can run as full fledged relational database server with the help of bundled server framework. Multiple remote clients can connect to database server simultaneously while the server takes care of concurrency issues and ACID properties.

• Again, .properties file and API calls are the only mechanisms to tune the system. However, in this mode Derby provides a Servlet that can be used for remote administration of Server. The servlet is first deployed in a servlet container like Tomcat which enables administration of the server over HTTP. However the servlet interface only provides management support without any support for monitoring the server. The proposed extensions will enable a server framework using derby to provide live monitoring of system along with the "regular" administration features found in most commercial database systems. Some of these features have been identified in the following sections.

5.3. Conclusion

• Whether running in embedded mode or as server, the developer or user has very limited choices for controlling the behavior of system. In light of the facts above it is desirable to have a monitoring and management extension for derby for that enables developers and administrators to view the performance of a Derby instance without writing any extra code. The exact list of features to be supported by such a extension are discussed below

6. Requirements Identified

6.1. Overview

• This section describes the requirements of the new monitoring and management system for Apache Derby. The requirements are classified in three categories:
  

• 1. Requirements for the Monitoring System: Requirements identified under this category are critical for assessing the health of the system. These are either counters or variables that provide insight into the state of the system.
  2. Requirements for Management System: Requirements identified under this category are essential to fine tuning the state of the system. These operations provide the user ability to change system properties.
  3. Requirements for Notification System: Requirements identified under this category are not essential for a management or monitoring system but provide extended functionality to enterprise applications that embed derby.
  

The requirements identified are for the core Derby engine and do not take into consideration any special requirements for a specific case.

6.2. Proposed Features

• Cateogory/Code	Features	Priority (Sanket)	Priority (David)
  a) Monitoring
  MON1	View derby system/database properties (derby.xxx)	0	0
  MON2	Sys info e.g. version, jvm version etc.	0	0
  MON3	Locks held in the system	1	2
  MON4	Number of commits/sec	0	1
  MON5	Number of roll backs/sec	0	1
  MON6	Time since last save point	2	2
  MON7	Number of system threads	0	1
  MON8	Number of users connected	0	1
  MON9	Number of open connections	0	1
  MON10	Transaction Granularity	2	2
  MON11	Open cursors	1	1
  MON12	Databases
  MON12.1	Database properties (derby.xxx)	0	0
  MON12.2	Database Objects	2	2
  MON12.3	Database Size	1	2
  MON13	Memory usage stats	1	1
  MON14	CPU usage stats	1	1
  MON15	Stats about Garbage collection	1	2
  MON16	View number of cached objects (Query)	2	2
  MON17	Deadlocks detected/trace	1	2
  MON18	Stats about disk I/O	0	1
  MON19	Query plan	2	2
  b) Management
  MGMT1	Modify derby/database properties (derby.xxx)	0	0
  MGMT2	Boot a database	1	2
  MGMT3	Shutdown a database	0	0
  MGMT4	Drop a database	2	2
  MGMT5	Create a database	2	3
  MGMT6	Create database objects	2	3
  MGMT7	Backup a database	1	1
  MGMT8	Restore a database from backup	1	1
  MGMT9	Recovery	1	? Not sure if this is needed - we do auto-recovery
  MGMT10	Force a checkpoint	2	3
  MGMT11	Rebuild indexes	2	2
  MGMT12	Security
  MGMT12.1	Modify Authorization	1	2
  MGMT12.2	Modify Authentication scheme	1	2
  MGMT13	Replacement of management servlet	2	3
  c) Notifications
  NOT1	When a transaction fails	0	1
  NOT2	When a database is booted	1	2
  NOT3	When a database is shutdown	1	2
  NOT4	When a user connects	2	4
  NOT5	When a user disconnects	2	4
  NOT6	Every time a query is fired	2	4
  NOT7	Every time a stored procedure/function is run	2	3
  NOT8	All database operations like creating/deleting/modifying etc.	2	4

  • This is not an exhaustive list and is open to further discussions by the community. In particular, I would like to see some more suggestions on notifications.

Some of these attributes or features may lack sufficient detail at the moment, but these details will be worked out as the work progresses.

6.3. Remote VS. Local

• In my opinion, remote access to these features should not be provided. My opinion is based on the following observations:

• Embedded Nature of derby: The scope of the database can not extend beyond that of application. If the application as a whole does not provide any remote administration feature, it does not make much sense for only a part of the application to be remotely accessible.
  

• Overhead involved: I think adding remote manageability will incur more overhead on the hosting machine. This will be against Derby’s design philosophy.
  

• Design Considerations: Let us think why such functionality will be needed:

  • a) Embedded inside application: If the developer really needs to provide remote management functionality for just the database, I think it is a design flaw. The database should be removed from the application and deployed as a server which can be administered more easily. In case he wants to make the entire application remotely accessible, it will not be very difficult.
    b) Embedded inside a server framework: In this case I think it is the server’s responsibility to expose this functionality to the management application. How would feel if your car’s speed-o-meter were on its tires, fuel gauges on the fuel tank, torque meter on the engine etc.
    c) As a database server: Continuing what I said in 5.2.3 and (2) above, I think it is the responsibility of the server framework to expose this functionality to outside world, not the database engine.
    

  However, keeping in mind the pressing demand by users of Derby, a middle path can be providing remote administration as "Optional Services" that can be enabled and disabled via some command line option when Derby is stared.
  

6.4. Conclusion

• 
  

  • The priorities assigned reflect my own personal views and are open for community discussion. The final rakings will be decided by the community.

7.0 References

• Java Platform Profiling Architecture, JSR 163 at http://www.jcp.org/en/jsr/overview

• Java 2 Standard Edition, http://java.sun.com/j2se/1.5.0/docs/api/

• Apache Foundation’ Derby project http://db.apache.org/derby

• IBM’s Cloudscape developer page: http://www-128.ibm.com/developerworks/db2/zones/cloudscape/

• Java Management Extensions (JMX): http://java.sun.com/j2se/1.5.0/docs/guide/jmx/index.html

• Java Specification Request for JMX: http://www.jcp.org/en/jsr/detail?id=160

• Java Specification Request for JMX: http://www.jcp.org/en/jsr/detail?id=03