Differences between revisions 4 and 5
Revision 4 as of 2006-06-12 22:32:42
Size: 19550
Comment:
Revision 5 as of 2009-09-20 22:12:27
Size: 19594
Editor: localhost
Comment: converted to 1.6 markup
Deletions are marked like this. Additions are marked like this.
Line 7: Line 7:
1. [#Revision_History Revision History][[BR]]
2. [#Disclaimer Disclaimer and Legal Information][[BR]]
3. [#About About this document][[BR]]
 3.1 [#Purpose Purpose][[BR]]
 3.2 [#Audience Intended Audience][[BR]]
 3.3 [#Notations Notations and Conventions][[BR]]
 3.4 [#Structure Structure of this document][[BR]]
 3.5 [#Note A note about ranking][[BR]]
4. [#Assumptions Assumptions and Dependencies][[BR]]
5. [#Analysis Requirement Analysis][[BR]]
 5.1 [#Overview Overview][[BR]]
 5.2 [#Current Current Scenario][[BR]]
  5.2.1 [#Embedded Embedded Instance][[BR]]
  5.2.2 [#Embedded_Server Embedded in Server Framework][[BR]]
  5.2.3 [#Network_Server Network Server][[BR]]
 5.3 [#Conclusion Conclusion][[BR]]
6. [#Identified Requirements Identified][[BR]]
 6.1 [#Feature_Overview Overview][[BR]]
 6.2 [#Feature_Identified Features Identified][[BR]]
 6.3 [#RemoteLocal Remote vs. Local][[BR]]
 6.4 [#Feature_Conclusion Conclusion][[BR]]
7. [#References References][[BR]]
1. [[#Revision_History|Revision History]]<<BR>>
2. [[#Disclaimer|Disclaimer and Legal Information]]<<BR>>
3. [[#About|About this document]]<<BR>>
 3.1 [[#Purpose|Purpose]]<<BR>>
 3.2 [[#Audience|Intended Audience]]<<BR>>
 3.3 [[#Notations|Notations and Conventions]]<<BR>>
 3.4 [[#Structure|Structure of this document]]<<BR>>
 3.5 [[#Note|A note about ranking]]<<BR>>
4. [[#Assumptions|Assumptions and Dependencies]]<<BR>>
5. [[#Analysis|Requirement Analysis]]<<BR>>
 5.1 [[#Overview|Overview]]<<BR>>
 5.2 [[#Current|Current Scenario]]<<BR>>
  5.2.1 [[#Embedded|Embedded Instance]]<<BR>>
  5.2.2 [[#Embedded_Server|Embedded in Server Framework]]<<BR>>
  5.2.3 [[#Network_Server|Network Server]]<<BR>>
 5.3 [[#Conclusion|Conclusion]]<<BR>>
6. [[#Identified|Requirements Identified]]<<BR>>
 6.1 [[#Feature_Overview|Overview]]<<BR>>
 6.2 [[#Feature_Identified|Features Identified]]<<BR>>
 6.3 [[#RemoteLocal|Remote vs. Local]]<<BR>>
 6.4 [[#Feature_Conclusion|Conclusion]]<<BR>>
7. [[#References|References]]<<BR>>
Line 30: Line 30:
[[Anchor(Revision_History)]] <<Anchor(Revision_History)>>
Line 36: Line 36:
[[Anchor(Disclaimer)]] <<Anchor(Disclaimer)>>
Line 44: Line 44:
[[Anchor(About)]] <<Anchor(About)>>
Line 46: Line 46:
[[Anchor(Purpose)]] <<Anchor(Purpose)>>
Line 48: Line 48:
[[BR]] <<BR>>
Line 50: Line 50:
[[Anchor(Audience)]] <<Anchor(Audience)>>
Line 52: Line 52:
[[BR]] <<BR>>
Line 54: Line 54:
[[Anchor(Notations)]] <<Anchor(Notations)>>
Line 56: Line 56:
[[BR]] <<BR>>
Line 70: Line 70:
[[BR]] <<BR>>
Line 72: Line 72:
[[Anchor(Structure)]] <<Anchor(Structure)>>
Line 74: Line 74:
[[BR]] <<BR>>
Line 76: Line 76:
[[Anchor(Note)]] <<Anchor(Note)>>
Line 78: Line 78:
[[BR]] <<BR>>
Line 80: Line 80:
[[Anchor(Assumptions)]]
[[BR]]
<<Anchor(Assumptions)>>
<<BR>>
Line 83: Line 83:
[[BR]] <<BR>>
Line 85: Line 85:
[[Anchor(Analysis)]] <<Anchor(Analysis)>>
Line 88: Line 88:
[[Anchor(Overview)]] <<Anchor(Overview)>>
Line 90: Line 90:
[[BR]] <<BR>>
Line 92: Line 92:
[[Anchor(Current)]] <<Anchor(Current)>>
Line 94: Line 94:
[[BR]]
       ''' 1. Monitoring the state the system ''' [[BR]]
  [[BR]]
       ''' 2. Tuning system parameters ''' [[BR]]
  [[BR]]
<<BR>>
       ''' 1. Monitoring the state the system ''' <<BR>>
  <<BR>>
       ''' 2. Tuning system parameters ''' <<BR>>
  <<BR>>
Line 100: Line 100:
[[BR]] <<BR>>
Line 102: Line 102:
[[Anchor(Embedded)]]
   When Derby runs as an embedded database engine sharing the same JVM there are quite a few ways for setting the properties for a Derby instance, both system wide and database. Such properties can be set either in the derby.properties file or can be at run time by: [[BR]]
  [[BR]]

  * Using parameters while starting application e.g. java -Dderby.xxx=something YourApplication. [[BR]]
  [[BR]]

  * Using API's provided. [[BR]]
  [[BR]]
<<Anchor(Embedded)>>
   When Derby runs as an embedded database engine sharing the same JVM there are quite a few ways for setting the properties for a Derby instance, both system wide and database. Such properties can be set either in the derby.properties file or can be at run time by: <<BR>>
  <<BR>>

  * Using parameters while starting application e.g. java -Dderby.xxx=something YourApplication. <<BR>>
  <<BR>>

  * Using API's provided. <<BR>>
  <<BR>>
Line 114: Line 114:
[[BR]] <<BR>>
Line 116: Line 116:
[[Anchor(Embedded_Server)]]

      [[BR]]Essentially, this is same as 5.2.1 where the application server or some other server framework becomes client for Derby. Most application servers come with a management console that lets administrators and developers configure various system wide settings locally or remotely. Since Derby does not provide any management framework that integrates into application server frameworks, developers of such systems using Derby have to write wrappers around Derby API calls to provide any management functionality, if at all.
<<Anchor(Embedded_Server)>>

      <<BR>>Essentially, this is same as 5.2.1 where the application server or some other server framework becomes client for Derby. Most application servers come with a management console that lets administrators and developers configure various system wide settings locally or remotely. Since Derby does not provide any management framework that integrates into application server frameworks, developers of such systems using Derby have to write wrappers around Derby API calls to provide any management functionality, if at all.
Line 122: Line 122:
[[BR]] <<BR>>
Line 124: Line 124:
[[Anchor(Network_Server)]] <<Anchor(Network_Server)>>
Line 131: Line 131:
[[BR]] <<BR>>
Line 133: Line 133:
[[Anchor(Conclusion)]] <<Anchor(Conclusion)>>
Line 135: Line 135:
[[BR]] <<BR>>
Line 137: Line 137:
[[Anchor(Identified)]] <<Anchor(Identified)>>
Line 140: Line 140:
[[Anchor(Feature_Overview)]]
     This section describes the requirements of the new monitoring and management system for Apache Derby. The requirements are classified in three categories: [[BR]]

 ''' 1. Requirements for the Monitoring System: ''' Requirements identified under this category are critical for assessing the health of the system. These are either counters or variables that provide insight into the state of the system. [[BR]]
 ''' 2. Requirements for Management System: ''' Requirements identified under this category are essential to fine tuning the state of the system. These operations provide the user ability to change system properties. [[BR]]
 ''' 3. Requirements for Notification System: ''' Requirements identified under this category are not essential for a management or monitoring system but provide extended functionality to enterprise applications that embed derby. [[BR]]
<<Anchor(Feature_Overview)>>
     This section describes the requirements of the new monitoring and management system for Apache Derby. The requirements are classified in three categories: <<BR>>

 ''' 1. Requirements for the Monitoring System: ''' Requirements identified under this category are critical for assessing the health of the system. These are either counters or variables that provide insight into the state of the system. <<BR>>
 ''' 2. Requirements for Management System: ''' Requirements identified under this category are essential to fine tuning the state of the system. These operations provide the user ability to change system properties. <<BR>>
 ''' 3. Requirements for Notification System: ''' Requirements identified under this category are not essential for a management or monitoring system but provide extended functionality to enterprise applications that embed derby. <<BR>>
Line 150: Line 150:
[[Anchor(Feature_Identified)]] <<Anchor(Feature_Identified)>>
Line 208: Line 208:
[[Anchor(RemoteLocal)]] <<Anchor(RemoteLocal)>>
Line 211: Line 211:
 * ''' Embedded Nature of derby: ''' The scope of the database can not extend beyond that of application. If the application as a whole does not provide any remote administration feature, it does not make much sense for only a part of the application to be remotely accessible.[[BR]]
 * ''' Overhead involved: ''' I think adding remote manageability will incur more overhead on the hosting machine. This will be against Derby’s design philosophy. [[BR]]
 * ''' Embedded Nature of derby: ''' The scope of the database can not extend beyond that of application. If the application as a whole does not provide any remote administration feature, it does not make much sense for only a part of the application to be remotely accessible.<<BR>>
 * ''' Overhead involved: ''' I think adding remote manageability will incur more overhead on the hosting machine. This will be against Derby’s design philosophy. <<BR>>
Line 216: Line 216:
  ''' a) Embedded inside application: ''' If the developer really needs to provide remote management functionality for just the database, I think it is a design flaw. The database should be removed from the application and deployed as a server which can be administered more easily. In case he wants to make the entire application remotely accessible, it will not be very difficult. [[BR]]
  ''' b) Embedded inside a server framework: ''' In this case I think it is the server’s responsibility to expose this functionality to the management application. How would feel if your car’s speed-o-meter were on its tires, fuel gauges on the fuel tank, torque meter on the engine etc. [[BR]]
  ''' c) As a database server: ''' Continuing what I said in 5.2.3 and (2) above, I think it is the responsibility of the server framework to expose this functionality to outside world, not the database engine. [[BR]]

 However, keeping in mind the pressing demand by users of Derby, a middle path can be providing remote administration as "Optional Services" that can be enabled and disabled via some command line option when Derby is stared. [[BR]]
  ''' a) Embedded inside application: ''' If the developer really needs to provide remote management functionality for just the database, I think it is a design flaw. The database should be removed from the application and deployed as a server which can be administered more easily. In case he wants to make the entire application remotely accessible, it will not be very difficult. <<BR>>
  ''' b) Embedded inside a server framework: ''' In this case I think it is the server’s responsibility to expose this functionality to the management application. How would feel if your car’s speed-o-meter were on its tires, fuel gauges on the fuel tank, torque meter on the engine etc. <<BR>>
  ''' c) As a database server: ''' Continuing what I said in 5.2.3 and (2) above, I think it is the responsibility of the server framework to expose this functionality to outside world, not the database engine. <<BR>>

 However, keeping in mind the pressing demand by users of Derby, a middle path can be providing remote administration as "Optional Services" that can be enabled and disabled via some command line option when Derby is stared. <<BR>>
Line 223: Line 223:
[[Anchor(Feature_Conclusion)]]
 [[BR]]
<<Anchor(Feature_Conclusion)>>
 <<BR>>
Line 229: Line 229:
[[Anchor(References)]] <<Anchor(References)>>

Apache Derby

Requirement Specifications for Monitoring and Management Extensions using JMX

Document Contents

1. Revision History
2. Disclaimer and Legal Information
3. About this document

4. Assumptions and Dependencies
5. Requirement Analysis

6. Requirements Identified

7. References

1. Revision History

Version

Version Information

Date

Modified by

Comments

1.0

Document Created

Tuesday, June 06, 2006

Sanket Sharma

Comments are welcome!

Copyright 2005 The Apache Software Foundation or its licensors, as applicable.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

3. About this Document

3.1. Purpose

  • This document identifies the requirements and design goals for adding Management extensions to Apache Derby as Google Summer of Code 2006 project. Consequently the document identifies a set of requirements, categorizes them and attempts to assign priority to the identified requirements for their implementation. However, the priorities assigned reflect my own personal views and are left open for the community to decide. The final priorities will be decided by the community as a whole. This document does not describe any high level architecture or implementation details. These will be described in a subsequent documents.


3.2. Intended Audience

  • The intended audience includes a wide community of those interested in developing monitoring and management applications for the Apache Derby. It also includes the end users of the database who will ultimately use these features and the developers who might use these features for fine tuning their applications.


3.3. Notations and Conventions

  • The documentation set uses the following conventions.


  • Convention

    Explanation

    Example

     Monospace 

    Filename

     MyFile.java 

    Directory Names and Path Names

     \MyDir\MyDir2 

    Commands and Command line options

     ecl-02 

    Method names, Classes, Interfaces in running text

    Use the  createService()  method to...

    Parameters or other placeholders

     doSomething(int val1, int val2) 

     Monospace bold 

    User input

     [C:] dir 

    Italics

    Emphasis; Introducing or defining terms

    The term access takes as its subject

    [ ]

    Optional Items

     -Fa[c]  Indicates these command-line options:  -Fa  and  -Fac 

    {|}

    Choice of one item from a selection of two or more items.

    -aX{K | W | P} Indicates these command-line options:

     -aXK 

     -aXW 

     -aXP 


3.4. Structure of this document

  • This section is divided into seven sections. Section one to four provide basic information about legal stuff and notational conventions used in this document. Section five describes what features currently exist in Derby to monitor and tune the performance of the system and why we need a better monitoring and management system. Section six then tries to identify features that might be useful to developers and administrators. While this section enumerates a whole bunch of features, this section also tries to assign a priority to their implementation. Section seven enlists the sources of information for this document.


3.5. A Note about Ranking

  • Developers and Derby users who read this document are requested to assign their priorities to the features enlisted in section six. I have provided an initial version of priorities based on my personal opinion. Developers and Users are requested add a column to the table, add their names as coloumn heading and assign priorities to the features. This will help us in deciding which features have to be implemented and in which order.


4. Assumptions and Dependencies



5. Requirement Analysis

5.1. Overview

  • Apache Derby is a relational, light weight, embedded database developed entirely in Java. Being an embedded database it is very flexible in its use and can be used either embedded inside an application sharing the same JVM as the application or can be embedded inside a server framework (such as Geronimo) or can be deployed as a standalone server (with the Network server framework) in a traditional client server environment. However, currently derby does not support any mature monitoring and management features which lets a developer or an administrator view statistics about the health of a system and fine tune system parameters via a management console or application. Such functionality is indispensable for any database system of any scale for a multitude of reasons. This section tries to identify the requirements for such a monitoring and management framework.


5.2. Current Scenario

  • The following section identifies the what is currently supported in derby with reference to:


  • 1. Monitoring the state the system


    2. Tuning system parameters


  • Although current features can not be termed as monitoring or management features, but they do provide some primitive functionality. The following section describes these in details with respect to the three ways Derby can be used - embedded and as network server.


5.2.1. Embedded Instance

  • When Derby runs as an embedded database engine sharing the same JVM there are quite a few ways for setting the properties for a Derby instance, both system wide and database. Such properties can be set either in the derby.properties file or can be at run time by:


    • Using parameters while starting application e.g. java -Dderby.xxx=something YourApplication.

    • Using API's provided.

  • While these features let an application read or update system properties, there is no interactive way of doing it. Further, these can not be termed as monitoring and management features. What is needed is a functionality which a developer can use to tune the system while examining its state from outside the application. A typical usage scenario would be when a user runs his application and then starts the management console to view the current state of system or to debug his application. Without writing any code in his application he is able to change the system state and view how the system responds. Such functionality should be available to all applications without any coding. The developer/administrator should be able to tune the system at runtime without changing his code.


5.2.2. Embedded inside a server framework


  • Essentially, this is same as 5.2.1 where the application server or some other server framework becomes client for Derby. Most application servers come with a management console that lets administrators and developers configure various system wide settings locally or remotely. Since Derby does not provide any management framework that integrates into application server frameworks, developers of such systems using Derby have to write wrappers around Derby API calls to provide any management functionality, if at all.

  • These servers usually generate a lot of runtime statistics and building monitoring extensions right into Derby will make integration much more transparent. The need for a standard management framework is accentuated in this case. Currently, the only way a developer can change system settings is via API calls or derby.properties file.


5.2.3. Network Server

  • In this mode Derby runs in embedded mode, wrapped inside a server framework that multiple remote clients connect to. Derby can run as full fledged relational database server with the help of bundled server framework. Multiple remote clients can connect to database server simultaneously while the server takes care of concurrency issues and ACID properties.
  • Again, .properties file and API calls are the only mechanisms to tune the system. However, in this mode Derby provides a Servlet that can be used for remote administration of Server. The servlet is first deployed in a servlet container like Tomcat which enables administration of the server over HTTP. However the servlet interface only provides management support without any support for monitoring the server. The proposed extensions will enable a server framework using derby to provide live monitoring of system along with the "regular" administration features found in most commercial database systems. Some of these features have been identified in the following sections.


5.3. Conclusion

  • Whether running in embedded mode or as server, the developer or user has very limited choices for controlling the behavior of system. In light of the facts above it is desirable to have a monitoring and management extension for derby for that enables developers and administrators to view the performance of a Derby instance without writing any extra code. The exact list of features to be supported by such a extension are discussed below


6. Requirements Identified

6.1. Overview

  • This section describes the requirements of the new monitoring and management system for Apache Derby. The requirements are classified in three categories:

  • 1. Requirements for the Monitoring System: Requirements identified under this category are critical for assessing the health of the system. These are either counters or variables that provide insight into the state of the system.
    2. Requirements for Management System: Requirements identified under this category are essential to fine tuning the state of the system. These operations provide the user ability to change system properties.
    3. Requirements for Notification System: Requirements identified under this category are not essential for a management or monitoring system but provide extended functionality to enterprise applications that embed derby.

The requirements identified are for the core Derby engine and do not take into consideration any special requirements for a specific case.

6.2. Proposed Features

  • Cateogory/Code

    Features

    Priority (Sanket)

    Priority (David)

    a) Monitoring

    MON1

    View derby system/database properties (derby.xxx)

    0

    0

    MON2

    Sys info e.g. version, jvm version etc.

    0

    0

    MON3

    Locks held in the system

    1

    2

    MON4

    Number of commits/sec

    0

    1

    MON5

    Number of roll backs/sec

    0

    1

    MON6

    Time since last save point

    2

    2

    MON7

    Number of system threads

    0

    1

    MON8

    Number of users connected

    0

    1

    MON9

    Number of open connections

    0

    1

    MON10

    Transaction Granularity

    2

    2

    MON11

    Open cursors

    1

    1

    MON12

    Databases

    MON12.1

    Database properties (derby.xxx)

    0

    0

    MON12.2

    Database Objects

    2

    2

    MON12.3

    Database Size

    1

    2

    MON13

    Memory usage stats

    1

    1

    MON14

    CPU usage stats

    1

    1

    MON15

    Stats about Garbage collection

    1

    2

    MON16

    View number of cached objects (Query)

    2

    2

    MON17

    Deadlocks detected/trace

    1

    2

    MON18

    Stats about disk I/O

    0

    1

    MON19

    Query plan

    2

    2

    b) Management

    MGMT1

    Modify derby/database properties (derby.xxx)

    0

    0

    MGMT2

    Boot a database

    1

    2

    MGMT3

    Shutdown a database

    0

    0

    MGMT4

    Drop a database

    2

    2

    MGMT5

    Create a database

    2

    3

    MGMT6

    Create database objects

    2

    3

    MGMT7

    Backup a database

    1

    1

    MGMT8

    Restore a database from backup

    1

    1

    MGMT9

    Recovery

    1

    ? Not sure if this is needed - we do auto-recovery

    MGMT10

    Force a checkpoint

    2

    3

    MGMT11

    Rebuild indexes

    2

    2

    MGMT12

    Security

    MGMT12.1

    Modify Authorization

    1

    2

    MGMT12.2

    Modify Authentication scheme

    1

    2

    MGMT13

    Replacement of management servlet

    2

    3

    c) Notifications

    NOT1

    When a transaction fails

    0

    1

    NOT2

    When a database is booted

    1

    2

    NOT3

    When a database is shutdown

    1

    2

    NOT4

    When a user connects

    2

    4

    NOT5

    When a user disconnects

    2

    4

    NOT6

    Every time a query is fired

    2

    4

    NOT7

    Every time a stored procedure/function is run

    2

    3

    NOT8

    All database operations like creating/deleting/modifying etc.

    2

    4

    • This is not an exhaustive list and is open to further discussions by the community. In particular, I would like to see some more suggestions on notifications.

Some of these attributes or features may lack sufficient detail at the moment, but these details will be worked out as the work progresses.

6.3. Remote VS. Local

  • In my opinion, remote access to these features should not be provided. My opinion is based on the following observations:
  • Embedded Nature of derby: The scope of the database can not extend beyond that of application. If the application as a whole does not provide any remote administration feature, it does not make much sense for only a part of the application to be remotely accessible.

  • Overhead involved: I think adding remote manageability will incur more overhead on the hosting machine. This will be against Derby’s design philosophy.

  • Design Considerations: Let us think why such functionality will be needed:

    • a) Embedded inside application: If the developer really needs to provide remote management functionality for just the database, I think it is a design flaw. The database should be removed from the application and deployed as a server which can be administered more easily. In case he wants to make the entire application remotely accessible, it will not be very difficult.
      b) Embedded inside a server framework: In this case I think it is the server’s responsibility to expose this functionality to the management application. How would feel if your car’s speed-o-meter were on its tires, fuel gauges on the fuel tank, torque meter on the engine etc.
      c) As a database server: Continuing what I said in 5.2.3 and (2) above, I think it is the responsibility of the server framework to expose this functionality to outside world, not the database engine.

    However, keeping in mind the pressing demand by users of Derby, a middle path can be providing remote administration as "Optional Services" that can be enabled and disabled via some command line option when Derby is stared.

6.4. Conclusion


    • The priorities assigned reflect my own personal views and are open for community discussion. The final rakings will be decided by the community.

7.0 References

Requirement_Specifications (last edited 2009-09-20 22:12:27 by localhost)