Introduction

The Java Broker requires a mechanism that allows a preference, which is owned by a Principal, to be associated with an arbitrary object with the object hierarchy. An example of a preference could be:

a query
a chart
a preferred timezone
a dashboard layout

A preferences will

be owned by a User Principal.
associated with exactly one configured object.
have a unique ID (UUID)
have a visibility which will allow the preference to be private, or shared amongst a group or groups.
normally preferences will be updatable/deletable by the Principal that created it but:
- Special users will be able view/update/delete the preferences of another user. Users with this permission will normally belong to an identity maintainer role.
- Deleting a configured object will cause the associated preferences to be deleted automatically.
Preferences will be durable (survive broker restart)
Some preference types will be understood by the Broker (for example, the query API may later understand a request a execute a query preference). Other preference types may be private to a client. The server will be capable of storing/retrieving these private types, but will not accord them server side behaviour.

Preferences associated with a VirtualHost or below need to automatically propagated automatically amongst the nodes within a HA group. As an operator, this means I will be able to create, say, a query against my HA virtualhost and the same query will be available to me no matter where the mastership resides.

Unlike relationships between CO, preferences are loosely associated with model objects. This means that an operator will be able to remove, say a queue, even if preferences are still associated with it, be they belong to the operator himself or another Principal. In this case, the preference becomes orphaned. Orphaned preferences will be automatically purged by the system

The UI will allow an operator to clone a copy of a preference of another which is visible to him, in order that he may change his own copy.

Some preferences may refer to other preferences, for example, a dashboard preference may refer to one or more query or chart preferences.

For some preference types, a unlimited number of instances may be associated to a single configured object, for example, QUERY. For other preference types, the cardinality will be zero or one. An example will be Timezone.

Preferences need to be extensible and allow for the introduction of further preference types The storage mechanism needs to be versioned so that an automated upgrader can translate an older to a newer one.

By convention, we will associate preferences with either the Broker or Virtualhost. The model will permit, preferences to be associated anywhere, but we don't anticipate exploiting this at the moment.

High level design

REST API

The Preference API will be exposed over REST and must be part of the versioned API. To address the preferences the caller will simply append /preferences as the trailing part of the request. As a consequence, no configured object may have an operation called preferences.

The basic form of the URL to operate on the preferences of the current user only will be:

/api/version/<configured object category>/path/to/configured/object/userpreferences

Users may make preferences visible to other users. The basic form of the URL to access all the preferences visible to the current user only will be as follows. The results will not include the preferences that belong to the user.

/api/version/<configured object category>/path/to/configured/object/visiblepreferences

REST API /userpreferences - Current user preferences

Retrieve

GET will retrieve the preference(s) associated with the identified object that belong to the authenticated user only.

The response will be 200 unless:

The configured object does not exist (response 404)
The preference (by type/name or id) does not exist (response 404)

If no preferences are associated with the configured object the response will be an empty JSON list.

Retrieving one preference by id

The result will be exactly one object.

GET /api/latest/broker/userpreferences?id=75d5d2a4-0573-11e6-b512-3e1d05defe78
HTTP/1.1 200 Ok
 
{
 id: "75d5d2a4-0573-11e6-b512-3e1d05defe7",
 type: "QUERY",
 name: "mypref",
 description: "Acme Hot queues",
 visiblityList: ["ldap.mycompany.org:operators"],
 value: {
   select: "id,name,queueDepthMessages"
   where: "queueDepthMessages > 1000"
 },
 owner: "ldap.mycompany.org:kwall"
 createdDate: 123456789,
 updatedDate: 123456789
}

Retrieving one preference by type/name

The result will be exactly one object

GET /api/latest/broker/userpreferences/query/mypref
HTTP/1.1 200 Ok
 
{
 id: "75d5d2a4-0573-11e6-b512-3e1d05defe7",
 type: "QUERY",
 ....
}

Retrieving all preferences by type

The result will be a list of objects.

GET /api/latest/broker/userpreferences/query
HTTP/1.1 200 Ok
 
[{
  id: "75d5d2a4-0573-11e6-b512-3e1d05defe7",
  type: "QUERY",
  ....
 },
 {
  id: "0e046c3c-091b-4011-b1ac-a03b906684f2",
  type: "QUERY",
  ....
 }
]

Retrieving all preferences

The result will be a map where the key is a type and the value is a list.

GET /api/latest/broker/userpreferences/
 
HTTP/1.1 200 Ok
 
{ "query" : 
 [{
   id: "75d5d2a4-0573-11e6-b512-3e1d05defe7",
   type: "QUERY",
   ....
  },

  {
   id: "0e046c3c-091b-4011-b1ac-a03b906684f2",
   type: "QUERY",
   ....
  }
 ],
  "dashboard" : 
  [....]
 }

Create

A single preference can be created with a PUT request against the full URI (one ending including the preferences/type/name).

PUT /api/latest/virtualhost/myvhn/myvh/userpreferences/query/mypref
{
 description: "Acme Hot queues",
 visiblityList: ["ldap.mycompany.org:operators"],
 value: {
   select: "id,name,queueDepthMessages"
   where: "queueDepthMessages > 1000"
 }
}
 
HTTP/1.1 201 Created

One or more preferences can be added to the configured object by sending a POST to the URI userpreferences/<type> and sending a list, or to the URI /userpreferences and sending a map. This will add the preferences to those that already exist. No Location header can be returned as multiple URIs will have been potentially created.

(What if one of the new preferences fails a business rule?? Some of the preferences may be added, some not. What is returned?)

POST /api/latest/virtualhost/myvhn/myvh/userpreferences
{
  "query" : [{
   description: "Acme Hot queues",
   visiblityList: ["ldap.mycompany.org:operators"],
   value: {
     select: "id,name,queueDepthMessages"
     where: "queueDepthMessages > 1000"
   }
  }]
  "dashboard" : [...]
}
 
HTTP/1.1 201 Created

All preferences belonging to a configured object of a given type can be replaced by sending a PUT to the URI userpreferences/<type> and sending a list, or to the URI /userpreferences and sending a map. This will replace the preferences that already exist.

When creating a preference, values specified for owner, createdDate, lastUpdatedDate are always ignored by the model. visibilityList will be validated to ensure that domain described by it falls completely within the groups to which the user belongs.

Update

A single preference can be updated with a PUT request against the full URI (one ending including the /userpreferences/type/name). The request is the same as create using PUT above, except the 200 OK will be returned instead of 201 Created.

PUT /api/latest/virtualhost/myvhn/myvh/userpreferences/query/mypref
{
 description: "Acme Hot queues",
 visiblityList: ["ldap.mycompany.org:operators"],
 value: {
   select: "id,name,queueDepthMessages"
   where: "queueDepthMessages > 1000"
 }
}
 
HTTP/1.1 200 Created

One or more preferences can be updated on the configured object by sending a POST to the URI userpreferences/<type> and sending a list, or to the URI /userpreferences and sending a map. This will update preferences that already exist the implied locations, and add any that don't.

(What if there are no preference ids on the incoming request? or the ids don't match the preference at the implied location? I think in both case, the request should be rejected. This implies that the caller must call GET to get the id, and return the document with the necessary alter at

POST /api/latest/virtualhost/myvhn/myvh/userpreferences
{
  "query" : [{
   id: "75d5d2a4-0573-11e6-b512-3e1d05defe7",
   description: "Acme Hot queues updated",
   visiblityList: ["ldap.mycompany.org:operators"],
   value: {
     select: "id,name,queueDepthMessages"
     where: "queueDepthMessages > 1000"
   }
  }]
  "dashboard" : [...]
}
 
HTTP/1.1 201 Created

Preferences can be replaced on the configured object by sending a PUT to the URI preferences/<type> and sending a list, or to the URI /preferences and sending a map. This will remove all preferences that exist at that level and replace them with the preferences within the request. It is legal for the client to send an empty list or empty map - this will cause the server to remove the preferences leaving none.

Delete

A single preference can be deleted with a DELETE against its full URI, or ID. All preferences of a particular type, or all preferences belonging to the user can be removed by specifying URI /userpreferences/<type> or /userpreferences respectively.

The delete will be rejected if the authenticated user does not match the preference's owner, unless the user holds the preference maintainer permission.

DELETE /api/latest/virtualhost/myvhn/myvh/userpreferences/query/mypref
DELETE /api/latest/virtualhost/myvhn/myvh/userpreferences/query/?id=75d5d2a4-0573-11e6-b512-3e1d05defe7
DELETE /api/latest/virtualhost/myvhn/myvh/userpreferences/query
DELETE /api/latest/virtualhost/myvhn/myvh/userpreferences

REST API /visiblepreferences - retrieve visible preferences for user

This API permits a caller to retrieve all preferences that are visible to the user, excluding those that belong to him.

A client application may permit the user to 'clone' one of these preferences so that the user may make his own modifications. There is no direct backend support for this above a GET to this URI, followed by a suitable POST/PUT to /userpreferences. The clone is entirely independent of the original.

For GET, /visiblepreferences works in the same manner as /userpreferences. The other HTTP methods are unsupported.

The a super user, all preferences are visible.

REST API operation /broker/delete(user)

This broker-level operation allows the a suitable permissioned user to delete all records personally owned by the identified user. This will delete the user's preferences. In future it may delete other items from belong personally to the user, such as last login records, human names mapping etc.

An organisation may hook this operation up to a 'leavers' feed.

Configured Object Changes

All ConfiguredObjects will have a set containing the preferences associated with that configured object.

The caller will use the ConfiguredObject to retrieve a UserPreferences facade object. It is the facade object that allows preferences, for this user, to be created/read/updated/deleted.

The UserPreferences facade will use a dedicated single thread for it work which will belong the the Configured Object. This approach will allow the individual REST API interactions to be atomic even when the request embodies multiple preferences. It is likely that the preference executor thread will be shared by all configured objects within a contiguous part of the model - that is, there will be a broker preference executor, and a per virtualhost preference executor.

The REST API uses path to retrive the correct ConfiguredObject API then retrieves the UserPreferences object.

public interface ConfiguredObject
{
  ...
  UserPreferences getPreferences();
  ...
}

public interface UserPreferences() 
{
  // Gets all preferences belonging to this user
  Map<String, Map<String,Preference> getPreferences()

  // replaces all current preferences with replacement.  returns preferences that were removed – used by PUT /userpreferences & DELETE
  Map<String, Map<String, Preference>> replace(Map<String, Map<String, Preference>>)

  // update/append preferences – used by POST /userpreferences
  updateOrAppend(Map<String, Map<String, Preference>>) 

  // replaces all prefs of given type with the replacement. returns preferences that were removed – used by PUT /userpreferences/type & DELETE
  Map<String, Preference> replace (String type, Map<String, Preference>)

  // update/append preferences within given type – used by POST /userpreferences/type
  updateOrAppend(String type, Map<String, Preference>)

  // // Gets all preferences visible to this user, does not include those that belong to him
  Map<String, Map<String, Preference> getVisiblePreferences()

  // Allows REST API to construct the Preference objects – does not persist of the object.  Object will be immutable
 Preference createPreference(String type, Map<String,Object>) 
}

UserPreferences#getPreferences

This method gets the preferences that belong to the current user.

Checks with the security manager that the access of the preference is allowed.

UserPreferences#getVisiblePreferences

This method gets the preferences that are visible to the user. This method needs to return all preferences that are visible to the user but do not belong to him.

UserPreferences#replace

Replaces current preferences with new ones (at either the top-level or type level), returning the ones that were removed.

Identifies all the preferences that belong to the user that need to be removed
1. For each preference, call the SecurityManager#authoriseDelete.
For each replacement preference
1. create new preference objects populating the id and the current user as owner
2. call the SecurityManager#authoriseCreate
Tells the store listener about preferences that have been removed and those that have been added.
Return those preferences that were removed

UserPreferences#updateOrAppend

Updates or appends new preferences (at either the top-level or type level).

Separate new preferences from the updates. Do this by examining the incoming parameter: those with an id are updates.
For each update preference:
1. Find the stored preference with the same ID.
2. create a new preference object taking the mutable values from the incoming object and immutable values from stored preference (id, name, owner)
3. SecurityManager#authoriseUpdate on the preference
For all other preferences:
1. create new preference objects populating the id and the current user as owner and other fields from the incoming
2. call the SecurityManager#authoriseCreate
Ensure that preference names remains unique for current user within each type
Tells the store listener about preferences that have been updated and those that have been added.

UserPreferences#createPreference

Create a preference value object. Does not perform ACL check and does not store the preference in the store.

ConfiguredObject#getUserPreferences

Gets the UserPreference facade. The caller can be assured that this object will always return a non-null value, even for those users who have no permission to access or mutate preferences.

Preferences Model Object

Preference object represents an instance of a preference. Each Preference will have a PreferenceValue encapsulating the type specific details of preference. There will be a GenericPreferenceValue used to represent arbitrary preferences that the server requires no knowledge (these will have a type name beginning X-). There will also specific concrete implementations of PreferenceValue that represent preferences types which the server has knowledge.

Concrete instances of Preference/PreferenceValue will be immutable.

Preference Objects do not make access decisions.

public interface Preference<V extends PreferenceValue>
{
 UUID getId();
 String getName();
 String getType();
 String getDescription();
 Principal getOwner();
 ConfiguredObject getAssociatedObject()
 List<Principal> getVisibiltyList();
 Date getCreatedDate();
 Date getLastUpdatedDate();
 
 V getValue();
 Map<String,Object> getAttributes();
}
 
public interface PreferenceValue
{
  Map<String,Object> getAttributes();
}

Preference#getAttributes

Returns a r/o map containing the attributes of the preference. The map will have key/value pairs for id, name, type, description, owner, visibilityList, createdDate, lastUpdateDate. It will also have a key value whose value is a map containing the key/value pairs for the preference value itself.

Preferences Factory

There will be a preference factory which will work rather like the object factory. The Model will give the ConfiguredObject a preference factory. BrokerModel will have a hard relationship with a PreferencesFactoryImpl which will know how to create Preference and PreferenceValue objects from the attributes.

The factory will use the type discriminator to determine the actual type of PreferenceValue. If the discriminator begins with "X-" the GenericPreferenceValue will be used, other a specific concrete PreferenceValue will be used.

Preference Storage

In the store an instance of a Preference will look like the example below.

{
 id: <UUID - immutable>,
 name: <name - immutable>,
 type: <type discriminator>
 description: <description>,
 associatedId: <UUID of model object - immutable>,
 owner: <Domain prefixed Principal - normally that belonging to a user - immutable>,
 visibliltyList: [<Domain prefixed Principal - normally that belonging to a group>],
 value: {
   ...
 },
 createdDate: 123456789,
 updatedDate: 123456789
}

Security Manager

The security manager will gain responsibilities for authorising preference view/mutate operations.

The Security Manager will first need to check that the user has access to the ConfiguredObject itself. For the moment the ordinary user operations we will probably use the existing ACL rules CONFIGURE BROKER and ACCESS VIRTUALHOST

For the super-user, we will need a new permission will be required (ACCESS PREFERENCES). A user in the identity maintainer role will normally hold this.

SecurityManager#authoriseCreate

Returns true if the user is allow to create preferences. It will be true if:

the user belongs to all the groups identified in the visibilityList
the owner matches the authenticated user.

SecurityManager#authoriseUpdate

Returns true if the user is allow to update the preference. It will be true if:

the user belongs to all the groups identified in the visibilityList
the owner matches the authenticated user.

SecurityManager#authoriseDelete

Returns true if the user is allow to delete this preference:

authenticated user owns the preference

SecurityManager#authoriseView

Returns true if the user is allow to view this preference:

authenticated user owns the preference, or
authenticated user belongs to a group that is visible to the visibilityList

Preference Store

We will have a PreferenceStore interface similiar to the DurableConfigurationStore but with some notable difference

open/close
~~upgrade~~
~~visit~~
create
update
delete

Compared to the DurableConfigurationStore the PreferenceStore has the following differences:

no upgrade method. The store should handle upgrades transparently during startup/open. This is considered a fault in the original API
no visit method. Rather the open method should handle initial loading of the data. See below for more detail.

API is in terms of PreferenceRecords (analogous to ConfiguredObjectRecords) which have minimal knowledge of the internal structure except from the UUID. The ID is necessary to identify records for deletion and update operations.

Broker and VirtualHost have separate preference stores.

The Broker and VirtualHost configuration will have an attribute of a custom ManagedAttributeValueType containing a type and an optional attributes filed for type specific configuration (e.g., store location).
The concrete instances of SystemConfig and VirtualHostNode responsible for creating the preference store too.
For storage mechanism BDB, JDBC, Derby there would be a restriction that the preference store must be co-located with the configuration.
For storage mechanism JSON the preferences would be stored in a separate file.

interface PreferenceStore
{
    // see below for the Updater and Recoverer interfaces
    Collection<PreferenceRecords> openAndLoad(PreferenceStoreUpdater updater);

    // safely persist all state and close all resources
    void close();

    // adds preferences to the store
    void create(Collection<PreferenceRecord> preferences);

    // updates existing preferences. throws an exception if preference with given id is not already in the store
    void update(Collection<PreferenceRecord> preferences);

    // updates existing preferences. if a preference with given id is not already in the store it will be added
    void updateOrCreate(Collection<PreferenceRecord> preferences);

    // remove preferences from the store. throws an exception if preference with given id is not already in the store
    void delete(Collection<PreferenceRecord> preferences);
}

interface PreferenceRecord
{
    UUID getId();
	Map<String, Object> getAttributes();
}

Recovery

At recovery time the PreferenceRecords in the PreferenceStore will need to be converted into Preference objects. This needs to occur after the configured objects are recovered. In this phase, objects corresponding to the records must be created. The recovery algorithm will be similar to that of configured object recovery as preferences may reference other preferences, so some preferences (Dashboards) may have to await the creation of others before they may be created. As preferences are created they need to be assigned to their associated configured object.

If during the recovery phase, a preference is encountered that refers to a configured object that no longer exists, the preference can be dropped. If a preference is found that refers to a preference that no longer exists, the reference can be removed. The latter would happen if a dashboard of user B refers to a query of user A. If user A deletes their query, user B's reference will be left dangling. On next recovery, the dangling reference must be removed. (The UI should probably just display a ? where the query would have been).

The idea behind getting rid of the visit method is to be able to use a store that is essentially write-only after it has been read from disk. However the loading needs some customization hooks:

we want to ability to update individual preferences. This may include updating, creating and deleting preferences. This might require multiple passes.
Since the update step might arbitrarily modify the preferences the store needs to be informed about the changes so that the update can be persisted.
preferences need to ability to reference each other. This is done by storing UUIDs. The objects using the preferences are responsible for resolving those references.

The first to points are the responsibility of the PreferenceStoreUpdater while the second should be performed by the PreferenceStoreRecoverer.

The recoverer is responsible for

Convert PreferenceRecord to Preference
Convert generic attributes to a type specific PreferenceValue
Attach Preference to AbstractConfiguredObject

interface PreferenceStoreUpdater
{
    Collection<PreferenceRecord> updatePreferences(String currentVersion, Collection<PreferenceRecord> preferences);
    String getLatestVersion();
}

Persistence

Preferences are considered immutable. All changes to preferences should go through a UserPreferences facade which is responsible for updating the store.

High Level Class Diagram

preferences

Tasks

Sequence of tasks

Implement REST API, preferences model and configured objects changes. After this work it will be possible to view, add and remove preferences at runtime, but preferences will not be persisted. There will be no security.
Change the UI to allow query preferences to be added/removed/viewed
Implement security manager.
Reimplement timezones, refresh period, open tabs(?) as preferences
Implement preference persistence and recovery
Remove old preference model object. Implement a configuration upgrader to remove old preferences from configuration (suggest we do not try and maintain the current user settings)

Once item 1) is done, the remaining steps can be parallelised.

Open Questions

Should preferences have metadata? At the moment I think we can live without it
How will preferences map to AMQP Management?
What limits should be placed on the maximum number of preferences the user is allowed to have?

Child pages

Preference Store

Introduction

REST API

REST API /userpreferences - Current user preferences

Retrieve

Retrieving one preference by id

Retrieving one preference by type/name

Retrieving all preferences by type

Retrieving all preferences

Create

Update

Delete

REST API /visiblepreferences - retrieve visible preferences for user

REST API operation /broker/delete(user)

Configured Object Changes

UserPreferences#getPreferences

UserPreferences#getVisiblePreferences

UserPreferences#replace

UserPreferences#updateOrAppend

UserPreferences#createPreference

ConfiguredObject#getUserPreferences

Preferences Model Object

Preference#getAttributes

Preferences Factory

Preference Storage

Security Manager

SecurityManager#authoriseCreate

SecurityManager#authoriseUpdate

SecurityManager#authoriseDelete

SecurityManager#authoriseView

Preference Store

Recovery

High Level Class Diagram

Tasks

Open Questions