ORB Monitoring Extensions for AS 8.2

It has become apparent that some changes are needed in the ORB monitoring framework for AS 8.2. The current monitoring code in the ORB is overly long and difficult to construct. We also need to consider upcoming requirements in AS 8.2 in areas such as dynamic configuration changes and self-tuning. I would like to build a dynamic MBean directly in the ORB so that we can more readily experiment with different maangement tools, and also to fully validate the ORB monitoring framework design.

Simplifying the Usage of ORB Monitoring

Currently the ORB monitoring code is rather long. For example, here is the code in CorbaInboundConnectionCacheImpl:

    protected void registerWithMonitoring()
    {
        // ORB
        MonitoredObject orbMO = 
            orb.getMonitoringManager().getRootMonitoredObject();

        // REVISIT - add ORBUtil mkdir -p like operation for this.

        // CONNECTION
        MonitoredObject connectionMO = 
            orbMO.getChild(MonitoringConstants.CONNECTION_MONITORING_ROOT);
        if (connectionMO == null) {
            connectionMO = 
                MonitoringFactories.getMonitoredObjectFactory()
                    .createMonitoredObject(
                        MonitoringConstants.CONNECTION_MONITORING_ROOT,
                        MonitoringConstants.CONNECTION_MONITORING_ROOT_DESCRIPTION);
            orbMO.addChild(connectionMO);
        }

        // INBOUND CONNECTION
        MonitoredObject inboundConnectionMO = 
            connectionMO.getChild(
                MonitoringConstants.INBOUND_CONNECTION_MONITORING_ROOT);
        if (inboundConnectionMO == null) {
            inboundConnectionMO =
                MonitoringFactories.getMonitoredObjectFactory()
                    .createMonitoredObject(
                        MonitoringConstants.INBOUND_CONNECTION_MONITORING_ROOT,
                        MonitoringConstants.INBOUND_CONNECTION_MONITORING_ROOT_DESCRIPTION);
            connectionMO.addChild(inboundConnectionMO);
        }

        // NODE FOR THIS CACHE
        MonitoredObject thisMO = 
            inboundConnectionMO.getChild(getMonitoringName());
        if (thisMO == null) {
            thisMO =
                MonitoringFactories.getMonitoredObjectFactory()
                    .createMonitoredObject(
                        getMonitoringName(),
                        MonitoringConstants.CONNECTION_MONITORING_DESCRIPTION);
            inboundConnectionMO.addChild(thisMO);
        }

        LongMonitoredAttributeBase attribute;

        // ATTRIBUTE
        attribute = new 
            LongMonitoredAttributeBase(
                MonitoringConstants.CONNECTION_TOTAL_NUMBER_OF_CONNECTIONS, 
                MonitoringConstants.CONNECTION_TOTAL_NUMBER_OF_CONNECTIONS_DESCRIPTION)
            {
                public Object getValue() {
                    return new Long(CorbaInboundConnectionCacheImpl.this.numberOfConnections());
                }
            };
        thisMO.addAttribute(attribute);

        // ATTRIBUTE
        attribute = new 
            LongMonitoredAttributeBase(
                MonitoringConstants.CONNECTION_NUMBER_OF_IDLE_CONNECTIONS, 
                MonitoringConstants.CONNECTION_NUMBER_OF_IDLE_CONNECTIONS_DESCRIPTION)
            {
                public Object getValue() {
                    return new Long(CorbaInboundConnectionCacheImpl.this.numberOfIdleConnections());
                }
            };
        thisMO.addAttribute(attribute);

        // ATTRIBUTE
        attribute = new 
            LongMonitoredAttributeBase(
                MonitoringConstants.CONNECTION_NUMBER_OF_BUSY_CONNECTIONS, 
                MonitoringConstants.CONNECTION_NUMBER_OF_BUSY_CONNECTIONS_DESCRIPTION)
            {
                public Object getValue() {
                    return new Long(CorbaInboundConnectionCacheImpl.this.numberOfBusyConnections());
                }
            };
        thisMO.addAttribute(attribute);
    }

Fully formatted, this is 82 lines of source. Here are some thoughts on simplifying assumptions:

  1. (BIG assumption): Use a Dynamic MBean constructed by introspection on the MonitoredXXX objects reachable from the Root MonitoredObject in the ORB. This eliminates the need for MonitoringConstants, and consequently simplifies the declarations.

    There is a problem with this: the app server currently has a surprisingly complicated way of getting from ORB MonitoredObject (which is dynamic) to a Dynamic MBean by going through a static Stats layer (ugh). We could simply keep the necessary constants for the AS in MonitoringConstants, but I would rather get rid of them.

  2. Any reasonable name (let's say a Java identifier) is legal for MonitoredObjects.

  3. MonitoredAttribute names must start with a capital letter and be Java identifiers (SomethingLikeThis).

  4. All descriptions for MonitoredObjects and MonitoredAttributes are stored in a resource file (Can we make this dynamic if necessary?)

  5. The key in the resource file for object or attribute "XXX" is "XXX.description".

  6. The MonitoredAttribute SomethingLikeThis is implemented by a method named somethingLikeThis. RO means only a getter (TYPE somethingLikeThis()) while a RW attribute has a setter as well (void somethingLikeThis(TYPE)).

  7. The TYPE of an attribute is determined from the getter/setter methods.

  8. We provide a utility class MUtil in spi.monitoring.

With these assumptions, we could get the example above down to:

protected void registerWithMonitoring()
{
    MonitoredObject thisMO = MUtil.get( orb, "/Connections/Inbound/" + getMonitoringName() ) ;
    thisMO.addAttribute( "NumberOfConnections", this ) ;
    thisMO.addAttribute( "NumberOfIdleConnections", this ) ;
    thisMO.addAttribute( "NumberOfBusyConnections", this ) ;
}

That's it! Just 4 lines of code.

Here the get method get or creates the MonitoredObjects as needed (as Harold noted, a "mkdir -p" like operation). addAttribute uses all of the assumptions mentioned above to derive the appropriate information.

MonitoredObjects for JavaBeans

A further extension to this is possible. Consider the case of ORBData. ORBData has a lot of methods that look like:

     (some data type) getXXXX() ;

This is the standard pattern for a JavaBean accessor method for a property named "XXXX". In addition, we have methods like:

    boolean isJavaSerializationEnabled() ;

    boolean showInfoMessages() ;

A method name that returns a boolean and starts with "is" is a JavaBean accessor method. Unfortunately ORBData has a lot of boolean methods that do not start with "is", so we cannot directly use the JavaBean Introspector. However, the job is simple enough that that is not necessary. Alternatively we could pull all of the information out of the ParserTable class. In any case, the idea is to create a MonitoredObject for ORBData that reflects all of the configuration data for the ORB. We will discuss why this is useful a bit later.

Some ORB Monitoring Problems

Currently there are two different ways of getting information the root monitored object: either we call

MonitoringFactories.getMonitoringManagerFactory().createMonitoringManager(
  (root name), description ).getRootMonitoredObject()

or we call

orb.getMonitoringManager().getRootMonitoredObject()

Having two different ways of getting information is a frequent source of problems. These two mechanisms agree when the (root name) is the same as the corresponding value in the ORB, but that is currently not the case, due to the change I made to make sure that the monitoring code does not get overwritten by multiple ORBs. Hosever, the createMonitoringManager call will create a new MonitoringManager and root MonitoredObject whenever it is called. Currently I see two difference MonitoringManager objects in the app server: one with root named "orb", and one with root named "orb_S1AS-ORB_1", which comes from the app server's standard ORB id and the change I made in ORB b42 to fix bug 4919770.

Another problem occurs here because the ThreadPool always uses the first form to get to the root MonitoredObject, which is indepedent of the ORB. This means that all ThreadPools, no matter how they are created or where they are used, will be associated with the root MonitoredObject that corresponds to the first ORB created with ORB id "". This means that the threadpool monitoring will work regardless of the ORB id in the current app server. I still cannot reproduce Sheetal's bug, but it seems unlikely to be a significant problem.

On the other hand, the connection monitoring goes straight to the ORB to get the root monitored object. Some suggestions for 8.2:

  1. There should be only one possible root MonitoredObject, and a single MonitoringManager. The ORB should create its monitoring root under this top-level (probably unnamed) root object, using the current naming convention.

  2. The ThreadPool should continue to be ORB independent, as it can be used anywhere. However, the threadpool really should not both be independent of the ORB and have its monitoring registered under the ORB. Instead, we should register the threadpool under its own root, independent of the ORB.

  3. We only need one MonitoringManager, which simply provides access to / in the monitoring hierarchy.

Current hierarchy (multiple named roots):

orb
        threadpool
        Connections
                Inbound
                Outbound
orb_XXX_1
orb_XXX_2
etc

New hierarchy (this requires app server changes):

(root)
        threadpoolroot
                threadpool_1
                threadpool_2
                (etc. if we create multiple ThreadPoolManagers)
        orbroot
                orb_XXX_1
                        Connections
                                Inbound
                                Outbound
                        (plus more TBD as we work on 8.2 and 9)
                orb_XXX_2
                orb_YYY_1

Note that the old hierarchy is confusing when there are multiple ORBs. There is no confusion in the new hierarchy. For example,

orb.getMonitoringManager().getRootMonitoredObject() 

should just return the ORB's root, which is:

/orbroot/orb_XXX_n

MonitoringFactories.getRootMonitoredObject should just return / (and this replaces the current MonitoringManagerFactory.createMonitoringManager call).

Adding Notifications to ORB Monitoring

Given the possibility of changing MonitoredAttribute values, it becomes necessary to consider how the ORB can find out about changes. For this, we need to extend the ORB monitoring framework with events and event listeners.

Model:

  1. An attribute set method is called.

  2. The attribute examines the registered listeners and invokes the attributeChanged method on each listener, giving the listener the MonitoredObject, the MonitoredAttribute, and the old and new values of the attribute.

  3. Control returns from the attribute set method.

The other case we need to consider is creating and destroying MonitoredObjects. Here we want to receive either an ObjectCreated or an ObjectDestroyed notification on the parent MonitoredObject of the MonitoredObject that was created or removed. Here the notification just needs the parent and child MonitoredObjects.

A listener needs to implement the following interfaces as needed:

interface AttributeListener {
    void attributeChanged( MonitoredObject object, MonitoredAttribute attribute, Object oldValue,
        Object newValue ) ;
}

interface ObjectListener {
    void objectCreated( MonitoredObject parent, MonitoredObject child ) ;
    void objectRemoved( MonitoredObject parent, MonitoredObject child ) ;
} ;

MonitoredObject needs new methods to handle the registration of listeners. I think it may also be useful to have a scope associated with the registration, so that a listener can be registered either for events from one MonitoredObject, from the immediate descendants of a MonitoredObject, or from all descendants of a MonitoredObject. Since we are still using J2SE 1.4, we will map these scope values to integer constants. Then we need to add the following methods to MonitoredObject:

interface MonitoredObject {
    ...
    void registerAttributeListener( AttributeListener listener, int scope ) ;

    AttributeListener[]  attributeListeners() ;

    void removeAttributeListener( AttributeListener listener ) ;

    void registerObjectListener( ObjectListener listener, int scope ) ;

    ObjectListener[] objectListeners() ;

    void removeObjectListener( ObjectListener listener ) ;
}

An Architecture for Dynamic Configuration

The basic idea here is simple: provide attribute change notification events and configurations agends that listen for these events. But how exactly does this work?

First, I think we do not need the complex model that JavaBeans use, where some events can be vetoed. We still need to decide what the notification model should be, what events we support, and how those events get translated into Mbean notifications.

ORB dynamic MBean Support

The ORB will probably need to be configurable in where it obtains an MBean server. On J2SE, the ORB should use java.lang.management.ManagementFactory.getPlatformMBeanServer. What to do in J2EE is still TBD.

What we need to define here is how to map the ORB MonitoredObject/MonitoredAttribute framework into MBeans using the JMX framework. There are some significant differences in approaches taken in the two framworks. The JMX approach is more complex, with more features including such items as relations. The ORB Monitoring Framework only supports a single hierarchy, which is built in to the definition of the MonitoredObject. The genesis of the ORB framework is for monitoring (mostly) performance-related attributes of the system, while JMX is intended to be a fully general management and monitoring framework.

Part of the intent here is to grow the ORB framework to be more general, so that it can also support ORB management. Primarily this comes about through the addition of notifications. So far I have seen little need to add actions, but that is also easily possible.

There are several problems to solve here:

  1. For each MonitoredObject, there is a dynamic MBean whose attributes correspond to the MonitoredAttributes of the Monitored Object.
  2. The name of the MBean is derived from the nesting of the MonitoredObjects starting at the root.
  3. There is an object in the MBean pacakge that listens for MonitoredObject events (registered against the root, scoped for all descendants). This handles events as follows:
    1. For an attributedChanged event, the attribute on the corresponding MBean is updated.
    2. For an objectCreated event, an MBean corresponding to the MonitoredObject is created and registered with the MBean server.
    3. For an objectDestroyed evend, the MBean corresponding to the destroyed MonitoredObject is de-registered from the MBean server.

Dynamic Tracing in Java Applications

Solaris 10 has recently introduced the dtrace facility, which provides extremely useful tools for dignosing and tuning the Solaris OS and C/C++ applications. What about Java? Here I am not just interested in things like the Java stack. Instead, I want to consider extremely large and complex apps like the app server, which has many interacting modules, each of which has its own unique requirements for monitoring. As monitoring is a big push for AS 8.2 and 9, it seems that we should seriously consider what we need to do here.

In fact, there is a project to do exactly this: it's called jtrace, which is work being done in Janet Koenig's group with Sanjay Radia and A. Sundar. More information is available in jplan, and it is currently scheduled for Dolphin (J2SE 7). Unfortunately this work is a very long way (3+ years for J2SE, probably >4 for J2EE) out.

I wonder if we could do something sooner? There are several possibilities:

  1. Add instrumentation at the ORB SPI level. This is a very specific case, and somewhat labor-intensive to add. The big question still remains about what to do for the data collection language: DTrace's D is quite nice and fairly large.
  2. Similar to 1, but add the instrumentation at the Container level (in the component-based development sense; see components for a draft of these ideas).
  3. Combine several ideas: java.lang.Instrument in J2SE 5.0, byte-code instrumentation using ASM or BCEL, remote communications (perhaps using CORBA?), and the current Dtrace to create more-or-less what is needed. The key piece here is implementing a custom provider for DTrace (for which there is no information at present).
  4. It might be nice to be able to use MBeans as well with a tracing tool of some sort.