Author: Steve Totten

## Fault Tolerant (FT) CORBA Services

Points of contact: Dale Wilson and Steve Totten

- Introduction
- FT CORBA Services
  - Replication Manager
  - Fault Notifier
  - Fault Detector
  - Fault Detector Factory
  - Redundancy of FT CORBA Infrastructure Services
- Sample FT Application
  - Replica Factory
  - Replica
  - Object Group Creator
  - Client
  - Prototype Architecture
- Propagating IOGRs
- IOGR Creation and Manipulation
- Bootstrapping of FT CORBA Infrastructure and Application
- Future Work

### Introduction

Object Computing, Inc. (OCI) and the Distributed Object Computing (DOC) group at Vanderbilt University's Institute for Software Intensive Systems (ISIS) collaborated on a research and development (R&D) effort to demonstrate the viability of the OMG FT CORBA specification (defined in Chapter 23 of the CORBA 3.0 specification), with some extensions, as a platform for building fault-tolerant DRE applications.

The OCI team designed, implemented, and tested FT service-level entities and other components needed to support an FT infrastructure in TAO, and a sample application to demonstrate TAO's FT capabilities. The ISIS team provided enhancements to TAO's ORB Core to support fault tolerance in applications, including implementing ORB-core-level features defined in the FT CORBA specification.

Extensions to the FT CORBA specification investigated during the project included:

1. Adding a `SEMI_ACTIVE` replication style similar to that described here.
2. Separating interfaces and type definitions that are common across multiple specifications into a Portable Group module as described in the OMG Data Parallel Processing specification and the Unreliable Multicast Inter-ORB Protocol specification
3. Adding factory registration and a fault detector factory interfaces.
4. Adding mechanisms for bootstrapping FT CORBA systems.
5. Defining protocols for operating between the ORB core and FT services.

### FT CORBA Services

#### Replication Manager

The Replication Manager is perhaps the most visible of FT CORBA's infrastructure components. Fault tolerant services interact with the replication manager to create object groups, manage an object group's properties, control an object group's membership, and so forth. The Replication Manager is also solely responsible for the creation and maintenance of Interoperable Object Group References (IOGRs). According to the FT CORBA specification, the Replication Manager's operations are defined by three separate interfaces:

- Property Manager: Defines operations for setting properties, such as replication style, at several levels: by group, by type, or by default.
- Object Group Manager: Defines operations to add and remove members of an object group, change the primary member of an object group (for passive replication styles only), specify or get the locations of object group members, and get the current object group reference and object group identifier.
- Generic Factory: Defines operations for creating and destroying objects. The replication manager's realization of these interfaces effect the creation and destruction of object groups. These operations are also realized by application specific object factories to create and destroy replicas as object groups are created and maintained.

Note: The Data Parallel (DP) CORBA final adopted specification defines a new `PortableGroup` module, including the three interfaces listed above, to share common interfaces and their supporting types among DP CORBA, FT CORBA, Load Balancing, and other specifications. It is identical to a subset of the FT CORBA specification with a few changes to make it more generic to group management. TAO already has an implementation of the PortableGroup module that we adapted for reuse by the Replication Manager's implementation for this project.

In addition, the Replication Manager serves the role of a consumer for fault report events propagated to it via the Fault Notifier, so it must realize the Structured Push Consumer interface from the CosNotifyComm module (defined in the OMG's Notification Service specification (formal/02-08-04). Our design provides a Fault Consumer/Analyzer framework and concrete fault consumer and fault analyzer implementations that are tailored for use by the Replication Manager. The classes making up this framework, and the relationships among them, are shown in the figure below.

#### Figure 1: Fault Consumer/Analyzer Framework

The OCI team also added a Factory Registry interface that is also realized by the Replication Manager to allow various types of factories, such as Replica Factories and Fault Detector Factories (all of which implement the Generic Factory interface) to register with the Replication Manager. The Replication Manager then uses these factories when necessary to add members (replicas) to object groups or to create a new Fault Detector at a specific location to monitor a replica.

The Replication Manager's interfaces are defined in FT_ReplicationManager.idl. The Replication Manager also supports interfaces and types from the PortableGroup module that is defined in PortableGroup.idl and additional interfaces and types from the FT module that is defined in FT_CORBA.idl. Source code for the Replication Manager's implementation is found in the FT_ReplicationManager directory.

#### Fault Notifier

FT CORBA's Fault Notifier is based upon a subset of the OMG's Notification Service specification (formal/02-08-04). The Fault Notifier typically gathers fault reports from Fault Detectors as well as from application- or platform-specific fault detectors.

The Fault Notifier can support an arbitrary number of Fault Detectors and consumers because it is based upon the Notification Service. Components interested in receiving fault reports assume the role of push consumer with respect to the Fault Notifier. The Replication Manager is one such component, as described above; an application may provide its own fault analysis capability by connecting an application-specific fault analyzer as a consumer to the Fault Notifier. (In fact, a real-world application will likely participate intimately in identifying and analyzing faults. One way this could be done is to "plug-in" an application-specific fault analyzer to the Replication Manager, using the Fault Consumer/Analyzer framework described above.)

The Fault Notifier's interfaces are defined in FT_Notifier.idl. Source code for the Fault Notifier's implementation is found in the Fault_Notifier directory.

#### Fault Detector

The Fault Detector is the basic component in FT CORBA for monitoring a fault tolerant system's software components, processes, and processing nodes and reporting faults. The FT CORBA specification defines a single monitoring style, the pull monitoring style, in which a Fault Detector periodically issues a CORBA request (`is_alive`) to monitored objects and reports faults for those objects that fail to respond. A fault detector that monitors a single replica may be co-located on the same host as that replica. If the replica fails (defined as failure to reply to the detector's `is_alive` invocation within a prescribed time-out period), the detector issues a fault report to the Fault Notifier, which it finds via the Replication Manager. In our example application, the detector then exits since the replica that it was monitoring no longer exists. Fault detectors can also be deployed on other nodes and used to monitor other FT CORBA infrastructure components, such as a Fault Detector or Fault Detector Factory on another host. The pull monitoring style is used to monitor these components as well.

#### Fault Detector Factory

Fault Detectors are created and managed by a Fault Detector Factory. There may be many Fault Detector Factories deployed in a typical fault tolerant system. The Fault Detector Factory implements the Generic Factory interface.

Fault Detectors are created in the same process as their Fault Detector Factory. Each Fault Detector runs in its own thread and monitors its replica according to its prescribed monitoring interval (defined by a property on the object group). The Fault Detector Factory owns the thread manager for these threads. If a replica member is removed from an object group, the Fault Detector Factory that "owns" the Fault Detector that is monitoring that replica can cause the detector to "quit," thereby causing it to clean up any resources and its thread to exit.

Fault Detector Factories register with the Replication Manager via the Factory Registry interface. The Replication Manager then uses these Fault Detector Factories to create new Fault Detectors as needed to monitor replicas as they are created.

The Fault Detector Factory's interfaces are defined in FT_FaultDetectorFactory.idl. Source code for the Fault Detector and Fault Detector Factory implementations is found in the Fault_Detector directory.

#### Redundancy of FT CORBA Infrastructure Services

To achieve fault tolerance, a system must not have a single point of failure. This includes not only application services, but infrastructure services as well. In the case of this project, the following FT infrastructure services need to be made fault tolerant via redundancy:

- Replication Manager
- Fault Notifier
- Fault Detector Factories
- Replica Factories

One of the initial goals of this project was to provide redundant implementations of each of these services after first providing basic non-redundant implementations. Unfortunately, due to complexities encountered during implementation and the relatively short time frame for the project, we did not complete development of redundant versions of the various FT infrastructure services. We encourage this to be given a high priority for any follow-on work.

Note that making the Replication Manager redundant will require direct access to the lower-level state synchronization mechanism (i.e., via a synchronization strategy) while other FT infrastructure services can likely be made fault tolerant using the full range of FT CORBA mechanisms.

### Sample FT Application

#### Replica Factory

A Replica Factory is an application-defined entity that implements the Generic Factory interface. There may be many Replica Factories deployed in a typical fault tolerant application. Each Replica Factory acts as an agent for the Replication Manager to create and manage replica members of object groups of a specific type at a specific location. Replica Factories register with the Replication Manager via the Factory Registry interface. The Replication Manager then uses these Replica Factories to create new replicas as needed when creating object groups or adding new members to existing object groups.

We have provided a sample implementation of a Replica Factory as part of our example application for this project. It implements the Generic Factory interface from the FT module defined in FT_CORBA.idl. Source code for the example application's Replica Factory is found in the FT_App directory.

#### Replica

A Replica is an application object that serves as a member of an object group. Each replica implements an application-defined interface. In addition, each replica must implement the Pull Monitorable interface so it can be monitored by a Fault Detector. Replicas are created by Replica Factories by the Replication Manager or by another application. Each new replica is then added to an object group and managed by the Replication Manager.

We have provided a sample implementation of a Replica as part of our example application for this project. A Replica must implement the `PullMonitorable`, `Checkpointable`, and `Updateable` interfaces, which are defined in FT_Replica.idl. For our example application, a test replica interface is defined in FT_App/FT_TestReplica.idl. The implementation of the test replica is also in the FT_App directory.

#### Object Group Creator

The Object Group Creator is a utility for creating an object group. It can be used by an application to create an initial set of objects in a system. The Object Group Creator finds the Replication Manager and uses its Factory Registry interface to get a list of factories it can use to create objects of the desired type. The Object Group Creator can be used in different ways depending upon if the object group's `MembershipStyle` property value is application-controlled membership or infrastructure-controlled membership.

- Application-controlled membership: If application-controlled membership is being used, the Object Group Creator calls the Replication Manager to create an empty object group, then calls the factories to create members for the group. Members are added via the Replication Manager's `add_member` operation.
- Infrastructure-controlled membership: If infrastructure-controlled membership is being used, and the object creator is configured to set factories at the type level, the object creator optionally passes the set of factories to the `set_type_properties` operation of the Replication Manager, then calls Replication Manager's `create_object` operation to create an object group.

After creating the object group, the Object Group Creator can optionally write the group's IOGR to a file or bind it in the Naming Service so it can be accessed by clients.

The Object Group Creator can exist as a stand-alone utility or it can be integrated with an application. Our example application includes an implementation of the Object Group Creator in the FT_App directory.

#### Client

A client application obtains the object group reference from a file or from the Naming Service and invokes operations on it as it would a normal IOR. In the `SEMI_ACTIVE` replication style, only the primary replica receives and processes each request. The state synchronization strategy developed by the ISIS team for this project is used synchronize state between the primary and backup replicas with the completion of each request. If the primary replica fails, the transparent reinvocation mechanism inherent in the FT ORB (also developed by the ISIS team) causes the client's failed request to be automatically reinvoked on a backup replica. Meanwhile, the fault detection mechanisms described above are used to notify the Replication Manager of the fault and the Replication Manager takes the necessary actions to maintain the object group's integrity. Our example application includes a simple client in the FT_App directory.

#### Prototype Architecture

The figure below shows the architecture of a prototypical FT system and the relationships among the various FT infrastructure and application-defined components described above.

#### Figure 2: Architecture of Prototypical FT System

The steps involved in orderly start-up and operation of an FT system are numbered in Figure 2 and described below:

1. Start the Naming Service. (This step is optional as none of the FT components actually depends upon the Naming Service.)
2. Start the Replication Manager.
3. Start the Fault Notifier.
4. The Fault Notifier finds the Replication Manager and registers with it.
5. The Replication Manager connects as a consumer to the Fault Notifier.
6. Start one or more Fault Detector Factories.
7. The Fault Detector Factories register with the Replication Manager's Factory Registry.
8. Start one or more Replica Factories.
9. The Replica Factories register with the Replication Manager's Factory Registry.
10. Start the Object Group Creator.
11. (not shown) The Object Group Creator finds the Replication Manager and gets a list of Fault Detector Factories for the Replication Manager's Factory Registry.
12. (not shown) The Object Group Creator gets a list of Replica Factories from the Replication Manager's Factory Registry.
13. The Object Group Creator creates an object group via the Replication Manager's Generic Factory interface.
14. The Object Group Creator creates one or more Replicas via Replica Factories.
15. Each Replica Factory creates a Replica.
16. The Object Group Creator creates a Fault Detector for each Replica via the Fault Detector Factories.
17. Each Fault Detector Factory creates a Fault Detector for a Replica.
18. Each Fault Detector finds the Replication Manager and gets the Fault Notifier from the Replication Manager.
19. Each Fault Detector connects as a supplier to the Fault Notifier.
20. The Object Group Creator adds each Replica as a member to the object group via the Replication Manager's Object Group Manager interface.
21. The Replication Manager generates a new IOGR for each added Replica and updates each Replica member of the object group with the new IOGR.
22. The Object Group Creator optionally binds the IOGR of the object group with the Naming Service or publishes its IOGR in some other way, such as a file.
23. Start a Client.
24. The Client optionally resolves the object group by name from the Naming Service or resolves it in some other way, such as from a file or via a corbaloc ObjectURL.
25. The Client invokes a request on the object group. This request is carried out by the primary Replica of the object group.
26. Each Fault Detector periodically pings its Replica via the Replica's PullMonitorable interface.
27. If a Replica fails, the Fault Detector pushes a structured fault report to the Fault Notifier.
28. The Fault Notifier pushes the structured fault report as an event to the Replication Manager's consumer.
29. (not shown) The Replication Manager removes the failed member from the object group, selects a new primary for the object group, generates a new IOGR, and updates each Replica member of the object group with the new IOGR.
30. (not shown) The Replication Manager may also add new members to the object group if the number of replicas has fallen below the object group's MinimumNumberReplicas property. When it adds new members, the Replication Manager also generates a new IOGR and updates each Replica member of the object group with the new IOGR.

### Propagating IOGRs

The FT CORBA specification requires the Replication Manager to create and maintain IOGRs. It also requires the FT ORB to perform most-recent IOGR processing, whereby the FT ORB can update a client using an old IOGR, by means of a `LOCATION_FORWARD` reply, with a new IOGR. However, the specification fails to define a way for the Replication Manager to propagate revised IOGRs to the FT ORBs of object group members. Therefore, the OCI and ISIS teams agreed upon a simple interface (`tao_update_iogr`) by which the Replication Manager can propagate revised IOGRs to the FT ORB for each member of an object group (e.g., after failure of a primary replica, selection of a new primary member, and generation of a new IOGR by the Replication Manager). While this interface is TAO-specific, it accomplishes one of our research goals of investigating formal protocols by which different ORB implementations of FT CORBA could be made interoperable. The ISIS team implemented this interface and the OCI team incorporated its use within the Replication Manager.

### IOGR Creation and Manipulation

To support FT CORBA, the Replication Manager must be able to create and manipulate IOGRs. For example, the Replication Manager's realization of the Generic Factory interface must return an IOGR. Also, upon receiving a fault report on an object group, the Replication Manager may need to remove a member, designate a member as the new primary replica, and generate a new IOGR that can then be propagated to each member of the object group.

For the purposes of this project, we used TAO's existing IORManipulation library for creating and managing IOGRs. However, the IORManipulation library lacked certain features that were needed. We worked with the ISIS team to define extensions to the IORManipulation library to:

- Support the creation of "empty" IORs that have no profiles.
- Support the complete replacement of all the profiles in an IOR.
- Support the addition of the following tagged components:

- TAG_MULTIPLE_COMPONENTS
- TAG_GROUP
- TAG_FT_PRIMARY

While the IORManipulation library can be used to create and manipulate IOGRs, a longer term approach may be to use a specialized implementation of the Object Reference Template and IORInterceptor abstractions defined in sections 21.5.3 and 21.5.4 of the CORBA 3.0 specification, respectively.

### Bootstrapping of FT CORBA Infrastructure and Application

FT CORBA infrastructure and application components must collaborate to achieve fault tolerance. To do so, infrastructure and application components must be started and initial objects and object groups created in an orderly fashion. Much of this "bootstrapping" can be accomplished by scripting. However, an entity to control the creation of initial objects and object groups can greatly simplify certain aspects of the bootstrapping process.

The sample application provided as part of this project uses an Object Group Creator utility to create initial objects and object groups. The Object Group Creator is implemented as a library that can be easily integrated with other parts of an application. A simple wrapper is also provided allowing the Object Group Creator to be used as a stand-alone executable.

### Future Work

During the course of this research, we uncovered several areas for further research and development, including:

- Adding redundancy to the FT infrastructure services.
- Extending FT support to additional platforms, such as the VxWorks RTOS.
- Incorporating FT capabilities into an advanced application, such as TAO's RT Notification Service, or Naming Service.
- Integrating the FT RT Event Service with FT Services. The current implementation of FT RT Event Service in TAO is not based on the FT Services implemented by OCI because they are developed independently.
- Investigating improvements to the mechanisms used to detect faults in replicated application objects.
- Investigating advanced fault analysis capabilities and mechanisms by which application- or platform-specific fault analyzers to be "plugged" into the Replication Manager or other components.
- Investigating formal protocols that will allow interoperability among different ORB implementations of FT CORBA.
- Refactoring of the Replication Manager and Portable Group implementations to consolidate the representation of object groups.
- Employing standard mechanisms, such as Object Reference Template and IORInterceptors, instead of the TAO-specific IORManipulation library, for the creation and manipulation of IOGRs.
- Investigating performance and scalability issues in FT systems.
- Enabling the configuration and enforcement of FT quality of service (QoS) properties, such as bounds on fault detection and recovery times.