By downloading this software the user agrees to accept this software entirely as is, with absolutely no warranty whatsoever, expressed or implied. The responsibility for ensuring fitness and correctness for any purpose lies entirely with the user. Questions about this document to be sent to mead-support@lists.andrew.cmu.edu

• What is MEAD?
  • Fault-Tolerance Fundamentals
• Installation and Distribution
  • Currently Supported Configurations
  • Distribution
  • Known Caveats and Recommendations
  • Reporting Problems & Obtaining Support
• CORBA Application Requirements
• Building MEAD
  • Debug Build
• MEAD Library Parameters
• Running and Configuring Spread
  • Spread Segment
  • Spread Timeouts
  • Spread Related Errors
• Running MEAD on Emulab
  • Must-Read Notes
• Other Configurations
• Trouble-shooting
• References
• Contributors
  

What is MEAD?

The Middleware for Embedded Adaptive Dependability (MEAD) infrastructure aims to enhance distributed real-time middleware applications with new capabilities including:

1. Transparent, yet tunable, fault-tolerance in real time
2. Proactive dependability
3. Resource-aware system adaptation to crash, communication and timing faults
4. Scalable and fast fault-detection and fault-recovery

More information, including publications on the MEAD system, can be found at the MEAD website.

Fault-Tolerance Fundamentals

This release of MEAD focuses on non-adaptive replication, i.e., the replication style does not change on the fly. Given the current state-of-the-art in ORBs, which are not completely stateless black-boxes, the unit of replication (also known as a replica) is effectively the process or the container, and not the individual object or the component. Note that replicating processes automatically replicates the components/objects hosted within those processes. The replication styles currently offered by MEAD are active replication and warm passive replication.

Active replication (see figure immediately below) allows multiple replicas when spawned, to join a replica group. Each individual replica processes each invocation and sends results back to the client. The beauty lies in the fact that MEAD suppresses duplicate responses, and the client believes it is communicating with a single server. Checkpoints (i.e., state snapshots) are triggered in active replication only when a new replica is launched and needs to synchronize its state with those of the other running replicas.

Warm passive replication (see figure immediately below) designates one replica to be the primary and the others to be backups, in the event of the primary's failure. Only the primary replica processes invocations from the client and responds to the client. All backups in the replica group periodically receive checkpoints from the primary that allow them to synchronize their respective states with that of the primary.

Both replication styles offer fault tolerance through redundancy. Active replication can be resource intensive (because of increased CPU usage and bandwidth usage), but provides faster response and recovery times. On the other hand, warm passive replication typically conserves resources, but the response time is bounded by the network and processing speed of the primary replica. Also, increased bandwidth and CPU usage might result if the size and frequency of checkpoints is sufficiently large. Warm passive replication's recovery times are also slower because the recovery time includes failover time from the crashed primary to the backup.

Installation and Distribution

Emulab is the distributed test-bed of choice for the MEAD system. If you have never used Emulab before, please refer to the Emulab "Getting Started" Tutorial. More details to get MEAD running on Emulab can be found in the Running MEAD on Emulab section.

The current version of MEAD supports, and has been tested with, the following platforms, ORBs and third-party software.

Platforms: Red Hat Linux 9 (The BBN-RH9-SS7-8 image on Emulab).
Compiler: g++ 3.2.2
Networks: 100Mbps Ethernet
ORBs: TAO v 1.4.1 and ACE v 5.4.1
Third-Party Software: Spread group communication system v 3.17.1

MEAD uses the Spread group communication system for inter-object communication. More details can be found below under the Running and Configuring Spread section.

The directory containing the latest release of the MEAD software distribution is located on Emulab at /groups/pces/uav_oep/mead_cmu/release/mead.

The current MEAD release is comprised of library code for the MEAD replication mechanism, pre-built Spread libraries for Red Hat 9, a stateful CORBA application in the counter directory and a stateless CORBA application in the stateless directory, a readme file, makefiles, and scripts for execution. MEAD is comprised of the following directory structure:

./MEAD    (root of the MEAD distribution)
    /counter
    /docs
    /replication
          /bin
          /include
          /lib
          /obj
          /src
    /scripts
    /spreadlinuxbin
          /docs
          /include
    /stateless

The replication directory contains the mead library and replication mechanism. The directory spreadlinuxbin contains the pre-built Spread libraries. Finally, the scripts directory contains bash scripts that can be used to load and run the sample CORBA applications.

The current version of MEAD provides support for CORBA clients to discover servers through the Interoperable Object Reference (IOR) mechanism. The server outputs its IOR into a file that the client can subsequently read, in order to connect to the server. Future versions of MEAD will support the use of the CORBA Naming Service in order for clients to discover servers.

In the interests of fault containment and effective replication, we recommend hosting only one replica of a server per node. While it is possible to host more than one server replica per node, if the node is to contain the fault, and a single processor-crash should not affect more than one replica, running only one replica of a server process per node is ideal.

The current version of MEAD requires the CORBA application to be deterministic in behavior, i.e., any two replicas of the server, when starting from the same initial state, and receiving the same set of invocations in the same order, should reach the same final state. This eliminates the use of local timers, shared memory and any other OS primitive that can lead to irreproducible behavior across different nodes in a distributed system. Determinism is a common assumption in the development of fault-tolerant distributed systems; while ongoing development in the MEAD project aims to eliminate the need for determinism, the current version of the MEAD requires this of the application.

If you face a specific problem with running or installing the MEAD system, please fill out the details of the problem using the Support Request Form. For more general questions on MEAD, please email us at mead-support@lists.andrew.cmu.edu

CORBA Application Requirements

In order to restore application-level state in the event of a server crash, the Fault-Tolerant CORBA standard requires every CORBA object to support an additional Checkpointable interface, with methods for the retrieval and assignment of application-level state. This interface is an abstract class and can be inherited by the application when working with MEAD. Note that State can be defined in a custom way, based on the CORBA object's state. MEAD simply invokes these methods in order to perform checkpointing and state transfer for both warm passive and active replication. After implementing these functions, the application programmer does not need to worry further about them (except, of course, for updating the implementations of these methods should the application's definition of state change) MEAD handles ORB-level and infrastructure-level state so that the application programmer does not need to worry about them..

   exception NoStateAvailable{};
  exception InvalidState{};

  interface Checkpointable {
    State get_state() raises(NoStateAvailable);
    void set_state(in State s) raises(InvalidState);
  };

When developing CORBA applications for MEAD, it is necessary to implement get_state() and set_state() at the object level, and get_global_state() and set_global_state() for void pointer data at the application level (extern functions).

At the application level, the example below uses void pointers to offer a less restrictive implementation. The data type state can be any type of structure containing all of the current state for the process as well as CORBA object state from above. This C code can be used to declare the external functions for get_global_state() and set_global_state():

  extern "C" void *get_global_state (int *size);
 extern "C" void set_global_state (void * state, int size);

The sample program "counter" contains simple usage of both the Checkpointable interface and the extern'd functions.

Building MEAD

The MEAD library (libmead.so) requires both ACE/TAO and Spread to function. These must be installed and correctly configured prior to building MEAD. More details can be found at the TAO main website and Spread website, respectively. Documentation and tutorial information on TAO development can be found at the TAO development website.

Makefiles tailored to BBN-RH9-SS7-8 are included in the replication, stateless, counter, and the scripts directories. The makefiles found in the counter and stateless directories build client and server CORBA applications. The replication directory's makefile will build the MEAD library as well as a timing library that can be used for empirical evaluation. The scripts directory supports a "make clean" option to remove any unnecessary remnants caused by IOR files.

Before running make, remember to "source source_this_for_Emulab" at the root of the MEAD distribution.

The replication makefile makes use of the Spread library (libspread.so) found in the spreadlinuxbin directory. If not using Red Hat 9 it will be necessary to install and configure Spread on your target platform. For convenience this MEAD distribution includes pre-built Spread binaries in the spreadlinuxbin directory.

Finally, both the sample CORBA applications as well as the replication library makefiles can compile the target application in debug mode. This will enable the inner workings of the replication library as well as the CORBA applications to be viewed from the standard output.

To Enable Debugging

To enable debugging, look for the mINCLUDES (counter/stateless) or CPPFLAGS (replication) variable that enables debugging in the respective makefiles. They should be preceded by a comment about debugging and will essentially enable -DDEBUG.

Running and Configuring Spread

Spread Overview

Spread is a group communication system developed at John Hopkins University and currently developed by Spread Concepts LLC and CNDS at John Hopkins University. Additional information can be found at the Spread website as well as the Spread User Manual.

The Spread daemon can be launched using this command:
                 % spread -c <spread.config.file>

Spread Segment

The Spread configuration file contains options for the Spread daemon, and comments on the configuration of each option. MEAD is mostly concerned with the Spread_Segment { }, which contains a list of <hostname> <IP> pairs for each node in the cluster, as well as a broadcast address and port for execution. The following is a sample configuration file used in the Emulab environment. The broadcast address can differ in range (i.e., in the first three dotted-decimal places) from the IP addresses of the three nodes; this is specific to the Emulab environment where the nodes seem to be multi-homed. The broadcast is for local connectivity and the individual host IPs are actual addresses that can be resolved for the machine.

  Spread_Segment BroadcastAddress:SpreadPortNumber {
      node0  node0_IP_address
      node1  node1_IP_address
      node2  node2_IP_address
        :      :
  }

Finally, only use the computer name for the <Hostname> parameter and not the fully qualified name for the computer (e.g. use node0 not node0.test.pces.emulab.net).

Spread Timeouts

When compiling the Spread library and the Spread daemon, there is a series of timeouts that can be set can be configured and compiled. These can significantly affect the performance of both Spread and MEAD. More details can be found in the Spread User Manual.

Spread Related Errors

The following lists a few common errors related to configuration of spread that are returned from the MEAD library, more details can be found at: http://www.spread.org/docs/docspread.html

• MEAD: Could not connect to SPREAD daemon! (error=-2)
  • The Spread_Segment may have not been setup properly to include the <Hostname> <IP> where the current application resides.
  • The port used in the Spread_Segment may be different from the port used in the MEAD library

Running MEAD on Emulab

The primary test-bed for MEAD software is Emulab. The sequence of steps below details a run using the BBN-RH9-SS7-8 Emulab image. This run will replicate three servers in either active or warm passive replication styles (you can configure your choice) and will use a single client to make requests of the three-way replicated server. Some of the following steps need to be executed on every assigned Emulab node (step is prefaced by [every node]) or only on one assigned Emulab node (step is prefaced by [one node]). The symbol % represents the shell command-line prompt.

1. Setting up the Emulab Experiment
   • Enter the following values in the "Begin a Testbed Experiment" form on Emulab
     • Select Project: pces (If this choice is not available to you, please contact us).
     • Group: pces/uav_oep
     • Description: Replication with MEAD
     • Your NS file: Please download the sample NS file that we have provided in order to simplify the configuration of the topology for this experimental run. You can edit this file for future runs.
   • The topology consists of four nodes, one 100Mbps LAN; three nodes will be used to host three individual server replicas, and the fourth node will be used for the client application.
   • Use the OSID BBN-RH9-SS7-8 for each of the nodes when setting up your experiment with Emulab.
   • Wait to receive an email from the Emulab Testbed Operations team (notifying you of the availability of your requested experimental configuration) before proceeding further.
   • Once you receive the notification about your experiment being started, ssh into the machines assigned to you, according to the instructions on Emulab.
2. [any node] Obtaining the MEAD source distribution
   • The MEAD distribution is available on users.emulab.net in the directory: /groups/pces/uav_oep/mead_cmu/release/mead/
   • Use cp (or scp) to copy the MEAD source to your working directory (for convenience, copy mead-1.0.tar.gz).
   • If you copied the .tar.gz file, then, you should do the following to extract the MEAD directory:
     % tar -zxf mead-1.0.tar.gz
     A directory called MEAD is created in the process of extracting the .tar.gz file.
3. [every node] Obtaining the right environmental settings
   • Enter the MEAD directory and obtain the right environmental settings for your binary and library search-paths:
     % source ./source_this_for_Emulab
4. [any node] Building the MEAD library and the test applications
   • Enter the replication sub-directory and type make, e.g.,
     % cd replication
     % make
   • Similarly, enter the counter sub-directoryand type make
   • Similarly, enter the stateless sub-directory and type make
5. Setting up the Spread configuration
   • Find out the following information for your assigned Emulab nodes: each node's IP address, each node's hostname, and the broadcast address.
     • [any node] The Emulab machines appear to be multi-homed (five interfaces per node). To see this for yourself, try the following:
       % ifconfig -a
       You will see interfaces eth0 through eth4 defined.
     • [any node] For the broadcast address, look at the output of:
       % ifconfig eth3
       and replace the last part of the dotted-decimal IP address with 255, e.g., if you see an IP address of 10.1.1.4, the broadcast address is 10.1.1.255
     • [every node] For each node's IP address, look at the output of::
       % hostname -i
       to obtain the IP address associated with the interface.
     • [every node] For each node's hostname, look at the output of::
       % hostname
       to obtain the qualified hostname associated with the interface. You will only need the first part of this address, i.e., for node8. *.pces.emulab.net, you only need to remember node8).
   • [any node] Edit the file MEAD/scripts/spread.conf.emulab to specify the set of Emulab nodes that you are using for your experiment.
     • Edit the Spread_Segment by listing the <hostname> <ip> pairs within the parentheses.
     • The default Spread port number is 6011, but can be changed; however, the new value of the port must be identical in two places: (i) the Emulab configuration file MEAD/scripts/spread.conf.emulab, and (ii) the SPREAD_PORT environment variable supplied to the application (and, indirectly, to MEAD) at run-time.
     • Here is an example for a broadcast address of 10.1.1.255, a Spread port number of 6011 and four nodes (node6, node7, node8 and node9) with their respective IP addresses:
         Spread_Segment 10.1.1.255:6011 {
              node6  155.101.132.92
              node7  155.101.132.93
              node8  155.101.132.94
              node9  155.101.132.95
         }
       
6. [every node] Running Spread
   • Launch the Spread daemon (spread) on each host that will run either a server or a client. The Spread daemon can be launched from the MEAD root directory, as follows:
     % cd MEAD/scripts
     % ./startSpread
7. There are four scripts that demonstrate the execution of the two provided test applications. All of the scripts are found in the MEAD/scripts directory, and display usage information. The scripts will not launch if the Spread daemon is not running. The scripts provide settings for most of the MEAD environment variables. If you wish to bypass the scripts and run processes directly at the command-line, please read the MEAD Library Parameters section of this document.

   Choose one of the two applications below (either counter or stateless) and launch one server replica on each of the three nodes of your cluster. Finally, launch the client as instructed below. Servers (replicas) can be killed at any time using Control-C. Launching a server script will add a new server replica to the group, initialize its state, and the either start normal processing (for active replication) or state transfer (for warm passive replication).
   
   counter is a "stateful application", where the server maintains the count of user requests, and implements the Checkpointable interface for the CORBA objects and exports global state for the actual application.
   

   • To run a three-way actively replicated version of this test application:
     % cd MEAD/scripts
     % ./counterServer_run SERVERGROUPID ACTIVE
     % ./counterClient_run CLIENTGROUPID SERVERGROUPID
   • To run a three-way warm passively replicated version of this test application:
     % cd MEAD/scripts
     % ./counterServer_run SERVERGROUPID WARM_PASSIVE
     % ./counterClient_run CLIENTGROUPID SERVERGROUPID
   
   stateless is a "stateless application" that sends a string containing the process identifier of the process and the current number of requests that the object has serviced.
   
   • To run a three-way actively replicated version of this test application:
     % cd MEAD/scripts
     % ./statelessServer_run SERVERGROUPID ACTIVE
     % ./statelessClient_run CLIENTGROUPID SERVERGROUPID
   • To run a three-way warm passively replicated version of this test application:
     % cd MEAD/scripts
     % ./statelessServer_run SERVERGROUPID WARM_PASSIVE
     % ./statelessClient_run CLIENTGROUPID SERVERGROUPID

   After the server replicas are done processing invocations from the client, systematically kill (^C) the server replicas, until the system is down to the last remaining replica. Then, bring the other two server replicas back up. Finally kill the previously remaining server replica. The system will still continue to function, due to MEAD's fault-tolerant support.

8. [every node] Cleaning Up
   • Kill the Spread daemon (at the end of all your experiments, and not before, or during, your experiments):
     % cd MEAD/scripts
     % ./killSpread

Here are common things to watch out for, when running MEAD:

• The source_this_for_Emulab file must be sourced only from the MEAD root directory, where it is located.
• The Spread daemon must be running on every node on which you expect to run clients or servers using MEAD.
• The CLIENTGID and the SERVERGID should not be the same for a process; these identifiers represent the client and server object group names, respectively, and should be distinct.
• When replicating a server, ensure that all of the replicas of the server are launched with the same object group identifier and the same replication style, i.e., within the same group, all of the replicas must possess the same GID and should have the same REPLICATION_STYLE
  • For example, make sure when launching an actively replicated server, ensure that you use the same command on all three server machines (./counterServer_run GGG ACTIVE).
• Pure clients should always be launched using ACTIVE replication style in order to enable duplicate suppression (the client scripts in the MEAD/scripts directory automatically take care of this), regardless of the server's replication style.

MEAD Library Parameters

The MEAD library uses several environmental variables for configuration. The following table offers the names and descriptions of the variables as well as acceptable values.

Name	Description	Value
LD_PRELOAD	Used to specify that run time linking and loading should include LD_PRELOAD's libraries first. Note that the library will be loaded for all applications in that environment or process address space.	libmead.so.1
REPORT_CALLS	REPORT_CALLS is used to filter the debugging messages returned by the MEAD library. The parameter as a string value that includes the names of intercepted system call functions. Only used when the replication library is compiled using the -DDEBUG Option.	all, socket, read, writev, bind, select, connect, accept, ETC.
REPLICATION_STYLE	Used to set the replication style for the application that is being launched.	ACTIVE or WARM_PASSIVE
GID	Object group name.	alpha-numeric strings less than 26 bytes. We recommend 3 to 8 bytes.
SERVER_ID	Object group name for a server that a client application will connect to.	alpha-numeric strings less than 26 bytes. We recommend 3 to 8 bytes.
SERVER_ID2	Object group name for another type of server that the client can connect to.	alpha-numeric strings less than 26 bytes. We recommend 3 to 8 bytes.
IS_STATELESS	Lets MEAD know that the application is stateless.	yes/no
SPREAD_PORT	Port number that MEAD should look for the Spread daemon to run on.	dynamic port range only.
REPORT_ONLY	REPORT_ONLY set MEAD to only Report. Basically, this only intercepts calls, prints debug info and forwards the calls on. Debugging information will only be displayed if the application is compiled using the -DDEBUG option.	yes/no
EXPER_INFO	N/A	N/A

Other Configurations

MEAD will run in other environments that have configured functional copies of both ACE/TAO and Spread. The examples above should work as well provided that:

• The proper paths are set for the environment to build and runtime libraries.
• MEAD makefiles for both the MEAD library and the sample applications are modified to include the proper paths to libraries and include files
• The MEAD library is rebuilt using the version of Spread for the intended platform
• The sample CORBA applications are rebuilt using the version of ACE/TAO for the intended platform
• All of the scripts in the scripts sub-directory should be changed to include the proper path for Spread

Trouble-shooting

Title	Description/Possible Solution
LD_PRELOAD is not set	In most cases the CORBA application will still function, without the replication mechanism.
When Spread is not running	MEAD will return the following error message: Could not connect to SPREAD daemon! (error=-2). The application will not actually launch.
env variables are not correct	Critical environmental variables are checked during initialization. MEAD will return error messages that variables are not set.
LD_LIBRARY_PATH does not include right libraries	These are standard errors that will be returned by the run time linker and loader. In most cases they include the name of the library that has not been pathed properly.

References

• Publications on the MEAD sytem: http://www.ece.cmu.edu/~mead
• The Spread group communication system: http://www.spread.org/
• Emulab distributed test-bed: http://www.emulab.net/
  • Emulab "Getting Started" Tutorial
• Fault-Tolerance CORBA specification: http://www.omg.org/docs/formal/04-03-21.pdf

Contributors

Contributors to MEAD include Priya Narasimhan, Tudor Dumitras, Aaron Paulos, Soila Pertet, Charlie Reverte, Joe Slember and Deepti Srivastava.