Team 5: Virtual Casino

FT Test 2: Fault Tolerance with interactive replication manager.

Functionality:

Fault Tolerance

System Elements:

• The CORBA ORB: Sun's implementation of an ORB for java, supports naming and rpc's.
• Casino Server: The middleware server responsible for serving blackjack to clients. The game server registers one interface with the orb naming service, the Floor interface which contains methods allow entrance to and exit from the gaming system. It also exports two other interfaces on request; the Bank interface and the Tables interface. These handle the monetary transactions and the actual game mechanics respectively.
  
• Database Server: The MS-SQL system that stores all the state for one middleware server right now and multiple ones in the future.
• Replication Manager: The application that starts the servers, monitors them for failure, restarts them if they do fail and maintains the Naming Service.
• Player Application: The application that the player uses to play the game. It has one interface, the Player interface, that it exports to the game server to allow the game server to control the flow of game play.
• Player: The human controlling the client application

Use Case:

• Two or more Casino Servers are started on separate hosts by the Replication Manager, all using the same java orb. They are registered with sequential names in the naming service. One of these server is designate the "primary" and will handle all calls from Player Applications.
• One Player Application is started on the same or different host from the servers and it asks for and gets a reference to the primary Casino Server from the naming service.
• The Player is about to buy "chips" from the casino when the primary Casino Server is killed by the replication manager.
• The Replication Manager detects within one second that the primary server is no longer available. The Replication Manager then switchs the entry in the CORBA naming service for the primary server to point to the next Casino Server.
• The Player Application detects a CORBA communications exception and then goes to the naming service and asks for the primary Casino server again.
• The Player is able to buy chips using the new Primary Casino server with little delay.

Test distribution:

• This can only be run out of the build tree right now.
  

Setting up the System

• Installing the binaries.
  • All the project binaries other than the database are in the 'tar' file. Unpack this in your afs home directory. Since we have a distributed file system to work with we can install one image and use it on any host.
• Running the Database Server
  • The Database Server is considered "sacred" and is always running.
• Running the Casino Server
  • The Casino is controlled by the Replication Manager is is not generally started directly.
• Running the Replication Manager
  1. Log in to any host in the ECE linux cluster and make sure that you are using the BASH shell.
  2. Change directory to where you unpacked the tarball.
  3. Source the project environment file with the command ". .project.settings".
  4. Make sure that the orb is running or if it is not start it with the startOrb script. Wait a minute for the orb to finish starting.
  5. Start the Replication Manager with the command ./startRepman.sh repman.props. You will see output like this:
     
     [etwilson@boggle app]$ ./startRepman.sh
     
     **************************************
     * Virtual Casino Replication Manager *
     **************************************
     
     ---Setting Verbose Mode
     ---repman.host.list=battleship trouble yahtzee boggle candyland checkers chess chutesnladders clue devilbunny drlucky girltalk go jenga monopoly othello pictionary risk roborally scrabble settlers sorry stratego trivialpursuit
     ---repman.replica.count=2
     ---repman.start.command=cd ~/app; ./startServer.sh -v -p server.props.etwilson
     ---repman.kill.command=killall -9 java >/dev/null 2>/dev/null
     ---Created ORB instance.
     ---Connected to naming service.
     Enter command >
     
     
  6. You are now at the replication manager's prompt where typing "h" will give you the list of possible commands:
     
     Enter command > h
     
     List Hosts - list all hosts
     List Replicas - list host with running replicas
     List Available - list hosts available to run a replica
     STart <host>... - start new replicas on <host>s
     KIll <host> - kill replica running on <host>
     KIll ALL - kill all replicas and exit
     KIll Auto <sec> - automatically kill the primary every <sec> seconds
     KIll Auto OFF - stop automatic primary killing
     KIll Primary - kill the primary replica
     Set - display current settings
     Set Restart (ON|OFF) - restart replicas automatically / manually
     Set PIng <sec> - set ping interval to <sec> seconds
     Set Primary <host> - promote the replica running on <host> to primary
     Quit - exit the program
     Help - display command help
     
     Enter command >
     
     
  7. You now need to start two replicas using the "STart <host>" command:
     
     Enter command > start trouble battleship
     Info: Starting replica 'Floor0'
     ---Execing ssh trouble cd ~/app; ./startServer.sh -v -p server.props.etwilson -n 0 
     Floor0.out>Running server on trouble with PID: 30212
     ---try to find new replica
     ---not yet available
     Floor0.out>Info: Setting Verbose Mode
     Floor0.out>---Loading properties file: server.props.etwilson
     Floor0.out>
     Floor0.out>***********************************
     Floor0.out>* Virtual Casino Blackjack Server *
     Floor0.out>***********************************
     Floor0.out>
     Floor0.out>---Created ORB instance.
     Floor0.out>---Using database username of gene
     Floor0.out>---Using database pasword of philISGR8!
     ---try to find new replica
     ---not yet available
     ---try to find new replica
     ---not yet available
     ---try to find new replica
     ---not yet available
     Floor0.out>---Created Floor db object.
     Floor0.out>---Created Player db object.
     Floor0.out>---Created Bank db object.
     Floor0.out>---Created Table db object.
     Floor0.out>---Looking for naming service on roborally:6505
     ---try to find new replica
     ---not yet available
     Floor0.out>---Resolved NameService.
     Floor0.out>---Created Bank object.
     Floor0.out>---Created Tables object.
     Floor0.out>---Added name Floor0.
     Floor0.out>Info: Server ready and running .....
     ---try to find new replica
     ---new replica is OK
     Replica started on trouble
     Info: Starting replica 'Floor1'
     ---Execing ssh battleship cd ~/app; ./startServer.sh -v -p server.props.etwilson -n 1 
     Floor1.out>Running server on battleship with PID: 11769
     ---try to find new replica
     ---not yet available
     Floor1.out>Info: Setting Verbose Mode
     Floor1.out>---Loading properties file: server.props.etwilson
     Floor1.out>
     Floor1.out>***********************************
     Floor1.out>* Virtual Casino Blackjack Server *
     Floor1.out>***********************************
     Floor1.out>
     ---try to find new replica
     ---not yet available
     ---try to find new replica
     ---not yet available
     Floor1.out>---Created ORB instance.
     Floor1.out>---Using database username of gene
     Floor1.out>---Using database pasword of philISGR8!
     ---try to find new replica
     ---not yet available
     Floor1.out>---Created Floor db object.
     Floor1.out>---Created Player db object.
     Floor1.out>---Created Bank db object.
     Floor1.out>---Created Table db object.
     Floor1.out>---Looking for naming service on roborally:6505
     ---try to find new replica
     ---not yet available
     Floor1.out>---Resolved NameService.
     Floor1.out>---Created Bank object.
     Floor1.out>---Created Tables object.
     ---try to find new replica
     ---not yet available
     Floor1.out>---Added name Floor1.
     Floor1.out>Info: Server ready and running .....
     ---try to find new replica
     ---new replica is OK
     Replica started on battleship
     Enter command >
     
     
  8. You can see which replicas are running using the "list replicas" command or its shorter version "l r":
     
     Enter command > l r
     
     List of hosts running a replica:
        0   trouble
        1   battleship
     
     Enter command >
     
     
  9. You now need to set one of the replicas to be the primary using the "set primary <host>" command:
     
     Enter command > set primary trouble
     Info: Floor0 is the new primary
         trouble is now the primary
     Enter command > Floor0.out>There are 0 players in bank DB.
     
     
  10.  The last thing to do is to set restart to be automatic:
      
      Enter command > set restart on
          Automatic restart is now ON
      Enter command >
      
      
• Running a Player Application
  1. Log in to any host in the ECE linux cluster and make sure that you are using the BASH shell.
  2. Change directory to where you unpacked the tarball.
  3. Source the project environment file with the command ". .project.settings".
  4. Edit the player.props file to make sure that the location of the orb is properly specified.
  5. Start the Player Application with the command "startPlayer.sh", you will see something similar to this:
     
     [etwilson@trouble app]$ ./startPlayer.sh -p player.props.gene
     Starting the Player Application...
     
     
     ***************************************
     ** Virtual Casino Player Application **
     ***************************************
      ---Starting the Player Application...
      ---Setting up the Player Server thread ...
      ---Getting reference to the Player ...
     ...Registering the Player with the Floor
      Q--Please enter your Player name:
     
     

Running the Test

1. At the Player Application Prompt " Q--Please enter your Player name: ", enter your name and you will see output like this:
   
    Q--Please enter your Player name:
   GeneW
    ---Contacting the Casino Floor...
    ---Found the naming service
    ---Found the Floor interface
    -->Registering the Player with the Floor...
    ---Player is registered with the Floor with playerID = 1
    -->Registered the Player with the Naming Service as: 1
    ---Found the Bank interface
    ---Found the Tables interface
   ...Buy chips from the Bank
    Q--How many chips do you want to buy?
   
   
2. At this prompt, stop.
3. At the replication manager shell, type "KIll Primary".  You will see something similar to this:
   
   Enter command > kill primary
   ---Killing replica on trouble
   ---Execing ssh trouble killall -9 java >/dev/null 2>/dev/null 
   ---ExitValue: 0
   Replica on trouble has been killed
   Enter command > ---ExitValue(2): -119
   Info: Failed to ping replica #2
   Info: Floor1 is the new primary
   ---Unbinding 'Floor2'
   Floor1.out>There are 0 players in bank DB.
   ---Killing replica on trouble
   ---Killing replica on trouble
   ---Execing ssh trouble killall -9 java >/dev/null 2>/dev/null 
   ---ExitValue: 1
   Info: Nothing to kill
   Exit value: 1
   ---Execing ssh trouble cd ~/app; ./startServer.sh -v -p server.props.etwilson -n 2 
   Floor2.out>Running server on trouble with PID: 30479
   ---try to find new replica
   ---not yet available
   Floor2.out>Info: Setting Verbose Mode
   Floor2.out>---Loading properties file: server.props.etwilson
   Floor2.out>
   Floor2.out>***********************************
   Floor2.out>* Virtual Casino Blackjack Server *
   Floor2.out>***********************************
   Floor2.out>
   Floor2.out>---Created ORB instance.
   Floor2.out>---Using database username of gene
   Floor2.out>---Using database pasword of philISGR8!
   ---try to find new replica
   ---not yet available
   ---try to find new replica
   ---not yet available
   ---try to find new replica
   ---not yet available
   Floor2.out>---Created Floor db object.
   Floor2.out>---Created Player db object.
   Floor2.out>---Created Bank db object.
   Floor2.out>---Created Table db object.
   Floor2.out>---Looking for naming service on roborally:6505
   Floor2.out>---Resolved NameService.
   Floor2.out>---Created Bank object.
   Floor2.out>---Created Tables object.
   Floor2.out>---Added name Floor2.
   Floor2.out>Info: Server ready and running .....
   ---try to find new replica
   ---new replica is OK
   ---New replica started on trouble
   Floor1.out>Floor: Added player Gebne to database assigning player #:1
   
   
4. Now type the number of chips that you want to buy into the Player Application prompt. The Player Application should not indicate that there is anything wrong as it will transparently binds to the new replica.
   

Limitations

• The database should be cleared before the replication manager is started.
• The orb should be started fresh before the replication manager is started.
• The properties files must be correctly configured.
• The properties files must not be corrupted or deleted.
• The system does not provide security.
• No encryption of data.
• No logic checks to prevent false wins and artificially high win amounts.