TEST NUMBER 4

Functionality:  Fault Tolerance

Use case:  Client failover 
	- server 1, 2, & 3 are up 
	- client connects randomly to one of the servers 
	- that server goes down 
	- client connects to another server

Test distribution:  http://www.ece.cmu.edu/~ece841/team1/fault_tolerance/build/release

Test sequence:  
	1) setup as defined in the Setup, Installation & Run Instructions (SIRI)
	2) install servers as defined in the SIRI
	3) start replication manager as defined in the SIRI
	4) wait for all of the servers to come up
	5) start the stock ticker as defined in the SIRI
	6) start the client as defined in the SIRI
		- enter a user name
		- will see an INFO message saying that the client has 
		  connected to server X  (where X = 1, 2, or 3)
	7) determine the name of the server you are connected to
		- bring up a terminal (ex: >xterm&)
		- >cd TEMPDIR/pqclient
	        - >more serverlist.props
		- locate the name of the server associated with X
	8) kill server X
		- go to the Replication Manager
		- select option 4
		- a list will be displayed of the server names
		- enter the number next to the server name to kill it
	9) watch the failover
		- an exception will be caught on the client side (StockServerException, ServerNotReadyException or RemoteException)
		- a message will be displayed saying that a server exception has been detected & the time of detection
		- a message will be displayed indicating the server number the client is trying to connect to
		- a message will be displayed once the client has connected & time of connection		
		- the client application should complete the command started
		
Configuration/deployment issues:  
	- assumes that at least 2 servers are running

Known problems:  
	- If any unknown exceptions are caught, the client application will terminate.
	  (This will be modified later so that the client will restart itself.)
	- Client failover has been tested for the case that a failover is needed during a 
	  request for stock updates or during connection.  It has not been tested as well 
	  when failovers are needed during a login, when setting a user profile, or when
	  subscribing.
	- The difference between the time of detection and the time of connection is always 
	  less than 1 minute (as specified in our requirements).  We are currently not 
	  measuring the time between when the server actually is killed and the reconnection
	  occurs.