Fault Tolerance Testing Process


OBJECTIVE

This test script illustrates the scenario used to test graceful switch over of the client to the backup server when the primary application server is shut down forcefully. Success in this test is assumed to be a certification of fault tolerance for the AIRLINE RESERVATION SYSTEM (ARS).
 

APPROACH

A primary (128.237.11.178 – Vijay’s machine) and back up server (128.237.12.217 – Paulo’s server) were made operational. The primary server had the application, the replication manager – GOD and the credit card authorization software running. The back up server had the application server running and was configured to establish connection to the primary where the GOD and CC servers were hosted from. The plan entails two scenarios discussed below.
 

NOTE: Further testing will include Matt’s and Manny’s machines as well to extend the scope of this test

 
SCENARIO 1
  • Kill the primary during the buy tickets operation
  • Retrieve the transaction from the application on the new server and view the expected state
    •  
    RESULT
    The test succeeded.
     
    SCENARIO 2
  • Restart the server process (scenario 1 primary) that was killed in scenario one so as to make it the back up server for scenario 2
  • Start a new transaction of purchasing tickets
  • Kill the primary application server (Scenario 1 backup)
  • Gracefully failover to the back up (Scenario1 primary)
  • Retrieve the transaction using the reservation number provided during the make reservation process

    •  
    RESULT
    03/14/03 The test failed and code is presently under review (3/14/03 12.00 noon) for remedial action. This page will be updated as soon as the next test results are available.
    03/15/03 The code has been since modified and tested. This page will be updated before 4.00 pm today with the test scenario and the results.
    03/15/03 The web page has been since updated with the steps for scenario 2.
     

    The table below illustrates the sequence of actions performed to establish the faulat tolerance capabilities of the Airline Reservation System
     
    Action
    Expected Result
    Actual Result
    Comments
    Scenario1
    1. Start GOD and CC servers on    128.237.11.178 
    See the DOS windows confirming the startup of GOD and CC
    See the DOS windows confirming the startup of GOD and CC
    Step was  successful
    2. Start the application  server on 128.237.11.178 
    This will be the secondary server; as it comes up it will recognize the identity and existence of GOD and CC on the local host as well as the primary server
    		 Primary recognizes the GOD and CC servers A condition that may need resolution: We have to reset the “stateful firewall” property to release the machine for accessibility by other clients and servers if the property is turned; 
    3. Start the application server on 128.237.12.217. This will be the secondary server; as it comes up it will recognize the identity and existence of GOD and CC on the local host as well as the primary server Secondary recognizes GOD, CC and Primary  Step was successful
    4. Start client using ClientArs.bat on 128.237.11.178  Client will be able to look for the host machine that hosts God and CC servers; in the case of this test it is the localhost (128.237.11.178)  Client will start up to allow the user to use the ARS application being hosted by the primary Step was successful
    5. Search FlightPittsburgh to Bombay,Departing 05/15/2003, Roundtrip,Returning 06/15/2003Click “Search” Retrieval of 5 flights for each sector of travel Retrieves 5 flights for each sector of the travel If it does not retrieve, click search again until it does an entire retrieval; the query may be taking a little longer to retrieve and partial results may be delivered.

    Step was successful
    6. Select options 1037 & 1013 respectively;Then click “Make reservation” button Will bring up a login screen Login screen displayed Step was successful
    7. Enter username and click “ok” Will bring up the “ARS Reservation Screen” the “ARS Reservation Screen” retrieved Step was successful
    8. Enter the passenger name, cc holder name and card details and other required detailsAnd click “Make reservation” Expected to prompt if the tickets should be bought or the screen may be closed; also expected to provide the reservation number Provides the reservation number and the option to buy the tickets
    9. Select close You will be taken to the main menu Returned to the main menu Step was successful
    10. Select Reservations tab and select read reservations option on menu list It will bring up a screen that can accept the reservation number provided to you in our case #11; enter the number and expect to see the itinerary and options to cancel the reservation or buy the tickets Itinerary displayed with the options to buy or “cancel” the “reservation” Step was successful
    11. Select the “Buy tickets” option Will be directed to another scree called “ARS – Ticket Purchase” Step was successful
    12. Verify the payment information and make changes if desired and click  “Ok” Tickets were purchased message will appear “Tickets were purchased” message appears Step was successful
    13. Click CTRL C on your app server The app server will shut down after you choose Y Apps server is shut down on the local host Step was successful
    14. Select the Reservations tab in the ARS main screen;Select the read reservation option  Expect to be prompted for the reservation number; enter 11 and you will find the itinerary; Here is where the FAULT TOLERANCE test succeeds. The client is now on the back up server 128.237.12.217 
  • Itinerary is displayed.NOTE:No new login is needed
  • GOD’s grace the system’s awareness of 128.237.12.217 as the back up server helps the user to continue his operations with a graceful failover;
  • User has no knowledge of the failover to the backup;
  • All services are available to the client as if everything were normal; 
  • No Loss of data has occurred
    		•  Step was successful

    FT success with a graceful change over and no loss of data; the transactions are committed after the credit card component is reached and the purchase is authorized
    SCENARIO 2
    1. Start app server on 128.237.11.178 It will get the primary which is 128.237.12.217 The primary and backup from the first test scenarion will be reversed. Step was successful
    2. Repeat steps 4 to 12 from test scenarion 1 
    3. Execute CTRL C on app server screen running on primary (128.237.12.217)
    4. Select the Reservations tab in the ARS main screen;Select the read reservation option  The client is expected to switch over to 128.237.11.178 and be prompted for the resrevation number Screen appears for inquiry Step was successful
    5. Enter the reservation number  The screen will provide the itinerary Fault tolerance capability is demonstrated thorugh steps 5 and 6