18-749 Evaluation Report
Team 2: The House
Chief Experimenter: Joohoon Lee
Members: Paul Cheong, Jun Han, Suk Chan Kang, Mohammad Ahmad
Experimental Setup
- Each client run on separate machine | |||||||
- Two methods chosen for the experiment (two-way invocation) are highlighted in the Table 1. | |||||||
- The middle tier have 2 replicas | |||||||
- Our implementation has two replicas and fail-over, but the assumption of this experiment was that there is no fault-tolerance, thus all of our measurements were taken without any crashes on the server. | |||||||
- The following parameters are used (Total of 48 combinations) | |||||||
|
|||||||
- In order to support variable size of reply message, before the client starts invocating 10000 times for the experiment, the client lets the server know the size of the reply for the given run. The server then allocates a dummy data block of the requested size on the entity bean (thus it becomes stateful), and gives that reply when the client invokes the actual method for testing. We used String object to allocate the request data, assuming each character of the String would use 2-byte (JAVA stores the string as 2-byte unicode). Although, this may not be true, since we are interested in size of ¡°useful data¡±, this is acceptable assumption for this evaluation. | |||||||
- Each test methods are called 5000 times each to make 10000 requests per experiment | |||||||
- JAVA does not support a fine-grain measurement method, so we decided to use native method using JNI. The probe calls the native implementation of the timer which gives u-sec granularity. | |||||||
- Total of 7 probes were used in this experiments. The descriptions of each probe are in Table 2. | |||||||
- In JAVA, the size of objects cannot be easily determined, so we used simple approximations. Primitive data types are assumed to be 32-bit, and variable size of objects are represented using sizeof(object). The unit of size is ¡®byte¡¯. When getPlayerName is called for experiment, the id is always the same to fix the reply size |
METHOD | ONE_WAY |
DB_ACS |
SZ_REQ |
SZ_REPLY |
void savePlayer(int uid, int oid, String id, String name, Integer balance ) |
YES |
YES |
12 + sizeof(id) |
0 |
void createTable(int uid, int oid, String name ) |
YES |
YES |
8 + sizeof(name) |
0 |
void dealPlayer(int uid, int oid, String playerID) |
YES |
YES |
8 + sizeof(playerID) |
0 |
Card lastDealtCard(String playerID) |
NO |
YES |
sizeof(playerID) |
12 + sizeof(type) |
void dealDealer(int uid, int oid, String tableName) |
YES |
YES |
8 + sizeof(tableName) |
0 |
Card lastDealtDealerCard(String tableName) |
NO |
YES |
sizeof(tableName) |
12 + sizeof(type) |
void joinTable(int uid, int oid, String playerID, String tableName) |
YES |
YES |
8 + sizeof(playerID) + sizeof(tableName) |
0 |
void placeBet(int uid, int oid, String playerID, int bet) |
YES |
YES |
12 + sizeof(playerID) |
0 |
void startGame(int uid, int oid, String playerID, String tableName) |
YES |
YES |
8 + sizeof(playerID) + sizeof(tableName) |
0 |
void adjustBalance(int uid, int oid, String playerID, int adjust) |
YES |
YES |
12 + sizeof(playerID) |
0 |
int readBalance(String playerID) |
NO |
YES |
sizeof(playerID) |
4 |
String getPlayerName(String id) |
NO |
YES |
sizeof(id) |
Variable |
String getTableName(Integer uid) |
NO |
YES |
4 |
Variable |
Probe | File Name | Data |
Purpose |
1 |
DATA749_app_out_cli_${STY}_2srv_${C} cli_${IRT}us_${BYT}req_${HOST}_team${N}.txt |
Time in u sec |
Record the time of client invocation (request going out) |
2 |
DATA749_app_in_cli${STY}_2srv${C} cli_${IRT}us_${BYT}req_${HOST}_team${N}.txt |
Time in u sec |
Record the time of receiving the reply |
3 |
DATA749_app_msg_cli${STY}_2srv${C} cli_${IRT}us_${BYT}req_${HOST}_team${N}.txt |
Name of each invocation |
Keep track of the name of the invocating function to match later |
4 |
DATA749_app_in_srv${STY}_2srv${C} cli_${IRT}us_${BYT}req_${HOST}_team${N}.txt |
Time in u sec when the request is received |
Record the time of request received |
5 |
DATA749_app_out_srv${STY}_2srv${C} cli_${IRT}us_${BYT}req_${HOST}_team${N}.txt |
Time in u sec when each reply is served |
Record the time of request completion |
6 |
DATA749_app_msg_srv${STY}_2srv${C} cli_${IRT}us_${BYT}req_${HOST}_team${N}.txt |
Name of each invocation |
Keep track of the invocation from the server |
7 |
DATA749_app_source_srv${STY}_2srv${C} cli_${IRT}us_${BYT}req_${HOST}_team${N}.txt |
Name of client invocation |
Keep track of which client sent the each request |
Experiment 1 - # of Client = 1, Size of Reply = original, Request Time = 0ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 2 - # of Client = 1, Size of Reply = original, Request Time = 20ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 3 - # of Client = 1, Size of Reply = original, Request Time = 40ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 4 - # of Client = 1, Size of Reply = 256, Request Time = 0ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 5 - # of Client = 1, Size of Reply = 256, Request Time = 20ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 6 - # of Client = 1, Size of Reply = 256, Request Time = 40ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 7 - # of Client = 1, Size of Reply = 512, Request Time = 0ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 8 - # of Client = 1, Size of Reply = 512, Request Time = 20ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 9 - # of Client = 1, Size of Reply = 512, Request Time = 40ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 10 - # of Client = 1, Size of Reply = 1024, Request Time = 0ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 11 - # of Client = 1, Size of Reply = 1024, Request Time = 20ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 12 - # of Client = 1, Size of Reply = 1024, Request Time = 40ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 13 - # of Client = 4, Size of Reply = original, Request Time = 0ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 14 - # of Client = 4, Size of Reply = original, Request Time = 20ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 15 - # of Client = 4, Size of Reply = original, Request Time = 40ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 16 - # of Client = 4, Size of Reply = 256, Request Time = 0ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 17 - # of Client = 4, Size of Reply = 256, Request Time = 20ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 18 - # of Client = 4, Size of Reply = 256, Request Time = 40ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 19 - # of Client = 4, Size of Reply = 512, Request Time = 0ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 20 - # of Client = 4, Size of Reply = 512, Request Time = 20ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 21 - # of Client = 4, Size of Reply = 512, Request Time = 40ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 22 - # of Client = 4, Size of Reply = 1024, Request Time = 0ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 23 - # of Client = 4, Size of Reply = 1024, Request Time = 20ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 24 - # of Client = 4, Size of Reply = 1024, Request Time = 40ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 25 - # of Client = 7, Size of Reply = original, Request Time = 0ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 26 - # of Client = 7, Size of Reply = original, Request Time = 20ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 27 - # of Client = 7, Size of Reply = original, Request Time = 40ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 28 - # of Client = 7, Size of Reply = 256, Request Time = 0ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 29 - # of Client = 7, Size of Reply = 256, Request Time = 20ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 30 - # of Client = 7, Size of Reply = 256, Request Time = 40ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 31 - # of Client = 7, Size of Reply = 512, Request Time = 0ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 32 - # of Client = 7, Size of Reply = 512, Request Time = 20ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 33 - # of Client = 7, Size of Reply = 512, Request Time = 40ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 34 - # of Client = 7, Size of Reply = 1024, Request Time = 0ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 35 - # of Client = 7, Size of Reply = 1024, Request Time = 20ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 36 - # of Client = 7, Size of Reply = 1024, Request Time = 40ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 37 - # of Client = 10, Size of Reply = original, Request Time = 0ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 38 - # of Client = 10, Size of Reply = original, Request Time = 20ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 39 - # of Client = 10, Size of Reply = original, Request Time = 40ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 40 - # of Client = 10, Size of Reply = 256, Request Time = 0ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 41 - # of Client = 10, Size of Reply = 256, Request Time = 20ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 42 - # of Client = 10, Size of Reply = 256, Request Time = 40ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 43 - # of Client = 10, Size of Reply = 512, Request Time = 0ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 44 - # of Client = 10, Size of Reply = 512, Request Time = 20ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 45 - # of Client = 10, Size of Reply = 512, Request Time = 40ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 46 - # of Client = 10, Size of Reply = 1024, Request Time = 0ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 47 - # of Client = 10, Size of Reply = 1024, Request Time = 20ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 48 - # of Client = 10, Size of Reply = 1024, Request Time = 40ms | ||
Client Latency |
||
Server Latency | ||
Middle -ware Latency |
Experiment 1 |
Experiment 2 |
Experiment 3 |
Experiment 4 |
Experiment 5 |
Experiment 6 |
Experiment 7 |
Experiment 8 |
Experiment 9 |
Experiment 10 |
Experiment 11 |
Experiment 12 |
Experiment 13 |
Experiment 14 |
Experiment 15 |
Experiment 16 |
Experiment 17 |
Experiment 18 |
Experiment 19 |
Experiment 20 |
Experiment 21 |
Experiment 22 |
Experiment 23 |
Experiment 24 |
Experiment 25 |
Experiment 26 |
Experiment 27 |
Experiment 28 |
Experiment 29 |
Experiment 30 |
Experiment 31 |
Experiment 32 |
Experiment 33 |
Experiment 34 |
Experiment 35 |
Experiment 36 |
Experiment 37 |
Experiment 38 |
Experiment 39 |
Experiment 40 |
Experiment 41 |
Experiment 42 |
Experiment 43 |
Experiment 44 |
Experiment 45 |
Experiment 46 |
Experiment 47 |
Experiment 48 |
Number of Client = 1 | |
Number of Client = 4 | |
Number of Client = 7 | |
Number of Client = 10 | |
Challenges and Problems!
- The evaluation data has been collected using stratego as the server, and arkhamhorror, boggle, chess, clue, diplomacy, drlucky, girltalk, risk, roborally and settlers. However, the some of the machines experienced failure due to high contention, and we were forced to use mahjongg, go and othello to launch some of our clients when we were evaluation 10 client test (Experiment 38, 39, 41). (devilbunny rejected ssh connection, so it was eliminated from the list of machines we used. | |
- In general, there was a lot of variability in the measurement. We believe this is due to high contention in the network since a lot of groups were running the evaluations at the same time. Our results shows that the database is a major source of bottleneck. This makes sence because all of the teams are running the database on mahjongg, and one machine is overloaded with requests. | |
- One other variability comes from difference in workload in each cluster machines. We found that a machine with a common and well-known game name (e.g. chess, othello etc) have higher latency due to popularity | |
- When we were running tests with 10 clients, we have experienced a significant slow down in the latency. Our timer measurement had overflowed after long time, and produced a negative latency values as result. This is shown in the plots from Experiment 37~48. You can see that in some of the cases, the latency values are completely wrong (negative). We think this happened because of timer overflowed. However, towards the end of the experiment, some of the cluster machines were not available to use, and the database server got killed, so we could not run addtional tests to collect the correct data. | |
Interpretation of the Result
- We found 2 highest outliers in each experiment, and plotted the graph to show the break-down of the source of variability. In general, the database takes up the major portion of the client latency. However, we also found that this is not the case in all of the cases. In some of the cases, the middleware took a considerably longer than the database. In conclusion, the variabilty comes from a lot of sources, and it is hard to come up with deterministic way to predict the behavior of multi-machine network | |
- If you look at the Observable Trends, you can see that as the reply size becomes large, the latency grows. However, the inter-request time did not have much impact on the performance. We think this is because the way that JAVA handles the thread scheduling is not very accurate and does not garantee the amount of sleep time we requested. This trends can be seen in all of the cases with different number of clients. (1,4,7,10). As the number of client increases, the latency increases more rapidly. | |
- The Magical 1% graph confirms the data from the lecture. We had some outliers that made the interpretation of the data difficult, but after filtering out top 1% of the outliers, the data seems a lot more consistent and shows some correlations with varying parameters | |