18-649 Project 13
course webpage for due dates
Please submit all project-related correspondence to
- 11/23/15: Updated grade sheet xls/pdf to match bonus points in html
You have some additional time to complete this project. This
is so that you can juggle all the other project demands being placed on you by
other courses. We strongly advise you to set aside some time to do a thorough
review of your design documents before handing in your final project.
REMEMBER: this project phase has a heavy weighting in the
There are no design changes required for this project. You will simply
complete the testing that you started last week, follow up on any run-time
monitoring issues, and submit a final design portfolio.
1. Complete Testing of Your Elevator Design
You must run all tests on your latest design:
Note: You must pass all acceptance tests
in order to receive a grade in this course. There must also be zero failed
assertions (after startup transients) from your Runtime Monitor. That means
there can be no violations of the high level requirements.
- Pass all unit tests - all tests
must pass with 0 failed assertions
- Pass all integration tests - all
tests must pass with 0 failed assertions.
- All acceptance tests - run the following acceptance tests (all past tests
and three new ones). Be sure to use the "-b 200 -fs 5.0"
options to enable the finite bandwidth and new fast speed settings. Your
elevator must not reach 100% bandwidth utilization when running acceptance
In addition, please find 3 more test files that you must pass
In order to show that your elevator does indeed work, we ask you to
perform a demo for the TAs. The demo procedure is as follows.
You must pass all acceptance tests
in order to pass the Demo. There must also be zero failed
assertions (after startup transients) from The Course Staff Runtime Monitor.
All these tests are randomly generated by the TA test generation script.
In the event that your elevator does not pass the demo, the TA will provide you
with the test and stats file (containing the seed to replicate the error).
- A short test. (1-2 minutes runtime)
- A medium length test (8-12 minutes runtime)
- A long test (120-180 minutes runtime)
During the medium length test, your TAs will discuss with you some aspect of
your elevator. All group members must participate in this discussion in order
to receive full credit for the demo.
You will not need to stay for the long test once your discussion of the elevator
has concluded. Your TA will contact you regarding the results of the long test.
The demo procedure check list, followed by the TA's can be found here.
Teams will be limited to TWO practice runs.
Then there will be one final demo day that will be held the day before
the project is due. After two practice runs, if the third run is
unsuccessful, penalties will accrue for subsequent runs. There is also a late
penalty that applies after the demo deadline. Your penalty
for the project is the maximum of these two penalties and the penalty for a
late portfolio handin.
Any group member not
present for the demo will continue to accrue late penalties until the demo.
Demos that they did not attend still count towards their total attempts.
- i.e. total possible score = 100 * (.9)^( MAX(demos_past_3,
Most teams should be able to succeed with one or two practice runs.
The point is that TAs are not there to debug your project. You should honestly
believe you will pass before asking for a practice run. It is a good idea to
generate a large number of randomized acceptance tests for your elevator before
attempting even the first practice run.
Final demos are scheduled for the day before final project hand-in. All group
members must be present for the demo. TAs will only grade one hand-in per team.
If you tell them "grade this one" for an early demo, you forfeit any
right to any further hand-in attempts. (These rules are in place to prevent
abusive demos and hand-ins. If you have a special situation please let us know.)
Submit a Complete and Consistent Design Portfolio
For this final project, you must submit a complete and consistent design
portfolio. This includes (but is not limited to)
The above list is only a partial list to jog your memory. If you are in
doubt about your design portfolio, you should review the guidelines provided by
past project writeups. You should also take a look at the detailed
grading criteria for this project.
- Properly formatted portfolio:
re-read the guidelines on the
Portfolio Layout page.
Review the change log on that page to make sure you have included all portfolio
revisions. Make sure files are in the proper format, have proper headers,
all hyperlinks and inline images are working, etc.
- Complete, compiling design code:
before you submit your code,
review the procedure on
how the TAs will compile your code submission. It would be a good idea
for you to try this yourself and make sure your code compiles properly.
Be sure you include the code for your runtime monitor from part 1 above.
- Complete set of design artifacts:
the portfolio must contain all the required artifacts (sequence
diagrams, requirements, etc), including peer review logs where
- Complete end-to-end traceability:
Make sure that all of the traceability artifacts in your design
are complete and up-to-date. This is a major factor in determining the
"complete and consistent" criteria.
- Complete test logs: the input
and output files for each test must be included in the correct location in your
portfolio and properly linked into the appropriate test logs for Unit,
Integration, and Acceptance testing. You should make sure the result of
every test is fully documented, and that the Unit and Integration Test Summary
files are updated to include all your tests. You also need to make sure
the High Level Requirements verfication is up to date.
- Complete issue log: Your
issue log should reflect all the changes that occurred in your design over the
semester. You should have no outstanding issues for the final project
- All previous grading issues
addressed: review your feedback from previous projects and make
sure you have addressed each point.
3. Extra Credit
Note: To be
eligible for any extra credit, your elevator must be passing all unit,
integration, and acceptance tests listed above and it must be submitted on
time. Projects that are submitted late or that are not passing all tests
will not be considered for any bonus listed here.
3.1 Performance Competition
We will use the two performance metrics specified in the requirements
document and output by the simulator in the .stats file (lower is better for
performance; higher is better for satisfaction):
Performance=4 * (average passenger delivery time) + (maximum passenger delivery
Satisfaction Performance=100 - (various satisfaction metrics * x number of
times violated); similar to delivery performance, the sum is 4 times average
An unspecified number of workloads (.pass files) will be used in evaluating
total elevator performance. Some will be acceptance tests you have already
seen, and some will be new. NOTE: If your elevator crashes
during any test or fails to deliver all passengers in any
test, you will be disqualified from the competition. Groups will be ranked
(1=best, 10=worst) for each workload for each performance metric. Your overall
group score will be the sum of your ranks for all workloads. Of the eligible
groups, the one with the lowest sum will be the winner and all members thereof
will be awarded 1 percentage point on their final course grade.
Ties will be broken based on best
Satisfaction Performance score.
We will publish the performance scores on blackboard after all the
projects have been submitted. We will not publish any performance
statistics from other groups ahead of time.
You can take a look at previous years' performance results here:
Previous Performance Scores. Keep in mind, these do
not necessarily reflect the performance that groups will have for your
3.3 Fault Tolerance
Implement fault tolerance in the case of dropped messages.
Modify your elevator so it delivers all passengers with dropped messages
enabled and to work correctly with the new drive speed. By now you should know
the design process you should use.
To earn extra credit your elevator must work correctly when 25-50% of the
messages are dropped. No two messages of the same type are dropped in a row.
For Example: A drop rate of 100% will drop every other packet
Your elevator must continue to meet all of the high-level requirements we
have provided you both in the presence and absence of faults. Although your
performance when faults occur is relative, you are encouraged to deliver
passengers as quickly as possible.
To turn in your fault tolerant design, you should place a seperate
elevatorcontrol folder in the fault_tolerance section of your portfolio. This
copy of your code will only be used to check for fault tolerance, and not for
the performance testing
You can inject your own message drop faults by using a .fault file like
In order to run your acceptance test with this fault file use the command:
java simulator.framework.Elevator -pf basicpass.pass -ff basicfault.fault
-ff allows definition of a fault file
adding the -dropv flag causes verbose output of dropped messages(useful for
3.2 Complete Design Package
By the end of the semester, you will have produced a hefty design package.
Producing a complete design package of this size is a commendable feat, and we
will reward those who have accomplished it. If we judge your portfolio to be
complete and consistent, then all members of your team will receive the bonus
Note: only one group can win the performance
competition, but every group can win the portfolio bonus! The message is: GOOD
PROCESS IS OFTEN A MORE IMPORTANT GOAL THAN GOOD PERFORMANCE!
Handing In Results
- Each team shall submit exactly one copy of the assignment.
If you implement fault tolerance submit two portfolios : one with fault
tolerance and one without.
We will only grade the portfolio without fault tolerance for completeness.
- Follow the handin instructions detailed in the Project FAQ to submit your
portfolio into the afs handin directory (
- Be sure you follow the required format for
the directory structure of the portfolio and its location in the handin
- Be sure to follow ALL the portfolio
guidelines detailed in the Portfolio Layout page.
- Any submission that contains files with modification dates after the
project deadline will be considered late and subject to a grade deduction
policy page for more information).
Here's the minimum requirement spread sheet.
Though there are no constraints on
how you assign work for this project, you must still fill out the minimum
requirements sheet with your hours since project 12. The project will be late
until this sheet is handed in. We advise that you divide the work
as evenly as possible, but we are more concerned with you handing in a
consistent design portfolio and working code.
This project counts as one team grade. A
detailed grading rubric is available here. (PDF)
The rubric contains the detailed end-to-end
traceability checks. You should use this as a checklist for your final
handin. Remember that this project is worth considerably more than
This project is worth 175 points + up to 3.5 points bonus on your
- 30 points for complete unit and
- 30 points for complete acceptance
testing - This is our customer acceptance test. Demonstrate that your design
works! Run the specified acceptance tests to make sure it delivers all
passengers, meets high level requirements, and meets bandwidth constraints.
- 15 points for runtime
- 90 points for a complete and
consistent design portfolio with end-to-end traceability.
- 5 points For a list of which points corresponding to
which tasks were primarily completed by each team member (every point must be
assigned to a specific team member). Also for Improvements Log entry for
- 5 points for Improvements Log entry for "Overall
Project Comments". Considering the following: Now that you are at
the end of the semester, what did you think of this entire sequence of
projects? No doubt you have at least one suggestion for improvement. But please
also tell us the parts you liked the most so we know to keep them for next
year! Please consider these questions and add at least one comment to the
Improvements Log under the heading "Overall Project Comments".
- 1 point added to your final course grade for winning the
- 2 points added to your final course grade for a
complete design package. To receive this bonus, you must score at least
3.7/4 on the end-to-end traceability portion of the grading criteria.
Note 1: It is important that you submit a complete and consistent
end-to-end design package. In addition to the points awarded above for
traceability and meeting portfolio requirements, TAs can deduct points from the
project for failing to follow turn-in directions or having an end-to-end
design package with blatant gaps.
- 0.5 points added to your final course grade for
implementing fault tolerance in the case of dropped messages.
Note 2: Remember that a working final project is required to complete the
course. It is especially important that you "make it work"
for this final project! This means that your elevator must be
able to pass all acceptance tests (as outlined in the grading criteria above)
before you will receive a grade in this course.
Back to course home page