18-649 Project 12
Check the course webpage
for due dates
Please submit all project-related correspondence to
- 11/14/2014 Added Changelog
- 11/14/2014 Must justify acceptance tests that do not pass
In this project, you will finish your runtime monitoring, complete unit and integration testing, and begin acceptance testing.
1. Acceptance Stress Testing
1.1 Run Some More Acceptance Tests
We've provided a few more acceptance tests for you. You should look at
these tests as good examples of tests that your generator (in part 1.2)
You should run these tests and update your Acceptance Test Log. This
includes linking the test files, and filling a row in the table with the
test statistics. You must also update the test results for the previous
project acceptance tests with the results from your new elevator
Any tests that you do not pas should have a justification in the notes
section of your Acceptance Test
Log that describes the bug causing this behavior. This
justification should include the number of the bug in the Issue Log
(create a new entry in the log if you need to).
1.2 Write an Acceptance Generator
For this project you will need to write an acceptance test generator.
Your acceptance test generator:
Since your test generator will use some form of random number generator,
it's okay if not every generated test satisfies the conditions, however
your generator must be able to create tests that satisfy this criteria.
This means that if your test generator only sends passengers from 1
FRONT to 2 BACK, this is not an acceptable generator.
- Must be a command line tool
- Must be called AcceptTestGenerator
- Must take at least 1 argument: the number of passengers in the test
- May take more arguments if you decide to do so
- If called with no arguments, must print usage details, along with an example command line call
- Should make tests that roughly follow the criteria given in project 8
- Every landing should be used at least once as a source for a passenger,
and at least once as a destination. Note that some floors have more than
(e.g. 1 FRONT and 1 BACK), so both the 1 FRONT and 1 BACK landings
must be used at least once in the test.
- Landings that do not exist (e.g. 2 FRONT) must not appear in the
test. (This will cause the passenger injector to fail and the
simulator to crash)
- You do not need to add passengers in the car at the beginning, as in proj12aceptance2.pass. If you want to add this to your tests it is acceptable.
You cannot add passengers to the car at a time after 0s. This will cause the elevator to throw a runtime error
Your acceptance test generator can be written in whatever programming language you feel like, however, it must run on the ece machines. Additonally, if the generator is in a compiled language, you must include the source, and an executable that will run on the ece machines. For this reason we highly suggest a scripting language, such as Bash, Perl, or Python.
Here is a template of a bash script that will generate some tests that do not meet the criterion.
Include this test generator in your /acceptance_test folder.
1.3 Create and Run 40 Tests
Using your test generator, create and run 40 tests. Link them in your
Acceptance Test Log in a separate table below the first. Use
this table for stress tests, and the original table for the acceptance
tests that have been provided (e.g. proj7acceptance1.pass) or that you
have created yourself.
2. Write a Monitor
Write runtime requirements monitors for RT-8.1, RT-8.2, and RT-8.3, and add
them to the
RuntimeRequirementsMonitor java file. Information about runtime monitoring can
be found in Project 8. If there is a group
member who has not yet written a monitor, they must write one for this project. If you
are in a group of three, and everyone has written a monitor, then this
task can be assigned to any group members.
We highly advise that you run your monitors on your mid-semester elevator
(Sabbath mode), to make sure your monitors do in fact catch violations of the
high level requirements. If your project 7 elevator does meet any of the high
level requirements, we advise that you introduce behaviors that cause
violations, so as to ensure your monitor is not generating false negatives.
For the final demos, you will be required to pass your acceptance tests with
no warnings generated by the course monitor. The course monitor is quite
slack, but you should still make sure your monitors are actually detecting
faults, to give yourself plenty of time to debug your code prior to demos.
3. More Testing
3.1 Complete Unit Testing
Run all unit tests. For this project, all unit tests must pass
(0 failed assertions).
You should have already passed all of your unit test for last project,
so this should be a freebie.
3.2 Complete Integration Tests
First, double check your sequence diagrams to requirements
traceability to make sure it is up-to-date. Make sure your
sequence diagrams are consistent with the design changes introduced
in project 8 (and any updates you made in project 9 and 10), as well
as consistent with your updated message dictionary from
If needed, review the Integration
section of the Testing Requirements document.
Once your sequence diagrams are fully up-to-date, you will finish updating
your integration tests, completing the remaining integration tests that
you didn't update last project. If you have been keeping up with your
issue log, it should be easy for you to use it to identify which
integration tests need to be created or updated. You should
also make sure that the the traceability comments in all of your
integration tests are complete and up-to-date.
Update your Integration Test
Summary File with the new integration tests. Use the
verification script to ensure that the summary file is correct and
When you have completed this part of the project, you should have at
least one integration test for every sequence diagram. Have
someone other than the test author complete a peer review of the
updated tests and update the peer review section of the Peer Review Log . Remember
any peer reviews that result in unfixed issues, should be added to
the Issue Log.
Execute your integration tests and record the results in the Integration Test Log.
Note that you do not need to specify the "-b 200" flag for
For this project, you must pass (0 failed assertions) *all* integration tests, including those you wrote for this project.
4. Peer review
For this project, you should peer review all of your updated integration
tests and your acceptance test generator. Additionally, if you have not
been peer-reviewing your monitor designs, you should add peer reviews
for them as well.
Team Design Portfolio
The portfolio you submit should contain the most up-to-date
design package for your elevator system organized and formatted
according to the portfolio
guidelines. You are going to
update your portfolio every week, so be sure to keep an up to date
Ensure your design portfolio is complete and consistent.
The following is a partial list of the characteristics your
portfolio should exhibit:
- Changes requested by the TAs in previous projects have been
- All required documents are complete and up-to-date to the
extent required by the projects (you do not need to update files
or links related to future projects).
- All documents include group # and member names at the top of
the document. (This includes code, where this information
should appear in the header field)
- Individual documents have a uniform appearance (i.e., don't
look like they were written by 4 individual people and then
- The issue log is up to date and detailed enough to track
changes to the project.
Handing In Results
- Each team shall submit exactly one copy of the assignment.
- Follow the handin instructions detailed in the Project FAQ to
submit your portfolio into the afs handin directory ( /afs/ece/class/ece649/Public/handin/project10/group#/ontime/).
- Be sure you follow the
required format for the directory structure of the portfolio
and its location in the handin directories.
- Be sure to follow ALL
the portfolio guidelines detailed in the Portfolio Layout
- Any submission that contains files with modification dates
after the project deadline will be considered late and subject
to a grade deduction (see course
page for more information).
The Minimum Requirements spreadsheet is located
This project counts as one team grade. Points are assigned as
follows. A detailed grading rubric is available
This project assignment is worth 130 points:
Each team member must satisfy the minimum stated per-member requirements
(e.g., one object for each activity). Team members who omit any required
per-member activity will be penalized as described on the
course admin page.
- 25 points for a
test generator that generates tests that meet the criteria.
It must be able to run on the lab machines.
- 10 points for an up
to date Acceptance Test Log.
- 10 points for
successfully completing all unit tests (all tests must pass)
- 35 points for writing
and successfully completing all integration tests (all tests must
- 10 points for
the RT-8 requirement monitors
- 5 points for the peer
review of the acceptance test generator.
- 15 points for the peer
reviews of updated or newly created integration tests. You should
have at least 5 this project. Each team
member must complete at least one peer review.
- 15 points for the peer
reviews of runtime monitors. It is acceptable for these to have been created in previous projects. Each team
member must complete at least one peer review.
- 5 points for an entry in
the Improvements Log that
tells us what can be improved about this project. If you
encountered any minor bugs that we haven't already addressed,
please mention them so we can fix them. If you have no
suggestions, say so in your entry for this project.
Back to course home page