December 17, 2004
The Linux Test Project (LTP) incorporated Carnegie Mellon's Ballista software robustness testing technology in its November release. An open source joint project from SGI, IBM, OSDL, Bull, and Wipro Technologies, the LTP creates test suites to validate the reliability, robustness, and stability of Linux. The DARPA-funded Ballista project at Carnegie Mellon was headed by Philip Koopman, Associate Professor of ECE and CS, working with ECE and ICES students and staff. Kobey DeVale, a staff programmer at ICES who worked on Ballista technology transfer, was responsible for creating the publicly released version of Ballista used by LTP.
According Nigel Hinds, a Software Engineer at the IBM T. J. Watson Research Center who serves as a research advisor to the IBM Linux Technology Center Test team, before adding Ballista to the test suite, the LTP mainly tested the Linux kernel for correct operation given valid system call input.
But the case of invalid system call input had not been systematically tested, so IBM looked to Ballista's capability to characterize these exceptions. Ballista testing can find ways to make operating systems crash and force software packages to suffer abnormal termination instead of gracefully returning error indications.
"Given the rapid and often violent code changes in open source projects such as the Linux kernel, systematic testing of failure semantics is as important as functional testing," Hinds explained.
"The Ballista test suite has added an element of formal error semantics testing to the LTP. Ballista portability made integration with LTP straightforward," he accounted. "And, the data-type driven test harness makes it easy to add new tests to Ballista."
Hinds noted that the LTP includes over 2500 regression tests and is used by developers, SUSE, and other Linux distributors to help improve kernel reliability.