Designing an Efficient and Extensible Robustness Benchmark of a Real-Time Operating System

Zhou, Zheng-Mao; Zhu, Zhou-Rong; Cai, Ming

doi:10.1515/cait-2015-0008

Abstract

An important step in the development of a Real-Time Operating System (RTOS) is the validation of its tolerance properties. An abnormal input (fault injection) has become one of the most efficient ways to test the software robustness. In this paper we have designed a comprehensive robustness benchmark to evaluate the current popular RTOSs by faults injection. Firstly, we provide a set of uniform application program interfaces to ensure that the benchmark can be easily ported to a new RTOS. Then a package testing method has been used to improve the testing efficiency. Finally, a comprehensive robustness evaluation model is provided for the quantitative evaluation of RTOS robustness. Three popular RTOSs (Ucos2.62, Vxworks5.4 and Rtems4.10) have been evaluated with the help of our benchmark and we have found that Rtems performs best in robust evaluation, while Vxworks performs worst.

References

1. Zhao, Z. Z., J. H. Jiang. Research of Robustness Testing Methods for Operation Systems. - Computer Engineering and Applications, Vol. 43, 2007, No 7, pp. 93-97.
Search in Google Scholar
2. IEEE Std 610.12-1990 IEEE Standard Glossary of Software Engineering Terminology, 1990.
Search in Google Scholar
3. http://www.windriver.com/products/platforms/safety_critical_arinc_653/
Search in Google Scholar
4. Kropp, N. P., P. J. Koopman, D. P. Siewiorek. Automated Robustness Testing of off-theshelf Software Components. - In: Proc. of FTCS’98: Fault Tolerant Computing Symposium, Munich, Germany, 23-25 June 1998. IEEE Computer Society Press: Los Alamitos, CA, 1998, pp. 230-239.
Search in Google Scholar
5. Gehani, N. H. Exceptional C or C with Exceptions. - Software: Practices and Experience, Vol. 22, 1992, No 10, pp. 827-848.10.1002/spe.4380221003
Search in Google Scholar
6. Carrette, G. Crashme: Random Input Testing. http://people.delphiforums.com/gic/crashme.html
Search in Google Scholar
7. Miller, B. P., D.Koski, C. P. Lee, V. Maganty, R. Murthy, A. Natarajan, J. Steidl. Fuzz Revisited: A Re-Examination of the Reliability of UNIX Utilities and Services. Computer Science Technical Report 1286, Univ. of Wisconsin-Madsion, 1988.
Search in Google Scholar
8. Ghosh, A. K., M. Schmid, V. Shah. Testing the Robustness of Windows NT Software. - In: Proc. of Ninth International Symposium on Software Reliability Engineering (ISSRE’98), Los Alamitos, CA, November 1998, pp. 231-235.
Search in Google Scholar
9. Barton, J., E. W. Czeck, Z. Z. Segall. Fault Injection Experiments Using FIAT. - IEEE Trans. on Computers, Vol. 39, 1990, No 4, pp. 575-582.10.1109/12.54853
Search in Google Scholar
10. Kao, W., R. Lyer, D. Tang. FINE: A Fault Injection and Monitoring Environment for Tracing the UNIX System Behavior Under Faults. - IEEE Trans. on Software Engineering, Vol. 19, 1993, No 11, pp. 1105-1118.10.1109/32.256857
Search in Google Scholar
11. Carnegie Mellon University. Ballista Project. http://www.cs.cmu.edu/afs/cs/project/edrc-ballista/www/index.html
Search in Google Scholar
12. Pan, J., P. Koopman, D. Siewiorek. A Dimensionality Model Approach to Testing and Improving Software Robustness. - In: Proc. of 1999 IEEE AUTOTESTCON, September 1999, IEEE, Piscataway, NJ, USA, 1999, pp. 493-501.
Search in Google Scholar
13. Zhou, Z. M., Y. Zhou, M. Cai. An Approach to Evaluate the Robustness of Real Time Operating System. - In: Proc. of 2nd International Conference on Computer Science and Network Technology, Changchun, China, December 2012, pp. 306-310.10.1109/ICCSNT.2012.6525944
Search in Google Scholar
14. Zhou, Z. M., Y. Zhou, M. Cai, L. Sun. A Workload Model Based Approach to Evaluate the Robustness of Real-Time Operating System. - In: Proc. of 2013 IEEE International Conference on High Performance Computing and Communications & 2013 IEEE International Conference on Embedded and Ubiquitous Computing, Zhangjiajie, China, November 2013, pp. 2027-2033. 10.1109/HPCC.and.EUC.2013.292
Search in Google Scholar

Designing an Efficient and Extensible Robustness Benchmark of a Real-Time Operating System

Abstract

Paradigm

My account