A State-aware Approach for Robustness Testing of Embedded Real-Time Operating Systems

Document Type : Parallel and Distributed Processing-Deldari-Abrishami

Authors

• Raheleh Shahpasand ¹
• Samad Paydar ¹
• Yasser Sedaghat ¹
• Reza Ramezani ²

¹ Ferdowsi University of Mashhad

² University of Isfahan

10.22067/cke.v2i1.74607

Abstract

The Operating System (OS) is a major part of embedded software systems and its robustness has considerable influence on the robustness of the entire system. Thus, its robustness testing is critical for assessing the dependability of the system. In this paper, a state-aware approach is proposed to evaluate the robustness of components of embedded real-time OSs in the presence of different types of faulty inputs. This approach leads to identifying critical OS states, their criticality level, and the maximum and minimum level of the OS robustness. It also facilitates comparing the robustness level of OS’s components and helps the system developers to select the most appropriate fault tolerance techniques by considering the robustness level and timing limitations. The experimental results demonstrate the ability of the proposed approach in providing more information about the robustness vulnerabilities in the states of the system.

Keywords

• Robustness testing
• Embedded operating system
• Robustness level assessment
• Safety-critical systems
• Fault injection

---

References

[1] R. Ramezani, and Y. Sedaghat, "An overview of fault tolerance techniques for real-time operating systems". Proceedings of the 3rd International Conference on Computing and Knowledge Engineering, IEEE, Mashhad, Iran, October 31 - November 01, 2013, pp. 1-6.

[2] J. C. Knight, "Safety critical systems: Challenges and directions". Proceedings of the 24rd International Conference on Software Engineering, IEEE, Orlando, Florida, USA, May 19-25, 2002, pp. 547-550.

[3] E. A. Lee, "Embedded software". Advances in Computers, vol. 56, pp. 55-95, 2002.

[4] R. Natella, D. and H. S. Cotroneo, Madeira, "Assessing dependability with software fault injection: A survey", ACM Computing Surveys (CSUR), vol. 48, no. 3, pp. 44:1-44:55, 2016.

[5] S. M. A. Shah, D. Sundmark, B. Lindström, and S. F. Andler, Robustness testing of embedded software systems: An industrial interview study. IEEE Access, vol. 4, pp. 1859-1871, 2016.

[6] Z. Micskei, H. Madeira, A. Avritzer, I. Majzik, M. Vieira, and N. Antunes, "Robustness Testing Techniques and Tools," In Resilience Assessment and Evaluation of Computing Systems, Springer, 2012; 323-339.

[7] W. Torres-Pomales, "Software fault tolerance: A tutorial", Technical Report, NASA Langley Research Center, Hampton, VA United States, NASA-2000-tm210616, 2000.

[8] A. Shahrokni, and R. Feldt, "RobusTest: A framework for automated testing of software robustness", Proceedings of the 18th Asia Pacific Software Engineering Conference, IEEE, 2011, pp. 171-178.

[9] D. Cotroneo, and H. Madeira, "Introduction to software fault injection," In Innovative Technologies for Dependable OTS-based Critical Systems, Springer: Milan, 2013, pp. 1-15.

[10] D. Cotroneo, D. Di Leo, R. Natella, and R. Pietrantuono, "A case study on state-based robustness testing of an operating system for the avionic domain". Proceedings of the 30th International SAFECOMP Conference, Springer, Naples, Italy, September 19-22, 2011, pp. 213-227.

[11] D. Cotroneo, D. Di Leo, R. Natella, "SABRINE: State-based robustness testing of operating systems". Proceedings of the 28th International Conference on Automated Software Engineering, IEEE, 2013, pp. 125-135.

[12] B. P. Miller, L. Fredriksen, and B. So, "An empirical study of the reliability of UNIX utilities", Communications of the ACM, vol. 33, no. 12, pp. 32-44, 1990.

[13] B. P. Miller, D. Koski, C. Lee, V. Maganty, R. Murthy, A. Natarajan, and J. Steidl, "Fuzz revisited: A re-examination of the reliability of UNIX utilities and services", Technical Report, University of Wisconsin-Madison, Technical Report #1268, 1995, pp. 1-23.

[14] J. E. Forrester, and B. P. Miller, "An empirical study of the robustness of Windows NT applications using random testing". Proceedings of the 4th USENIX Windows System Symposium, Seattle, 2000, pp. 59-68.

[15] B. P. Miller, G. Cooksey, and F. Moore, "An empirical study of the robustness of macos applications using random testing". Proceedings of the 1st international workshop on Random testing, ACM, 2006, pp. 46-54.

[16] P. Koopman, and J. DeVale, "The exception handling effectiveness of POSIX operating systems", IEEE Transactions on Software Engineering; vol. 26, no. 9, pp. 837-848, 2000.

[17] P. Koopman, and J. DeVale, "Comparing the robustness of POSIX Operating Systems". Proceedings of the 29th Annual International Symposium on Fault-Tolerant Computing, IEEE, 1999, pp. 30-37.

[18] A. Johansson, et al., "On enhancing the robustness of commercial operating systems". Proceedings of the 1st International Service Availability Symposium (ISAS), Munich, Germany, May 13-14 , 2004, Springer, pp. 148-159.

[19] A. Albinet, J. Arlat, and J.-C. Fabre, "Characterization of the impact of faulty drivers on the robustness of the linux kernel". Proceedings of the in International Conference on Dependable Systems and Networks, IEEE, 2004, pp. 867-876.

[20] C. Sârbu, A. Johansson, N. Suri, and N. Nagappan, "Profiling the operational behavior of OS device drivers", Empirical Software Engineering, vol. 15, no. 4, 380-422, 2010.

[21] A. Johansson, N. Suri, and B. Murphy, "On the impact of injection triggers for OS robustness evaluation". Proceeding of the 18th IEEE International Symposium on Software Reliability, IEEE, 2007, pp. 127-136.

[22] S. Yang, B., Liu, S. Wang, and M. Lu, "Model-based robustness testing for avionics-embedded software". Chinese Journal of Aeronautics, vol. 26, no. 3, pp. 730-740, 2013.

[23] N. Coppik, O. Schwahn, S. Winter, and N. Suri, "TrEKer: Tracing error propagation in operating system kernels". Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, IEEE, 2017, pp. 377-387.

[24] R. Shahpasand, Y. Sedaghat, and S. Paydar, "Improving the stateful robustness testing of embedded real-time operating systems". Proceedings of the 6th International Conference on Computing and Knowledge Engineering, IEEE, Mashhad, Iran, October 20-21, 2016, pp. 159-164.

[25] Z. Zhou, Y. Zhou, M. Cai, and L. Sun, "A workload model based approach to evaluate the robustness of real-time operating system". Proceedings of the 10th International Conference on High Performance Computing and Communications & International Conference on Embedded and Ubiquitous Computing, IEEE, 2013, pp. 2027-2033.

[26] R. Alur, and D. L. Dill, "A theory of timed automata. Theoretical computer science, vol.126, no. 2, pp. 183-235, 1994.

[27] L. Parra, A., Lindoso, M. Portela, L. Entrena, F. Restrepo-Calle, S. Cuenca-Asensi, and A. Martinez-Álvarez, "Efficient mitigation of data and control flow errors in microprocessors", IEEE Transactions on Nuclear Science, vol. 61, no. 4, pp. 1590-1596, 2014.

[28] A. Johansson, N. Suri, B. Murphy, "On the selection of error model (s) for OS robustness evaluation". Proceedings of the 37th Annual International Conference on Dependable Systems and Networks, IEEE, 2007, pp. 502-511.

[29] S. Winter, C. Sârbu, N. Suri, and B. Murphy, "The impact of fault models on software robustness evaluations". Proceedings of the 33rd International Conference on Software Engineering, ACM, 2011, pp. 51-60.

[30] H. Fayyad-Kazan, L. Perneel, and M. Timmerman, "Linux PREEMPT-RT vs. commercial RTOSs: How big is the performance gap? GSTF Journal on Computing, vol. 3, no. 1, pp. 135-142, 2013.

[31] M. R. Guthaus, J. S. Ringenberg, D. Ernst, T. M. Austin, T. Mudge,and R. B. Brown, "MiBench: A free, commercially representative embedded benchmark suite". International Workshop on Workload Characterization, IEEE, 2001, pp. 3-14.

[32] V. Prasad, W. Cohen, F. C. Eigler, M. Hunt, J. Keniston, and J. Chen, "Locating system problems using dynamic instrumentation". Proceedings of the Linux Symposium, Ottawa, Ontario, Canada, July 20-23, 2005, pp.49-64.

[33] A. Shahrokni, and R. Feldt, "A systematic review of software robustness", Information and Software Technology, vol. 55, no. 1, pp. 1-17, 2013.