Duminda Wijesekera
ABSTRACT
Counterexamples produced by model checkers are frequently exploited for the purpose of testing. Counterexamples and test cases are generally treated as essentially the same thing, while in fact they can differ significantly. For example, it might take more than one test case to "cover" a given counterexample, because not all property violations can be illustrated with linear counterexamples. This paper presents a formal relationship between counterexamples and test cases in the context of the Computation Tree Logic (CTL), the logic of the popular model checker SMV. Given a test requirement as a CTL formula, we define what it means for a set of test cases to cover a counterexample associated with that requirement. This result can not only be used in the generation of a test set that satisfies a given test coverage criterion, but also in the determination of whether an extant test set satisfies the criterion. Our results can guide the production of counterexamples in model checkers explicitly intended to support testing.
- A. Abdurazik, P. Ammann, W. Ding, and J. Offutt. Evaluation of three specification-based coverage testing criteria. In Proceedings ICECCS 2000: 6th IEEE International Conference on Engineering of Complex Computer Systems, pages 179--187, Tokyo, Japan, September 2000. Google Scholar
Digital Library
- P. Ammann and P. E. Black. A Specification-Based Coverage Metric to Evaluate Test Sets. In Proceedings of the 4th IEEE International Symposium on High-Assurance Systems Engineering (HASE '99), pages 239--248, Washington, DC, USA, 1999. IEEE Computer Society. Google ScholarDigital Library
- P. Ammann, W. Ding, and D. Xu. Using a Model Checker to Test Safety Properties. In Proceedings of the 7th International Conference on Engineering of Complex Computer Systems (ICECCS 2001), pages 212--221, Skovde, Sweden, 2001. IEEE. Google ScholarDigital Library
- P. E. Ammann, P. E. Black, and W. Majurski. Using Model Checking to Generate Tests from Specifications. In Proceedings of the Second IEEE International Conference on Formal Engineering Methods (ICFEM'98), pages 46--54. IEEE Computer Society, 1998. Google ScholarDigital Library
- J. Callahan, F. Schneider, and S. Easterbrook. Automated Software Testing Using Model-Checking. In Proceedings 1996 SPIN Workshop, August 1996. Also WVU Technical Report NASA-IVV-96-022.Google Scholar
- J. J. Chilenski and S. P. Miller. Applicability of modified condition/decision coverage to software testing. Software Engineering Journal, pages 193--200, September 1994.Google ScholarCross Ref
- E. Clarke and H. Veith. Counterexamples revisited: Principles, algorithms, applications. In Verification: Theory and Practice, volume 2772 of Lecture Notes in Computer Science, pages 208--224, 2004.Google Scholar
- E. M. Clarke and E. A. Emerson. Design and synthesis of synchronization skeletons using branching-time temporal logic. In Logic of Programs, Workshop, pages 52--71, London, UK, 1982. Springer-Verlag. Google ScholarDigital Library
- E. M. Clarke, O. Grumberg, K. L. McMillan, and X. Zhao. Efficient generation of counterexamples and witnesses in symbolic model checking. In Proceedings of the 32st Conference on Design Automation (DAC), pages 427--432. ACM Press, 1995. Google ScholarDigital Library
- E. M. Clarke, O. Grumberg, and D. A. Peled. Model Checking. MIT Press, Cambridge, MA., 1 edition, 2001. 3rd printing.Google Scholar
- E. M. Clarke, S. Jha, Y. Lu, and H. Veith. Tree-like counterexamples in model checking. In LICS '02: Proceedings of the 17th Annual IEEE Symposium on Logic in Computer Science, pages 19--29, Washington, DC, USA, 2002. IEEE Computer Society. Google ScholarDigital Library
- E. A. Emerson and J. Y. Halpern. Decision procedures and expressiveness in the temporal logic of branching time. In STOC '82: Proceedings of the fourteenth annual ACM symposium on Theory of computing, pages 169--180, New York, USA, 1982. ACM Press. Google ScholarDigital Library
- A. Engels, L. Feijs, and S. Mauw. Test generation for intelligent networks using model checking. In E. Brinksma, editor, Proceedings of the Third International Workshop on Tools and Algorithms for the Construction and Analysis of Systems. (TACAS'97), volume 1217 of Lecture Notes in Computer Science, Enschede, the Netherlands, April 1997. Springer-Verlag. Google ScholarDigital Library
- A. Gargantini and C. Heitmeyer. Using Model Checking to Generate Tests From Requirements Specifications. In ESEC/FSE'99: 7th European Software Engineering Conference, Held Jointly with the 7th ACM SIGSOFT Symposium on the Foundations of Software Engineering, volume 1687, pages 146--162. Springer, September 1999. Google ScholarDigital Library
- K. L. McMillan. The SMV system. Technical Report CMU-CS-92-131, Carnegie-Mellon University, 1992.Google Scholar
- I. L. Li Tan, Oleg Sokolsky. Specification-based testing with linear temporal logic. In Proceedings of IEEE International Conference on Information Reuse and Integration (IRI'04), pages 493--498, 2004.Google ScholarCross Ref
- R. Meolic, A. Fantechi, and S. Gnesi. Witness and counterexample automata for actl. In Formal Techniques for Networked and Distributed Systems - FORTE 2004, volume 3235 of Lecture Notes in Computer Science, pages 259--275, 2004.Google Scholar
- A. Pnueli. The temporal logic of programs. In 18th Annual Symposium on Foundations of Computer Science, 31 October-2 November, Providence, Rhode Island, USA, pages 46--57. IEEE, 1977.Google ScholarDigital Library
- C. Ramakrishnan and R. Sekar. Model-based vulnerability analysis of computer systems. In Proceedings of the 2nd International Workshop on Verification, Model Checking and Abstract Interpretation, September 1998.Google Scholar
- S. Rayadurgam and M. P. Heimdahl. Generating MC/DC Adequate Test Sequences Through Model Checking. In Proceedings of the 28th Annual NASA Goddard Software Engineering Workshop, pages 91--96, 2003.Google ScholarCross Ref
- S. Rayadurgam and M. P. E. Heimdahl. Coverage Based Test-Case Generation Using Model Checkers. In Proceedings of the 8th Annual IEEE International Conference and Workshop on the Engineering of Computer Based Systems (ECBS 2001), pages 83--91, Washington, DC, April 2001. IEEE Computer Society.Google Scholar
- R. W. Ritchey and P. Ammann. Using model checking to analyze network vulnerabilities. In Proceedings of the 2000 IEEE Symposium on Security and Privacy (Oakland 2000), pages 156--165, May 2000. Google ScholarDigital Library
Index Terms
- Relating counterexamples to test cases in CTL model checking specifications
Recommendations
Fair stateless model checking
PLDI '08Stateless model checking is a useful state-space exploration technique for systematically testing complex real-world software. Existing stateless model checkers are limited to the verification of safety properties on terminating programs. However, ...
A game-based framework for CTL counterexamples and 3-valued abstraction-refinement
This work exploits and extends the game-based framework of CTL model checking for counterexample and incremental abstraction-refinement. We define a game-based CTL model checking for abstract models over the 3-valued semantics, which can be used for ...
Handling loops in bounded model checking of C programs via k-induction
The first attempts to apply the k-induction method to software verification are only recent. In this paper, we present a novel proof by induction algorithm, which is built on the top of a symbolic context-bounded model checker and uses an iterative ...
Comments