Testing UML Designs

Project Summary

The research is concerned with developing mechanisms that support the generation of design-level test cases from UML design models. The generated test cases can be used in UML design reviews and inspections to validate software designs.

Developing a UML design testing approach and test generation support requires addressing the following issues:

• Providing an analyzable representation of UML models and test artifacts, and
• providing a flexible test generation mechanism.

To address these issues the research is structured into the following research activities:

• Deriving test objectives from test-oriented formalizations of Class and Collaboration Diagrams.
• Defining strategies for creating test objectives. These strategies will be expressed in terms of model coverage goals.
• Providing techniques that can be partially automated for systematically generating test cases from test objectives.
• Assessing the effectiveness of the design testing technique developed in this research.

This research will produce results that can be used to develop industrial-strength techniques and tools for evaluating UML designs.

---

People

Current Project Members

• Sudipto Ghosh, Associate Professor, Computer Science Department, CSU.
• Robert B. France, Professor, Computer Science Department, CSU.
• Aritra Bandyopadhyay, MS student, Computer Science Department, CSU.

Past Project Members

• Anneliese Katharina Amschler Andrews, Professor, Department of Computer Science, University of Denver.
• Trung T. Dinh-Trong, Graduate Student, Computer Science Department, CSU.
• Orest Pilskalns, Assistant Professor, School of Electrical Engineering and Computer Science, WSU - Vancouver.
• Brent Wilkins
• Conrad Braganza
• Gerald Craig
• Mike Hamilton
• Nilesh Kawane

---

Funding

• "Testing UML Designs," Cost Extension, NSF, R. B. France, A. Andrews and S. Ghosh, $60,000, September 2005 - AUgust 2006.
• "Testing UML Designs", National Science Foundation SEL, R. B. France, A. Andrews and S. Ghosh, $300,000, September 2002 - August 2005.
• "Providing Support for Executing and Testing UML Design Models", S. Ghosh, Eclipse Innovation Grant, IBM, $20,000, January 2004 - December 2004.
• "Providing Support for Executing and Testing UML Design Models", S. Ghosh, IBM Innovation Grant, $23,500, January 2005 - December 2005.
• "Testing UML Designs", National Science Foundation REU, $6000, Summer 2005.

---

Publications

The documents contained in these area are included by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically.It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

• Aritra Bandyopadhyay (2009), "Test Input Generation using UML Sequence and State Machines Models", A. Bandyopadhyay and S. Ghosh, Proceedings of the 2nd International Conference on Software Testing, Verification, and Validation (ICST), Denver, CO, April 1-4, 2009.

  Abstract

  We propose a novel testing approach that combines information from UML sequence models and state machine models. Current approaches that rely solely on sequence models do not consider the effects of the message path under test on the states of the participating objects. Dinh-Trong et al. proposed an approach to test input generation using information from class and sequence models. We extend their Variable Assignment Graph (VAG) based approach to include information from state machine models. The extended VAG (EVAG) produces multiple execution paths representing the effects of the messages on the states of their target objects. We performed mutation analysis on the implementation of a video store system to demonstrate that our test inputs are more effective than those that cover only sequence diagram paths.

• T. T. Dinh-Trong (2007), "A Systematic Approach to Testing UML Designs" , Ph.D's Dissertation, Colorado State University, Fort Collins, Colorado

• Abstract

  In Model Driven Engineering (MDE) approaches, developers create and refine design models from which substantial portions of implementations are generated. During refinement, undetected faults in an abstract model can traverse into the refined models, and eventually into code. Hence, finding and removing faults in design models is essential for MDE approaches to succeed.

  This dissertation describes a testing approach to finding faults in design models created using the Unified Modeling Language (UML). Executable forms of UML design models are exercised using generated test inputs that provide coverage with respect to UML-based coverage criteria. The UML designs that are tested consist of class diagrams, sequence diagrams and activity diagrams.

  The contribution of the dissertation includes (1)~a test input generation technique, (2)~an approach to execute design models describing sequential behavior with test inputs in order to detect faults, and (3)~a set of pilot studies that are carried out to explore the fault detection capability of our testing approach.

  The test input generation technique involves analyzing design models under test to produce test inputs that satisfy UML sequence diagram coverage criteria. We defined a directed graph structure, named Variable Assignment Graph (VAG), to generate test inputs. The VAG combines information from class and sequence diagrams. Paths are selected from the VAG and constraints are identified to traverse the paths. The constraints are then solved with a constraint solver.

  The model execution technique involves transforming each design under test into an executable form, which is exercised with the generated inputs. Failures are reported if the observed behavior differs from the expected behavior. We proposed an action language, named Java-like Action Language (JAL), that supports the UML action semantics. We developed a prototype tool, named \emph{UMLAnT}, that performs test execution and animation of design models.

  We performed pilot studies to evaluate the fault detection effectiveness of our approach. Mutation faults and commonly occurring faults in UML models created by students in our software engineering courses were seeded in three design models. Ninety percent of the seeded faults were detected using our approach.

• T. Dinh-Trong, S. Ghosh, and R. B. France (2006), "A Systematic Approach to Generate Inputs to Test UML Design Models" , 17th IEEE International Symposium on Software Reliability Engineering (ISSRE), 6-10 November 2006, Raleigh, North Carolina, USA

• Abstract

  Practical model validation techniques are needed for model driven development (MDD) techniques to succeed. This paper presents an approach to generating inputs to test UML design models that are produced in the detailed design phase of an MDD project. A symbolic execution based approach is used to derive test input constraints from the paths of a Variable Assignment Graph, which integrates information from UML class and sequence diagrams. The constraints are solved using Alloy, a configuration constraint solver, to obtain the test inputs. The results of a pilot study carried out to explore the fault detection capability of the test inputs are reported.

• T. Dinh-Trong, S. Ghosh, R. B. France, M. Hamilton, and B. Wilkins (2005), "UMLAnT: An Eclipse Plugin for Animating and Testing UML Designs", Eclipse Technology Exchange Workshop, in conjunction with OOPSLA, San Diego, USA

• Abstract

  We describe the UML Animator and Tester (UMLAnT), which is an Eclipse plug-in for animating and testing UML models. UMLAnT can be used both by developers in the software industry and students who are learning concepts in object-oriented modeling. UMLAnT helps designers visualize the behavior specified in a UML model by displaying animated object diagrams and sequence diagrams. UMLAnT allows testers to specify test cases and notifies them about failures during test execution.

• T. Dinh-Trong, S. Ghosh, R. France, A. A. Andrews (2005), "Generating test data to test UML Design Models" , 2nd MoDeVa workshop - Model design and Validation, Montego Bay, Jamaica

• Abstract

  This paper presents an approach to generating inputs that can be used to test UML design models. A symbolic execution based approach is used to derive test input constraints from a Variable Assignment Graph (VAG), which presents an integrated view of UML class and sequence diagrams. The constraints are solved using Alloy, a configuration constraint solver, to obtain the test inputs.

• T. Dinh-Trong, S. Ghosh, R. France, B. Baudry, F. Fleury (2005), "A Taxonomy of Faults for UML Designs" , 2nd MoDeVa workshop - Model design and Validation, Montego Bay, Jamaica

• Abstract:

  As researchers and practitioners start adopting model-based software development techniques, the need to rigorously evaluate design models is becoming apparent. Evaluation techniques typically use design metrics or verification and validation approaches that target specific types of faults in the models. Fault models and taxonomies may be used to develop design techniques that reduce the occurrence of such faults as well as techniques that can detect these faults. Fault models can also be used to evaluate the effectiveness of verification and validation approaches. In this paper we present a taxonomy of faults that occur in UML designs. We also describe a set of mutation operators for UML class diagrams.

• T. Dinh-Trong, S. Ghosh, R. France, A. A. Andrews (2005), "A Systematic Approach to Testing UML Design Models" , 4th International Workshop on Critical Systems Development Using Modeling Languages (CSDUML), Fredrikstad, Norway

• Abstract:

  Practical validation techniques are needed for model driven development techniques to succeed. This paper presents an approach to testing UML design models. A symbolic execution based approach is used to derive test inputs from a Variable Assignment Graph (VAG), which presents an integrated view of UML class and sequence diagrams. The inputs are used to test executable forms of UML models derived from UML class and activity diagrams. We describe the design of a prototype tool that supports the testing of UML models.

• N. Kawane (2005), "EPTUD: An Eclipse Plugin For Testing UML Design Models", Master's thesis, Colorado State University, Fort Collins, Colorado.

• Abstract

  The Unified Modeling Language (UML) is the de facto standard for modeling software systems. It provides a wide range of notations to model software designs from different perspectives. In Model Driven Development (MDD) approaches, the UML is used to model a system at various levels of abstractions. Substantial system implementations are often generated from the design models. Finding and removing faults from the UML models during the early development stages can be less costly and require less effort than finding faults later in the refinements (including code). A number of UML drawing tools provide model verification support that is limited to syntax checking, and structural and consistency analysis. Current design evaluation techniques include walkthroughs, inspections, and other forms of reviews. All these techniques lack the rigor of systematic testing techniques and are not practical for large and complex software designs. Detecting faults when behavior is described by a large number of UML diagrams is error prone when carried out manually.

  This thesis presents an approach and a prototype implementation to test design models described by UML class diagrams and activity diagrams.

  In the approach, executable code is generated from the design models to simulate the behavior of a system. Test scaffolding is added to the generated code to automate test execution and failure detection. A number of failure conditions are identified and checks implemented to detect failures. The prototype is implemented as an Eclipse plugin. The plugin extends JUnit functionality to support testing of models. The approach and application of the plugin are illustrated with the help of a bookstore product inventory system.

• T. Dinh-Trong, N. Kawane, S. Ghosh, R. France, A. A. Andrews (2005), "A Tool-Supported Approach to Testing UML Design Models", 10th IEEE International Conference on Engineering of Complex Computer Systems (ICECCS), Shanghai, China

• Abstract:

  For Model Driven Development approaches to succeed, there is a need for model validation techniques. This paper presents an approach to testing designs described by UML class diagrams, interaction diagrams, and activity diagrams. A UML design model under test is transformed into an executable form. Test infrastructure is added to the executable form to carry out tests. During testing, object configurations are created, modified and observed. In this paper, we identify the structural and behavioral characteristics that need to be observed during testing. We describe a prototype tool that (1) transforms UML design models into executable forms with test infrastructure, (2) executes tests, and (3) reports failures.

• T. T. Dinh-Trong (2004), "A Systematic Approach to Testing UML Design Models", Doctoral Symposium, 7th International Conference on the Unified Modeling Language (UML), Lisbon, Portugal.

• Abstract:

  Finding and removing faults in the design phase can reduce software development cost and time to market. Designs are typically evaluated using walkthroughs, inspections, and other forms of reviews that lack the rigor of systematic testing techniques. This research proposes a systematic approach to testing design models described by UML class diagrams, sequence diagrams, and activity diagrams. An executable form of a UML design model under test is exercised using generated input. The expected behavior of a design under test is compared with the actual behavior that is observed during testing. Failures are reported if the observed behavior differs from the expected behavior.

• T. Dinh-Trong, N. Kawane, S. Ghosh, R. France (2004), "EPTUD: An Eclipse Plugin for Testing UML Designs", Lecture Notes in Computer Science (Vol. 3297/2005): UML Modeling Languages and Applications: 2004 Satellite Activities, Lisbon, Portugal, October 11-15, 2004, Revised selected papers.

• Abstract

  For model driven development approaches to succeed, there is a need for developing techniques for validating models. Studies show that many software faults occur in the design phase. Hence, it is essential to find and remove faults in design models. Currently, UML design models are typically evaluated using walkthroughs, inspections, and other informal types of design review techniques that are largely manual and consequently, tedious, error-prone and less effective.

• O. Pilskalns, A. Andrews, R. France, S. Ghosh (2003), "Rigorous Testing by Merging Structural and Behavioral UML Representations", Sixth International Conference on the Unified Modeling Language, 2003, San Francisco, CA, USA, October 20-24, 2003.

• Abstract:

  Error detection and correction in the design phase can reduce total costs and time to market. Yet, testing of design models usually consists of walk-throughs and inspections both of which lack the rigor of systematic testing. Test adequacy criteria for UML models help define necessary objectives during the process of test creation. These test criteria require coverage of various parts of UML models, such as structural (Class Diagram) and behavioral (Sequence Diagram) views. Test criteria are specific to a particular UML view. Test cases on the other hand should cover parts of multiple views. To understand testing needs better, it is useful to be able to observe the effect of tests on both Class Diagrams and Sequence Diagrams. We propose a new graph that encapsulates the many paths that exist between objects via their method calls as a directed acyclic graph (OMDAG). We also introduce the object method execution table (OMET) that captures both execution sequence and associated attribute values by merging the UML views. The merging process is defined in an algorithm that generates and executes tests.

• "Test Adequacy Assessment for UML Design Model Testing", S. Ghosh, R. B. France, C. Braganza, N. Kawane, A. Andrews and O. Pilskalns, International Symposium on Software Reliability Engineering, ISSRE 2003, Denver, CA,USA, November 17-20, 2003.

• Abstract :

  Systematic design testing, in which executable models of behaviors are tested using inputs that exercise scenarios, can help reveal flaws in designs before they are implemented in code. We present a testing method in which executable forms of the Unified Modeling Language (UML) models are tested. The method incorporates the use of test adequacy criteria based on UML model elements in class diagrams and interaction diagrams. Class diagram criteria are used to determine the object configurations on which tests are run, while interaction diagram criteria are used to determine the sequences of messages that should be tested. The criteria can be used to define test objectives for UML designs. In this paper, we describe and illustrate the use of the proposed test method and adequacy criteria.

  Keywords: design reviews, software testing, test adequacy criteria, UML, class diagram, collaboration diagram, category partitioning.

• "A Systematic Procedure for Testing UML Designs", Trung Thanh Dinh Trong.[Student paper for International Symposium on Software Reliability Engineering, ISSRE 2003, Denver, USA, November 17-20, 2003.]

• Abstract :

  Dynamic testing of design models, in which behavioral models are executed, can reveal flaws in the design level before they are implemented, and thus reduce the costs and time to market. This paper presents a systematic procedure for testing UML designs and observing the test results. This procedure is being incorporated into a prototype UML testing tool.

  Keywords: Design model, testing, UML.

• "Fault Detection Effectiveness of UML Design Model Test Adequacy Criteria", Nilesh Kawane.[Student paper for International Symposium on Software Reliability Engineering, ISSRE 2003, Denver, USA, November 17-20, 2003.]

• Abstract :

  UML models are widely used by software developers to model complex software systems. Testing the models can reveal flaws in the early stages of software development. Criteria based on UML class diagrams and interaction diagrams were proposed earlier to specify UML model elements that need to be covered during testing. This paper describes a case study that evaluated the fault detection effectiveness of these criteria.

  Keywords: Class diagram, collaboration diagram, software testing, test adequacy criteria, UML.

• A. Andrews, R. France, S. Ghosh, and G. Craig. "Test Adequacy Criteria for UML Design Models". Journal of Software Testing, Verification and Reliability, 13(2):95_127, April- June 2003.

• Abstract :

    Systematic design testing, in which executable models of behaviors are tested using inputs that exercise scenarios, can help reveal  flaws in designs before they are implemented in code. In this paper a technique for testing executable forms of UML (Unified Modeling Language) models is described and test adequacy criteria based on UML model elements are proposed. The criteria can be used to define test objectives for UML designs. The UML design test criteria are based on the same premise underlying code test criteria: coverage of relevant building blocks of models are highly likely to uncover faults. The test adequacy criteria proposed in this paper are based on building blocks for UML Class and Interaction Diagrams. Class Diagram criteria are used to determine the object configurations on which tests are run, while Interaction Diagram criteria are used to determine the sequences of messages that should be tested.

  Keywords: design reviews, software testing, test adequacy criteria, UML, class diagram, collaboration diagram, category partitioning.